Data Quality in the Databricks Lakehouse

We will have Vitalija Bartuseviciute and Alexander Klein, two of the top-speakers around Databricks as guests in Berlin, so this session is guaranteed to be packed with awesome insights and information. Be sure not to miss it!

As modern enterprises adopt the Lakehouse architecture to unify data engineering, analytics, and AI, maintaining high data quality becomes a foundational requirement. In this session, we explore how Databricks enables scalable, automated data quality management using a combination of open-source and native tools.

First, we will uncover Databricks Labs DQX. DQX is a simple validations framework for assesing data quality of PySpark dataframes. The main benefit is as it allows real-time quality validation during data processing rather than post-fact monitoring as Databricks Data Quality Monitoring does. This alllows quicker identification and resolution of data quality problems. As it allows to quarantine invalid data and investigate data quality issues before they actually are written to the table.

Then we will dive into Databricks Data Quality Monitoring. Databricks Data Quality Monitoring is a native feature which allows monitor the statistical properties and quality of the data in all of the tables in your account. Monitoring your data provides quantitative measures that help you track and confirm the quality and consistency of your data over time.