Practical Data Quality Monitors on Databricks: A 4-Layer Quality Strategy

Data quality management is not something a single mechanism can solve. On Databricks, combining four layers — in-pipeline quality control (DLT Expectations), table-level constraints (Delta Constraints), statistical quality monitoring (Lakehouse Monitor), and threshold-based notifications (SQL Alerts) — lets you build an end-to-end quality strategy from detection through notification and remediation.

Overview of the 4-Layer Quality Strategy

Each layer targets a different phase and detection timing. Moving from upstream to downstream through the pipeline, the design lays out defensive lines in the order prevent → detect → monitor → notify.

Layer 1: DLT Expectations (In-Pipeline Quality Control)

Delta Live Tables (DLT) Expectations is a mechanism for declaratively embedding quality rules inside the pipeline definition. For each Expectation you specify the behavior on violation (warn / drop / fail).

import dlt
from pyspark.sql.functions import col

@dlt.expect_or_drop("valid_amount", "amount > 0")
@dlt.expect_or_fail("not_null_user_id", "user_id IS NOT NULL")
@dlt.expect("valid_email", "email LIKE '%@%'")
@dlt.table(comment="Quality-validated order data")
def orders_clean():
    return dlt.read("orders_raw").filter(col("order_date") >= "2025-01-01")

• expect: continue processing while keeping violating rows and recording them in metrics (equivalent to warn)
• expect_or_drop: drop violating rows and continue processing (quarantine pattern)
• expect_or_fail: stop the entire pipeline when violating rows exist (for fatal quality violations)

Expectation results are recorded in the DLT pipeline's event log, and you can check the pass rate in the DAG view of the UI. It is well-suited for null removal and format validation during Bronze → Silver transformations and acts as a quality gate in the Medallion architecture.

Layer 2: Delta Constraints (Table-Level Constraints)

Delta Constraints is a mechanism for setting NOT NULL and CHECK constraints on Delta tables. Even for batch jobs or SQL-based ETL that do not use DLT, you can stake out a last line of defense on the table itself.

-- Add NOT NULL constraints
ALTER TABLE gold.orders
ALTER COLUMN order_id SET NOT NULL;

ALTER TABLE gold.orders
ALTER COLUMN customer_id SET NOT NULL;

-- Add CHECK constraints
ALTER TABLE gold.orders
ADD CONSTRAINT valid_amount CHECK (amount > 0);

ALTER TABLE gold.orders
ADD CONSTRAINT valid_status CHECK (status IN ('pending', 'shipped', 'delivered', 'cancelled'));

-- Inspect the constraints
DESCRIBE DETAIL gold.orders;

On a constraint violation, the entire INSERT/UPDATE/MERGE transaction is rolled back. The difference from DLT Expectations is that DLT is bound to a pipeline definition, whereas Delta Constraints are persisted on the table definition. The constraints apply no matter which ETL tool or job writes to the table, which makes them especially effective in multi-pipeline environments.

Layer 3: Lakehouse Monitor (Statistical Quality Monitoring)

Lakehouse Monitor is a feature that periodically measures column statistics (null rate, mean, distribution) and data drift on Unity Catalog managed tables. It has two modes: snapshot analysis and time-series analysis.

-- Create a Lakehouse Monitor (SQL)
CREATE MONITOR gold.orders
WITH (
  PROFILE_TYPE = 'SNAPSHOT',
  SCHEDULE = CRON '0 8 * * *',    -- Run every morning at 8:00
  OUTPUT_SCHEMA = gold
);

-- Time-series mode
CREATE MONITOR gold.daily_metrics
WITH (
  PROFILE_TYPE = 'TIME_SERIES',
  TIMESTAMP_COL = 'event_date',
  GRANULARITIES = ('1 day'),
  SCHEDULE = CRON '0 9 * * *',
  OUTPUT_SCHEMA = gold
);

Monitor automatically generates two tables.

• profile_metrics: per-column null rate, distinct count, mean, min/max, distribution histogram, and more
• drift_metrics: drift scores compared to a baseline (KL divergence, JS divergence, etc.)

When applied to an ML model's input table, it lets you detect feature distribution shifts (data drift) early and provides input for the model retraining decision.

Layer 4: SQL Alerts (Threshold-Based Notifications)

SQL Alerts is a mechanism that sends notifications when the result of a SQL query exceeds a threshold. You configure them against Lakehouse Monitor metrics tables or your own quality-check queries.

-- Example alert query: detect columns with > 5% null rate
SELECT
  column_name,
  ROUND(percent_nulls * 100, 2) AS null_pct
FROM gold.profile_metrics
WHERE log_date = CURRENT_DATE()
  AND percent_nulls > 0.05
ORDER BY null_pct DESC

Register this query as a Databricks SQL Alert and configure it to notify a Slack channel when at least one result row exists. The evaluation interval can be set as fast as 1 minute.

Alert Integration Flow

Here is the typical flow in which the four layers work together.

[Raw Data]
  │
  v
[DLT Pipeline]
  ├─ Expectation: expect_or_drop("valid_amount", "amount > 0")
  ├─ Expectation: expect_or_fail("not_null_id", "id IS NOT NULL")
  │   → Violating rows are quarantined / stop the run; metrics go to the event log
  v
[Silver/Gold Table]
  ├─ Delta Constraint: CHECK (amount > 0), NOT NULL (id)
  │   → Reject the transaction on constraint violation
  v
[Lakehouse Monitor] (every morning at 8:00)
  ├─ profile_metrics: null rate, distribution, summary statistics
  ├─ drift_metrics: drift from the baseline
  v
[SQL Alert] (fires when result rows > 0)
  └─ Slack / Email / Webhook → ops team investigates

Criteria for Choosing Between Layers

Best Practices for Real-World Design

• In Bronze → Silver transformations, use DLT Expectations (expect_or_drop) for null removal and format validation
• Always set Delta Constraints (NOT NULL + CHECK) on Gold tables to guarantee quality even with multi-path writes
• Set up Lakehouse Monitor (TIME_SERIES) on ML model input tables to detect feature drift daily
• Register an anomaly-detection query against profile_metrics as a SQL Alert and notify the ops team in Slack in real time
• Because the fail setting in Expectations has a large production impact, start by running with warn and switch to fail once it has stabilized

Check Your Understanding

Frequently Asked Questions