Reliable Databricks pipelines need more than a few validation rules. They need a repeatable way to measure freshness, completeness, uniqueness, validity, and referential integrity; isolate bad records; and turn failures into actionable incidents. This guide explains how to design Databricks data quality checks, use expectations without silently corrupting downstream data, and establish a monthly or quarterly review cycle for data reliability.
Overview
Data quality checks are controls that compare incoming data with explicit business and technical rules. In Databricks, those controls can be implemented in SQL or Python pipeline logic and, where supported by the pipeline runtime, expressed as expectations that measure whether records meet defined conditions.
An expectation should answer three questions:
- What condition must be true? For example, an order identifier must be present and an event timestamp must be within an acceptable range.
- What happens when the condition fails? The record may be retained with a failure count, dropped from the trusted output, or cause the pipeline step to fail.
- How will the failure be investigated? Operators need metrics, sample records, source context, and an owner.
Use expectations as part of a layered design rather than as the only line of defense. Schema enforcement protects structure, quality rules protect meaning, access controls protect sensitive data, and monitoring shows whether the system is behaving consistently. For related operational controls, see the Databricks Delta Lake Schema Evolution Guide and the Databricks Unity Catalog Permissions Guide.
What to track
Start with a quality inventory for each important dataset. Record the rule, its rationale, the expected threshold, the failure action, and the person or team responsible for responding. The following dimensions cover most recurring data quality reviews.
Freshness
Freshness measures whether data arrived within the expected time window. Compare the latest accepted event or ingestion timestamp with the current processing time. Track both the normal arrival interval and the maximum tolerated delay. A freshness check should distinguish a delayed source from a source that has stopped publishing entirely.
Completeness
Check required columns for null or empty values, and compare record counts with a sensible baseline. Completeness rules should be specific to the field: a missing customer identifier may be a hard failure, while a missing optional description may be reportable but acceptable. Track the failure rate, not only the count, because a small count can still represent a serious problem in a small batch.
Uniqueness
Identify keys that should not repeat, such as a transaction identifier or a source-system event ID. First define the grain of the table. A repeated identifier may be valid when the table stores versioned records, but invalid when it represents one current row per entity. Include duplicate counts and the columns used to calculate the key.
Validity
Validity rules test whether values fit an allowed format, range, or vocabulary. Examples include positive quantities, recognized status values, valid timestamps, and identifiers that match an expected pattern. Keep business validity rules close to their owner and document exceptions rather than weakening a rule until it produces no alerts.
Referential integrity
Check whether keys in a child dataset resolve to an approved parent dataset. This is particularly useful for customer, product, account, and organization relationships. Define how to handle late-arriving parent records before enabling a strict failure action. Otherwise, a correct child record can be rejected simply because ingestion order was different.
Rule-to-failure-action comparison
| Action | Use it when | Risk to manage |
|---|---|---|
| Measure and retain | You need visibility while assessing a new or noncritical rule. | Invalid records can reach downstream consumers if outputs are not clearly labeled. |
| Drop invalid records | A record cannot safely enter the trusted dataset and can be investigated separately. | Silent loss is possible unless dropped records and counts are retained. |
| Fail the update | Continuing would create materially incorrect data or violate a critical contract. | Consumers may experience delay; the incident process must be ready. |
Expectation syntax and available monitoring fields can vary by Databricks pipeline capability and runtime. Keep the rule expression simple, test it against representative data, and confirm the supported syntax in the documentation for the environment being deployed.
Quarantine design template
A quarantine table prevents invalid records from disappearing while protecting trusted outputs. At minimum, include:
- the original record or a serialized representation of it;
- source name, ingestion batch, file or message identifier, and ingestion timestamp;
- the failed rule name and a readable failure reason;
- pipeline, dataset, and rule version identifiers;
- an investigation status such as new, reviewed, corrected, replayed, or waived;
- owner, review timestamp, and remediation notes.
Keep quarantine data governed like production data. It may contain sensitive fields, and it should have a retention and access policy. A quarantine table is not a permanent substitute for correcting the source or transformation.
Cadence and checkpoints
Use operational alerts for immediate failures and a scheduled review for slower changes. A practical cadence has three levels.
- Every pipeline run: Check whether critical expectations passed, whether freshness is within the service window, and whether invalid-record volume changed sharply. Capture the run identifier and affected dataset.
- Weekly: Review recurring failures, quarantine backlog, unresolved ownership, and rules that are frequently waived. Compare failure rates by source and pipeline version.
- Monthly or quarterly: Reconfirm thresholds, required fields, reference data, retention, alert recipients, and business ownership. Review whether a rule still reflects the table grain and current process.
Alerting should be tied to a response, not merely a notification. An alert checklist should include the dataset and environment, failed rule, current and historical failure rate, affected time window, pipeline run, source owner, quarantine location, and escalation path. If lineage or catalog integrations are available in the environment, link the incident to the relevant dataset and downstream consumers.
For pipeline changes, review data quality checks alongside schema changes and deployment approvals. For model features or AI applications, connect input-quality trends with evaluation results; a model regression may be caused by a shift in source data rather than a change to the model itself.
How to interpret changes
Do not treat every failed record as an incident of the same severity. Interpret the change using four questions:
- Did the volume change? Compare the current failure rate with recent comparable runs. A rate is usually more useful than a raw count when batch sizes vary.
- Did the pattern change? Segment failures by source, partition, customer, geography, producer version, and rule. Concentration often points to a single upstream process.
- Did the business meaning change? A new status value or identifier format may be a legitimate source change. Confirm the contract before relaxing the validation rule.
- Did the pipeline change? Compare deployments, schema updates, joins, reference tables, and transformation logic with the first failing run.
Use this incident-triage worksheet:
- Detection: Which rule failed, when was it detected, and which runs are affected?
- Scope: How many records failed, what percentage of the batch is affected, and which downstream tables or applications depend on it?
- Cause hypothesis: Is the likely source a producer change, late data, schema drift, reference-data gap, duplicate delivery, or transformation defect?
- Containment: Should the pipeline continue, quarantine records, pause publication, or replay a known-good input?
- Resolution: What code, source, rule, or reference-data change is required?
- Evidence: Which samples, run logs, metrics, and lineage relationships support the conclusion?
- Follow-up: What test, alert, ownership change, or documentation update will prevent recurrence?
When a threshold is noisy, investigate before increasing it. Separate expected seasonality from genuine degradation, and prefer a documented exception with an expiration date over a permanent waiver.
When to revisit
Review this data quality design whenever a source is added, a business rule changes, a schema evolves, a reference dataset is replaced, or a downstream consumer reports unexpected results. Also revisit it after a major incident, a pipeline migration, a model retraining cycle, or a change in the acceptable freshness window.
At least monthly for active pipelines and quarterly for stable ones, confirm that every critical rule has an owner, a measured outcome, a defined failure action, and a tested recovery path. Retire checks that no longer describe the data, but preserve their history so trend comparisons remain understandable.
To make the review actionable, create a small quality register with these columns: dataset, dimension, rule, threshold, action, owner, last review date, current status, and next review date. Begin the next review by examining the rules with the largest change in failure rate, the oldest unresolved quarantine records, and any datasets whose consumers or schemas changed. If the data feeds an AI or machine-learning system, compare these findings with the model evaluation record and feature consistency checks; the Databricks LLM Evaluation Guide provides a related framework for tracking test sets and regressions.