Databricks Lakeflow Connect Guide: Choose Connectors, Monitor Syncs, and Recover Ingestion Failures

Overview

Lakeflow Connect is best evaluated as part of an end-to-end ingestion design rather than as a standalone connection screen. The connector is only one decision. You also need to define the source system, authentication method, landing tables, update behavior, schedule, schema policy, access controls, and downstream consumers.

Connector availability, setup screens, source limitations, supported authentication methods, and operational features can change over time. Confirm the current capabilities in the Databricks documentation and your workspace before committing to a design. The principles below remain useful even when the exact connector configuration changes.

A practical design usually separates three layers:

• Source connection: The credentials, network path, permissions, and connector configuration required to read from the source.
• Landing data: The tables or volumes where ingested records first arrive, with enough metadata to support auditing and recovery.
• Curated data products: Cleaned, modeled, and governed tables used by analytics, applications, machine learning, or downstream sharing.

Keeping these layers distinct makes troubleshooting easier. A source authentication failure is different from a schema change, and neither should be confused with a transformation error in a downstream job. For governed sharing requirements, review the Databricks Delta Sharing guide before exposing curated data to other teams or organizations.

What to track

Connector selection criteria

Begin with the source and the required freshness, not with a preferred connector name. Record the answers in a short design document so that the choice can be revisited later.

Do not assume that every source supports the same incremental-load behavior. If a source cannot provide dependable change information, document whether the design uses periodic full loads, a source-side export, a watermark, or another supported approach. Estimate the effect on source capacity and destination storage before scheduling recurring runs.

First-run validation checklist

Before enabling a recurring schedule, validate a deliberately small and representative slice of data.

1. Confirm that the connection uses a dedicated identity with only the permissions required for ingestion.
2. Test the network path and verify that the source can be reached from the intended workspace or compute context.
3. Record the source objects selected, the expected row or record grain, and any exclusions.
4. Run an initial load and compare counts, keys, timestamps, null patterns, and representative values with the source.
5. Verify that source updates, inserts, and deletes behave as expected for the chosen ingestion mode.
6. Inspect how the destination handles data types, nested fields, special characters, and schema changes.
7. Confirm that retries do not create unintended duplicates or skip records.
8. Check permissions for operators, downstream readers, and monitoring users.
9. Capture run identifiers, timestamps, status, error details, and the location of operational logs.

Use a landing table as an evidence layer when practical. Preserve ingestion metadata such as source object, ingestion timestamp, batch or run identifier, and an appropriate source key. This helps distinguish a source correction from a duplicate delivery and gives operators a safer starting point for replay.

Sync health review worksheet

For each recurring sync, record the following fields during an operations review:

• Source and connector name
• Business owner and technical owner
• Destination catalog, schema, and table or volume
• Expected schedule and freshness objective
• Last successful run and last failed run
• Records read, records written, and rejected records
• Average and maximum run duration
• Current watermark, cursor, or checkpoint
• Recent schema changes
• Credential expiration or rotation date
• Open alerts, known limitations, and next action

Track trends rather than relying only on the latest green status. A sync that succeeds but takes progressively longer may be approaching a source, compute, or destination limit. A stable record count that suddenly drops can indicate a filter, permission change, source outage, or checkpoint problem.

Cadence and checkpoints

Use different review frequencies for different risks. A lightweight schedule keeps ingestion work visible without turning every run into a manual inspection.

• Every run: Check completion status, duration, records processed, rejected records, and freshness. Alert on failures and on unusually large deviations from the normal range.
• Weekly: Review repeated warnings, slow runs, retry frequency, late-arriving data, and unresolved data-quality exceptions.
• Monthly: Reconcile representative source and destination counts, review access, confirm ownership, inspect storage growth, and check upcoming credential rotations or source maintenance.
• Quarterly: Reassess whether the connector, schedule, landing pattern, and destination design still match business requirements. Retire unused connections and update runbooks.
• After a source change: Pause or isolate the affected sync when appropriate, test the changed objects, and confirm schema and record behavior before restoring normal schedules.

A checkpoint should produce an actionable result. For example, “sync is healthy” is less useful than “the last 30 runs completed within the expected window, no records were rejected, the watermark advanced, and no source schema changes were detected.” Store the review outcome with the pipeline or operational documentation so another engineer can understand the decision.

How to interpret changes

When a sync changes behavior, first classify the symptom. This avoids applying a destructive fix to the wrong layer.

Recovery should favor reversibility. Preserve failed-run details, avoid deleting landing data before understanding the failure, and use a bounded replay window when the connector and source support it. If a checkpoint is suspected, document its current value before changing it. Reconcile the replayed interval afterward so that a successful status does not hide missing or duplicated data.

For a practical failure-recovery workflow:

1. Freeze unrelated configuration changes and identify the first failed run.
2. Classify the failure as authentication, connectivity, source availability, schema, checkpoint, destination, or downstream transformation.
3. Check whether a retry is safe and whether it could duplicate records.
4. Resolve the underlying issue in a test or controlled setting where possible.
5. Replay only the necessary interval or objects.
6. Compare source and destination counts and keys for the affected window.
7. Record the cause, resolution, validation evidence, and any preventive action.

When ingestion feeds AI applications, downstream model quality can also reveal ingestion problems. A sudden change in retrieval results, evaluation scores, or application responses may warrant checking the underlying data freshness and schema. The Databricks LLM evaluation guide and Databricks RAG tutorial provide related guidance for validating AI workloads after data changes.

When to revisit

Revisit this guide and your connector inventory monthly or quarterly, depending on the criticality of the data. Perform an additional review whenever a new source is onboarded, a source vendor changes its API or schema, credentials are rotated, a sync begins failing, freshness requirements change, or a downstream application reports unexpected results.

During each review, confirm four things: the connector is still available and appropriate, the authentication method remains supportable, incremental behavior still matches the source, and the destination remains suitable for current consumers. Also update screenshots, setup references, source limitations, and runbook instructions when the Databricks interface or connector capabilities change.

To make the next review practical, create one inventory row per sync and assign an owner. Add the expected schedule, freshness target, last successful run, current checkpoint, credential date, and next review date. Then choose one representative sync and complete the validation checklist. Small, documented checks are easier to repeat than a large annual audit, and they expose reliability drift before it becomes a data incident.

If the ingestion feeds governed sharing, model serving, or other production workloads, connect the review to those systems' operational checks. See the Databricks AI governance checklist for governance considerations and the Databricks Model Serving guide when refreshed data supports deployed models.