This Databricks RAG tutorial explains how to build a maintainable retrieval-augmented generation application, then monitor the data, retrieval, prompts, model outputs, and operating costs that determine its quality over time. You will learn what to track in Mosaic AI RAG workflows, how to structure evaluation checkpoints, and when to update embeddings, prompts, or foundation models.
Overview
A RAG application combines a language model with a retrieval system. Instead of asking the model to answer from its pretrained knowledge alone, the application searches a controlled collection of documents and places relevant passages in the prompt. The model then produces an answer grounded in that retrieved context.
A typical Databricks implementation has five logical layers:
- Ingestion: files, database records, support tickets, policies, or other approved sources are loaded into governed storage.
- Preparation: content is cleaned, split into chunks, enriched with metadata, and assigned stable identifiers.
- Indexing: chunks are converted into Databricks embeddings and made searchable through Mosaic AI Vector Search or another approved retrieval layer.
- Generation: a retriever selects context, a prompt assembles the question and evidence, and a foundation model generates the response.
- Evaluation and operations: quality, latency, failures, access controls, and usage are measured through repeatable checks.
The difficult part is rarely connecting these components once. The ongoing challenge is detecting when a change in source documents, chunking, embeddings, prompt instructions, or model behavior affects the user experience. Treat the RAG system as an evaluated application rather than a one-time prompt experiment.
Use Unity Catalog and the surrounding Databricks governance controls to define which data the application can use, who can access it, and which outputs or traces should be retained. For a broader control review, see the Databricks AI Governance Checklist.
What to track
Track the variables that can explain a quality change. A single thumbs-up rate or average latency is not enough to diagnose a RAG system.
1. Source and chunk health
- Number of documents and chunks added, changed, or removed.
- Document version, source timestamp, owner, and access classification.
- Chunk size, overlap, separators, and metadata fields.
- Missing text, duplicate chunks, malformed records, and failed ingestion jobs.
- Index freshness compared with the source system.
Stable document and chunk identifiers are especially useful. They let you determine whether an answer changed because the source changed or because the retrieval and generation configuration changed.
2. Retrieval quality
Record the query, retrieved chunk identifiers, similarity or relevance signals where available, rank position, and the number of results passed to the model. Build a labeled evaluation set containing representative questions and the documents or facts expected to support each answer.
Useful retrieval checks include recall of expected evidence, relevance of the top results, duplicate-result rate, and the percentage of questions for which no adequate evidence was found. A response can be fluent and still be incorrect if the retriever supplied weak context.
3. Prompt and generation behavior
Version the system prompt, user prompt template, model identifier, model parameters, retrieved-context format, and citation instructions. A practical prompt should tell the model to use the supplied evidence, distinguish documented facts from uncertainty, and avoid inventing citations. It should also define the expected response format.
Track groundedness, answer correctness, citation accuracy, refusal or abstention behavior, and whether the answer actually addresses the question. For high-impact workflows, include human review instead of relying only on automated judges. The Databricks Model Evaluation Guide provides a useful framework for organizing metrics and test sets.
4. Operational behavior
- End-to-end latency and the time spent in retrieval, prompt construction, and generation.
- Error rates, timeouts, empty retrieval results, and malformed model responses.
- Token or request volume, where the selected model and platform expose those measures.
- User feedback segmented by question type, application version, and data source.
- Access denials and attempts to retrieve content outside the user’s permitted scope.
Keep evaluation data separate from production feedback when possible. Production interactions can reveal new failure modes, but they should be reviewed for sensitive information before being added to a reusable test set.
Cadence and checkpoints
A recurring review schedule makes model tuning deliberate rather than reactive. The exact frequency depends on how often the source data changes and how consequential the answers are.
Every run or deployment
Run ingestion validation, index freshness checks, permission tests, and a small smoke-test set. Confirm that expected document identifiers can be retrieved and that the application returns citations or evidence in the intended format. Compare the candidate configuration with a known baseline before promoting it.
Weekly or after a material data change
Review failed queries, empty retrievals, user corrections, and newly added document types. Add representative failures to the evaluation set only after removing unnecessary sensitive content and defining the expected behavior. Investigate whether the problem is caused by source quality, chunk boundaries, metadata filters, or the generation prompt.
Monthly
Run the full regression set and compare retrieval, groundedness, correctness, citation, latency, and failure metrics with the previous version. Review changes to the embedding model, foundation model, prompt template, chunking strategy, and Vector Search index. Check whether access rules and Unity Catalog permissions still match the application’s intended scope.
Quarterly or before a major release
Reassess the evaluation set itself. It should represent current user questions, difficult edge cases, common ambiguity, and important document categories. Revisit the model selection, retrieval strategy, monitoring coverage, rollback plan, and data retention approach. A quarterly review is also a reasonable time to identify obsolete documents and unused indexes.
How to interpret changes
When a score or user signal moves, avoid changing the prompt immediately. First isolate the layer responsible for the change.
- Retrieval falls while generation remains stable: inspect embeddings, chunking, metadata filters, index freshness, and query rewriting. Test whether expected evidence appears in the top results.
- Retrieval is strong but answers are incorrect: inspect context ordering, prompt instructions, model parameters, and the model’s ability to follow the required response format.
- Answers are grounded but incomplete: check whether relevant passages are split across chunks, whether the result limit is too small, or whether the prompt asks the model to cover every supported point.
- Citations are present but unreliable: require citations to reference retrieved chunk identifiers or document metadata, then test citation correctness separately from answer fluency.
- Latency increases: separate retrieval latency from generation latency. Check result count, document size, prompt length, retries, concurrency, and model configuration before reducing context blindly.
- Quality improves but cost or latency worsens: compare the tradeoff on question categories. A larger model or more context may be justified for difficult queries but unnecessary for routine ones.
Use a baseline and change one major variable at a time. For example, compare the current embedding configuration with a candidate configuration while holding the chunking method, retrieval settings, prompt, and generation model constant. Store the evaluation results with the configuration version so a successful change can be reproduced or rolled back.
Do not assume that replacing an embedding model is a drop-in update. Re-embed the relevant corpus, build or update the corresponding index, run retrieval regression tests, and compare results on short, long, ambiguous, and domain-specific queries. Likewise, changing the foundation model requires generation and safety checks even when retrieval is unchanged. The Databricks Mosaic AI RAG tutorial can serve as a companion implementation reference.
When to revisit
Revisit this Databricks RAG tutorial and your implementation at least monthly for active applications and quarterly for relatively stable ones. Return sooner when a recurring data point changes: a new document source is added, the corpus grows substantially, permissions change, users report unsupported answers, latency crosses an internal target, or a model or embedding configuration is replaced.
Use this practical update sequence:
- Record the trigger and freeze the current configuration as a comparison baseline.
- Run ingestion, retrieval, generation, citation, access, and operational checks.
- Classify failures by source, retrieval, prompt, model, permissions, or infrastructure.
- Change the smallest relevant component and rerun the same evaluation set.
- Review representative outputs manually, especially failures that automated metrics may miss.
- Document the decision, configuration versions, known limitations, and rollback path.
- Schedule the next review and add confirmed failure cases to the maintained test set.
For deployment and monitoring concerns beyond the RAG pipeline, consult the Databricks Model Serving Guide. A RAG application remains dependable when its evidence, prompts, models, permissions, and evaluation data are treated as versioned parts of one system.
