Building Reliable RAG Pipelines: Data Freshness, Access Control, and Evaluation

A retrieval-augmented generation (RAG) application does not become reliable when documents are simply embedded. Reliability depends on whether the right version of an allowed source reaches the right user, and whether the application can recognize when it has no good answer. This guide focuses on that operational pipeline, not a catalog of vector databases.

Start with a concrete question set

Suppose a support specialist asks, “Can this customer renew under the old terms?” The answer may involve a versioned policy document, an account record, and an exception that requires legal review. Collect real, de-identified questions and mark expected sources, who may see each source, acceptable staleness, and when the system must decline or escalate.

The pipeline: source to answer

The diagram below shows the document-update path and the user-query path meeting at permission-aware retrieval. Keep the original document ID, version, owner, last-updated time, and deletion state attached to every indexed chunk. A revoked or deleted source must stop appearing in answers, even while an index update is pending.

Databricks describes RAG as retrieval, augmentation, and generation, with data pipeline, evaluation, monitoring, and governance as separate components. That split is useful because an answer can fail before the model sees the prompt.

Ingestion: make change and deletion paths explicit

For each source, choose a schedule or event trigger. Detect new records, changed versions, removed records, and revoked access. Reindex only affected material where practical, but make the reprocessing path replayable. Publish a freshness signal for each source so the application can warn or pause when a feed has not updated on time.

Preserve document structure when chunking: headings, table context, effective dates, and the title can determine whether a fragment is usable. A perfect embedding of a sentence stripped of its policy date is still an unsafe source for a time-sensitive answer.

Retrieval: filters first, then relevance

Apply tenant and user access constraints before the model can read retrieved text. Test whether ACL changes propagate to the index and what happens during the gap. If a record cannot be confidently authorized, leave it out. Audit the retrieved source IDs and the policy decision without recording unnecessary personal content.

Do not assume semantic similarity solves exact-match tasks. Microsoft documents keyword, vector, and hybrid search approaches, as well as filtering and reranking. Compare them against your question set: an invoice number, an unusual product code, and a policy paraphrase may each favor a different retrieval strategy.

Generation: give the model a contract

Pass only authorized passages with source metadata. Require the answer to distinguish source-backed statements from uncertainty and link to the underlying material. For a transactional fact such as current balance, call the authorized operational API instead of trusting a stale document chunk. Route disputes and missing evidence to a human.

Evaluation: separate retrieval from prose

Create a test matrix: known answer, ambiguous policy, stale version, deleted source, revoked user, cross-tenant query, prompt injection inside a document, and no matching source. Score whether the correct authorized evidence was retrieved before judging answer quality. Track citation correctness, unsupported claims, abstention, p95 latency, and cost against a baseline.

Databricks notes that formatting changes can affect retrieved chunks and recommends evaluating individual RAG components as well as the full application. Re-run the question set when chunking, permissions, source content, embeddings, or ranking change.

A minimal rollout

Start with one document collection, one user group, and a small set of real questions. Establish who owns the content, how quickly edits and revocations must appear, and who reviews failures. Only then add more sources or a more elaborate search architecture. The broad platform decisions belong in the modern AI data stack design, but the first proof is whether this one retrieval path can be trusted.

For help connecting source systems and setting up governed datasets, explore Tevpro’s Data Solutions service. For an AI application built on those data contracts, see Artificial Intelligence Solutions.