RAG in production: the decisions that matter after the demo

Running RAG in production means owning far more than a prompt and a vector database. A retrieval-augmented generation system that holds up for real users depends on how documents are parsed, how they are chunked, how search combines exact and semantic matching, whether permissions are enforced at retrieval time, how answers cite their sources, how the index stays fresh, and how quality is measured on every change. The demo proves the idea; these decisions decide whether it survives its first quarter.

What RAG is, and what it is not

Retrieval-augmented generation was introduced by Lewis et al. in "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (2020), which combined a pre-trained generator with a retriever over a dense vector index of Wikipedia. The enterprise pattern keeps the idea and changes almost everything else: the corpus is your documents, the retriever is usually a search stack rather than a single dense index, and the generator is a hosted or self-hosted large language model given the retrieved passages in its prompt.

The core promise is that the model answers from your material rather than from its training data, and can point to where the answer came from. Every production decision below is in service of keeping that promise.

Ingestion: the step everyone underestimates

Retrieval quality is capped by parsing quality. Enterprise corpora are PDFs exported from scanners, slide decks, spreadsheets, wiki pages with navigation chrome, and contracts with numbered clauses that refer to other clauses. If the parser flattens a table into a stream of numbers or drops the heading a paragraph sits under, no embedding model will recover it.

• Preserve structure. Keep headings, section numbers, table rows and list membership as metadata, not just as text.
• Treat tables and scanned pages as separate pipelines. They need OCR or table extraction, and they need their own spot checks.
• Record provenance on every chunk: source system, document id, version, page or section, and the access-control list it inherits.
• Look at the output. Sampling parsed chunks by eye is the single cheapest quality step in the whole pipeline.

Which RAG chunking strategy should you use?

Fixed-size chunks with overlap are a reasonable baseline and the wrong default for structured documents. Better options, roughly in order of effort:

1. Structure-aware chunking — split on headings, sections and clauses so a chunk is a unit a human would cite.
2. Parent-child retrieval — index small chunks for precise matching, but hand the model the larger parent section so it sees enough context.
3. Context enrichment — prepend each chunk with a short description of the document and section it came from. Anthropic's write-up on contextual retrieval reports that adding generated context to chunks before embedding and lexical indexing reduced retrieval failures on their tests, with a further reduction when reranking was added.

Do not choose by intuition. Change one variable at a time and measure retrieval recall on your evaluation set; chunking is the kind of decision where a plausible change quietly makes things worse.

Hybrid search and reranking

Pure vector search is good at paraphrase and poor at exact identifiers: product codes, clause numbers, error messages, people's names. Lexical search such as BM25 has the opposite profile. Production systems generally run both and merge the results. A simple, robust way to merge is reciprocal rank fusion, described by Cormack, Clarke and Büttcher in "Reciprocal rank fusion outperforms Condorcet and individual rank learning methods" (SIGIR 2009), which combines rankings without needing their scores to be comparable.

After fusion, a cross-encoder reranker that reads the query and each candidate together can reorder the top results more accurately than either first-stage retriever. It costs latency, so rerank a short candidate list rather than everything.

Permissions belong in retrieval, not in the prompt

The most serious production failure in enterprise RAG is not a wrong answer; it is a correct answer drawn from a document the user was never allowed to read. Telling the model "do not reveal confidential information" is not a control. Enforce access at retrieval time:

• Carry each source document's access-control list into the index as filterable metadata.
• Filter by the requesting user's identity and groups inside the search query, before anything reaches the model.
• Re-sync permissions when they change in the source system, not only when the content changes.
• Test it: include questions in your evaluation set that a restricted user must not be able to answer.

The OWASP Top 10 for LLM Applications (2025) lists sensitive information disclosure and vector and embedding weaknesses among its risks; both are directly relevant here.

Citations users can check

Every answer should carry citations to the specific passages it used, with links back to the source document and section. Citations do three jobs: they let users verify, they let reviewers grade faithfulness quickly, and they make "I could not find this in the documents" a respectable answer rather than a failure. Validate citations in code — the cited chunk id must be one that was actually retrieved — rather than trusting the model to invent well-formed references.

Freshness and re-indexing

An index is a copy, and copies go stale. Decide explicitly, per source, how quickly a change must be reflected: minutes for a support knowledge base, a day for a policy library. Use incremental updates driven by change events or modified timestamps, handle deletions (the step most pipelines forget), and keep document versions so an answer can be traced to the version that was live when it was given. Re-embedding the whole corpus when you change embedding models is a planned migration, not a background job.

Context, latency and cost

More retrieved context is not always better. Liu et al. showed in "Lost in the Middle: How Language Models Use Long Contexts" (2023) that models tend to use information at the beginning or end of a long input better than information in the middle. In practice that argues for retrieving fewer, better-ranked passages and placing the strongest ones where the model will attend to them.

Set a budget per request — tokens in, tokens out, end-to-end latency — and design to it. Cache embeddings for repeated queries, keep the reranking stage short, and route simple questions to smaller models where your evaluation shows they are good enough.

How do you evaluate a RAG system?

Measure retrieval and generation separately. For retrieval: did a correct passage appear in the top k, and how highly was it ranked. For generation: is every claim supported by the retrieved context, and does the answer address the question. We cover metrics, LLM-as-a-judge and regression testing in detail in how to evaluate an LLM application before it reaches production. The short rule: no change to chunking, embeddings, retrieval or prompts ships without a run against the evaluation set.

When not to use RAG

RAG is the right tool when answers live in documents that change, must be cited, or are subject to access control. It is the wrong tool when:

• The task is about form, not facts — a consistent tone, a classification scheme, an output format. Prompting or fine-tuning fits better.
• The corpus is small and static enough to fit comfortably in the model's context window, and per-document permissions are not a concern. Passing the documents directly can be simpler, though cost and the position effects above still apply.
• The answer requires computation over structured data. Querying a database through a tool beats retrieving text about the database.

Operating it

Once live, a RAG system is a search product and should be run like one. Log every query with the retrieved chunks, scores, model version and final answer. Track queries with no good retrieval results — they are your content gaps. Give users a lightweight way to flag a bad answer, and route those flags into review and then into the evaluation set. Watch cost and latency per answer alongside quality, because the cheapest fix for a slow system is often a better retriever, not a bigger model.

Frequently asked questions

Do we need a dedicated vector database?

Not necessarily. Many relational and search databases now support vector indexes alongside lexical search. Choose based on your scale, your need for hybrid search and filtering, and what your team already operates well.

Should we fine-tune instead of using RAG?

Fine-tuning changes how a model behaves; it is a poor way to teach it facts that change or must be cited. The two are complementary, and most enterprise question-answering starts with retrieval.

How do we stop the model answering from outside the documents?

Instruct it to answer only from the provided context, give it an explicit "not found" path, validate citations in code, and measure faithfulness on every release. Instructions alone are not enough; measurement is what tells you it worked.

KnowledgeBase, our retrieval product with citations and access control, packages many of these decisions. Where a custom system is the better fit, our AI and LLM Engineering team can start with a diagnosis of your corpus and questions — book a discovery call.