RAG in production: seven lessons from shipping retrieval-augmented features

Retrieval-augmented generation looks simple in a demo: split some documents, embed them, find the nearest chunks to a question and hand them to a language model. It usually works well enough on the ten questions the team tried. Then real users arrive, ask things nobody anticipated, and the feature starts confidently citing the wrong policy document.

Over the past two years we have shipped RAG features into support tools, internal knowledge bases, contract review workflows and a clinical documentation product. This article collects the seven lessons that changed how we build them. Each one comes with the concrete pattern we now use by default, and the code samples are TypeScript because that is where most of our RAG services live.

1. Chunking is a product decision, not a preprocessing step

The most common failure we see is not a bad model or a bad vector database. It is chunks that do not contain a complete answer. A fixed 500-token window will happily split a table from its header, a clause from its definition or a step from the warning that precedes it.

What works better for us:

• Split on document structure first. Headings, sections, list items and table boundaries. Fall back to token windows only inside very long sections.
• Carry context into each chunk. Prefix every chunk with its document title and heading path, for example Employee Handbook > Leave > Parental leave. This helps both retrieval and the model's interpretation.
• Keep chunks between roughly 200 and 800 tokens, with a small overlap only when splitting prose mid-section.
• Store a pointer to the parent section. Retrieve on small chunks for precision, then expand to the parent section when building the prompt if the chunk is too thin.

type Chunk = {
  id: string;
  docId: string;
  headingPath: string[];
  text: string;
  parentSectionId: string;
  acl: string[]; // who is allowed to see this chunk
};

function toEmbeddingInput(c: Chunk, docTitle: string) {
  return `${docTitle} > ${c.headingPath.join(" > ")}\n\n${c.text}`;
}

We treat chunking changes like schema migrations: versioned, re-run in full and evaluated before they go live.

2. Use hybrid search, then rerank

Pure vector search is good at paraphrase and bad at exact matches. Users search for error codes, product SKUs, clause numbers and people's names, and embeddings blur those. Keyword search has the opposite profile. Combining them is cheap and consistently improves recall in our evaluations.

If you are already on Postgres, pgvector plus built-in full-text search gets you a long way. Reciprocal rank fusion (RRF) merges the two ranked lists without needing to calibrate their scores against each other.

WITH semantic AS (
  SELECT id, RANK() OVER (ORDER BY embedding <=> $1) AS rnk
  FROM chunks
  WHERE tenant_id = $3
  ORDER BY embedding <=> $1
  LIMIT 50
),
keyword AS (
  SELECT id, RANK() OVER (ORDER BY ts_rank_cd(tsv, q) DESC) AS rnk
  FROM chunks, websearch_to_tsquery('english', $2) AS q
  WHERE tenant_id = $3 AND tsv @@ q
  ORDER BY ts_rank_cd(tsv, q) DESC
  LIMIT 50
)
SELECT COALESCE(s.id, k.id) AS id,
       COALESCE(1.0 / (60 + s.rnk), 0) + COALESCE(1.0 / (60 + k.rnk), 0) AS score
FROM semantic s
FULL OUTER JOIN keyword k ON s.id = k.id
ORDER BY score DESC
LIMIT 20;

The constant 60 is the conventional RRF value; it rarely needs tuning.

After fusion we pass the top 20–50 candidates through a cross-encoder reranker and keep the best 5–8. Rerankers read the question and each chunk together, so they catch relevance that embedding similarity misses. On one internal knowledge base, adding a reranker moved the correct chunk into the top five for about 15% more of our test questions, at a cost of 80–150 ms per query.

3. Build the evaluation set before you tune anything

Without an evaluation set, every change is judged by whoever tried a few questions last. That is how teams end up oscillating between chunk sizes for weeks.

We start every RAG project by building a set of 100–300 real questions, each labelled with the document sections that contain the answer and, where practical, a reference answer. Sources that work well: support tickets, search logs, questions from subject-matter experts, and deliberately awkward cases (ambiguous questions, questions with no answer in the corpus, questions that span two documents).

We measure two layers separately:

Retrieval metrics are deterministic and cheap, so they run in CI on every change to chunking, embeddings or ranking. Generation metrics use a model as a judge with a written rubric, and we spot-check a sample by hand each release because judges drift too.

function recallAtK(results: string[][], expected: string[][], k: number) {
  let hits = 0;
  results.forEach((ids, i) => {
    if (ids.slice(0, k).some((id) => expected[i].includes(id))) hits++;
  });
  return hits / results.length;
}

The single most useful habit: when a user reports a bad answer, add that question to the evaluation set before fixing anything.

4. Make citations structural, not decorative

Users trust RAG features when they can check them. Citations also give you a lever against hallucination: if every claim must point at a source, unsupported claims become visible.

Rather than asking the model to write "[1]" in free text and hoping, we give each retrieved chunk a stable short ID in the prompt and ask for structured output that separates answer segments from their source IDs. The application then validates that every cited ID was actually in the context, and renders links to the original section.

type CitedAnswer = {
  segments: { text: string; sources: string[] }[];
  answerable: boolean;
};

function validateCitations(ans: CitedAnswer, contextIds: Set<string>) {
  const invalid = ans.segments
    .flatMap((s) => s.sources)
    .filter((id) => !contextIds.has(id));
  if (invalid.length) throw new Error(`Unknown sources: ${invalid.join(", ")}`);
  const uncited = ans.segments.filter((s) => s.sources.length === 0);
  return { ok: uncited.length === 0, uncited };
}

When validation fails we either retry once or fall back to showing the top sources with a short "here is what we found" message. Both are better than a fluent answer with invented references.

5. Budget latency and cost per stage

A typical request in our systems does query rewriting, embedding, hybrid retrieval, reranking and generation. Each stage adds latency and cost, and users feel the total. We set a budget per stage and measure it in production traces.

Practical savings we have seen:

• Streaming does not reduce total time but makes a 3-second answer feel acceptable.
• Prompt caching for the stable system prompt and instructions cuts input cost noticeably on providers that support it.
• Routing by difficulty: a small, fast model handles most questions, and only questions that the reranker scores as low-confidence or multi-document go to a larger model.
• Fewer, better chunks. Sending 5 reranked chunks instead of 20 raw ones usually improves answers and reduces cost at the same time.

6. Enforce permissions at retrieval time

If your corpus contains anything that not every user may see, access control must happen in the retrieval query, not in the prompt. Telling the model "do not reveal HR documents to non-managers" is not a control. Once a chunk is in the context window, assume it can end up in the answer.

We store an ACL on every chunk (see the acl field above), filter on tenant and permission groups inside the SQL or vector query, and write tests that assert a user from group A never retrieves a chunk restricted to group B. Keeping ACLs in sync with the source system (a document management system, a wiki, a shared drive) is its own job; we re-sync on change events where the source supports them and nightly as a backstop.

7. Treat retrieved content as untrusted input

Documents can contain instructions. A support ticket, a scraped web page or an uploaded PDF may include text like "ignore previous instructions and…". This is indirect prompt injection, and RAG systems are exposed to it by design.

Our baseline guardrails:

• Separate instructions from data. Put retrieved text in clearly delimited blocks and tell the model it is reference material, not instructions.
• Limit what the model can do. A read-only Q&A feature is low risk. If the same model can call tools (send emails, update records), require confirmation from the user for any side-effecting action.
• Filter at ingestion for obviously hostile content in user-generated sources, and log which chunks contributed to each answer so incidents can be traced.
• Handle "no answer" explicitly. A system that is allowed and encouraged to say it could not find an answer is safer and more trusted than one that always produces something.
• Check outputs for sensitive data patterns (keys, national ID numbers) before displaying them, especially in multi-tenant systems.

Summary checklist

Before a RAG feature goes to real users, we check:

• Chunks follow document structure and carry title and heading context.
• Retrieval is hybrid (vector plus keyword) with a reranker on top.
• An evaluation set of real questions exists, including unanswerable ones.
• Retrieval metrics run in CI; generation quality is judged and spot-checked each release.
• Citations are structured, validated against the context and link to sources.
• Each pipeline stage has a latency and cost budget visible in traces.
• Permissions are enforced in the retrieval query and covered by tests.
• Retrieved text is treated as untrusted, and side-effecting actions need confirmation.

None of this is exotic. It is ordinary engineering discipline applied to a system that happens to include a language model, and it is the difference between a demo and a feature people rely on.