Building Production RAG Systems: Chunking, Retrieval, Reranking, and Evaluation

What RAG Actually Is

Retrieval-augmented generation grounds an LLM's answer in documents you retrieve at query time, instead of relying only on what the model memorized during training. It reduces hallucination, lets you answer over private or fresh data, and gives you citations. It has become the default architecture for enterprise LLM applications.

The naive version — "embed the docs, embed the query, stuff the top 5 into the prompt" — demos beautifully and fails in production. This guide covers the pipeline that actually holds up.

The Production RAG Pipeline

Documents → Chunking → Embedding → Vector Index
                                        │
Query → Rewrite → Hybrid Retrieval → Rerank → Context Assembly → LLM → Answer + Citations

Each stage has failure modes. Most bad RAG systems are bad at retrieval, not generation — the model can only be as good as the context you hand it.

Stage 1: Chunking

Chunking decides what a "unit of retrieval" is. Get it wrong and everything downstream suffers.

• Fixed-size chunks (e.g. 512 tokens with 10–15% overlap) are the simple baseline. Overlap prevents answers from being split across a boundary.
• Structure-aware chunking splits on headings, paragraphs, or code blocks so chunks align with semantic units. Almost always beats fixed-size for real documents.
• Small-to-big / parent-document: embed small chunks for precise matching, but feed the LLM the larger parent section for context. A very effective pattern.

Rules of thumb: smaller chunks improve retrieval precision but lose context; larger chunks do the reverse. Start at ~300–500 tokens with structure awareness, then tune against your eval set. Preserve metadata (source, title, section) — you'll need it for filtering and citations.

Stage 2: Retrieval (Go Hybrid)

Dense vector search (from your embedding model, indexed with HNSW) captures semantic similarity — great for paraphrases and concepts. But it's weak on exact matches: product codes, error strings, rare names, acronyms.

Hybrid retrieval runs dense search and a keyword/lexical method (BM25) and fuses the results, typically with Reciprocal Rank Fusion. This consistently outperforms either alone because the two methods fail on different queries. If you build one thing beyond naive RAG, build hybrid retrieval.

Optionally add query rewriting: use an LLM to expand, decompose, or clarify the query before retrieval (e.g. turn a follow-up into a standalone question). Helps a lot in multi-turn chat.

Stage 3: Reranking

Retrieval optimizes for speed and casts a wide net — say the top 50 candidates. A reranker (a cross-encoder that reads the query and each candidate together) then re-scores those 50 and keeps the best 5.

Why two stages? Bi-encoders (your embedding model) encode query and document separately — fast, cacheable, but coarse. Cross-encoders are far more accurate but too slow to run over the whole corpus. So you retrieve wide with the bi-encoder and rerank narrow with the cross-encoder. This retrieve-then-rerank pattern is one of the highest-ROI upgrades in RAG. See late interaction methods like ColBERT for a middle ground.

Stage 4: Context Assembly

You have your top chunks — now build the prompt deliberately:

• Order matters. Models attend most to the start and end of long contexts ("lost in the middle"). Put the strongest evidence at the edges.
• Deduplicate near-identical chunks; they waste budget and bias the model.
• Include citations (source + section) so the model can attribute claims and you can verify them.
• Budget the context. More chunks isn't better — irrelevant context actively degrades answers. Fewer, higher-precision chunks usually win.

Stage 5: Evaluation — The Part Everyone Skips

You cannot improve what you don't measure, and "it looks good in the demo" is not measurement. Evaluate the two halves separately:

Retrieval quality (does the right context get retrieved?)

• Context recall: did we retrieve the chunks needed to answer?
• Context precision: how much of what we retrieved was relevant?
• Build a gold set of question → relevant-chunk pairs, even 50–100 examples.

Generation quality (given the context, is the answer good?)

• Faithfulness: is every claim supported by the retrieved context? (catches hallucination)
• Answer relevance: does it actually address the question?
• Use an LLM-as-judge for these at scale, calibrated against human labels.

The key diagnostic: if faithfulness is high but answers are wrong, your retrieval is failing. If retrieval is good but answers are wrong, your generation/prompt is failing. Measuring both halves tells you which knob to turn.

Common Failure Modes

Symptom	Likely cause
Right topic, wrong details	Chunks too large; add reranking
Misses exact terms/codes	Dense-only retrieval; add BM25/hybrid
Confidently wrong answers	Low faithfulness; tighten prompt, add "say I don't know"
Good chunks, bad answers	Context ordering / too many chunks
Degrades over time	Stale index; set up re-embedding on updates

A Pragmatic Build Order

1. Start simple: structure-aware chunking + dense retrieval + top-5. Ship to an eval set, not users.
2. Add hybrid retrieval. Measure the lift.
3. Add reranking. Measure again.
4. Tune chunk size and context assembly against your metrics.
5. Add query rewriting only if multi-turn or complex queries demand it.

Each step is guarded by evaluation, so you know what actually helped.

Key Takeaways

1. Most RAG failures are retrieval failures — invest there first.
2. Hybrid retrieval (dense + BM25) and reranking are the two highest-ROI upgrades.
3. Chunking is a real design decision; structure-aware + small-to-big beats fixed-size.
4. More context is not better; precision and ordering beat volume.
5. Evaluate retrieval and generation separately, or you'll tune the wrong stage.

---

Related: Vector Databases in Production, GraphRAG, and How to Evaluate LLM Applications.