RAG in Production: What Actually Breaks

Retrieval-augmented generation demos well and fails quietly. A weekend prototype that answers questions over a folder of PDFs looks finished. The same system in front of real users, with real documents and real stakes, behaves differently — and the gap between the two is where most RAG projects stall.

What actually breaks

The failure is rarely the model. It is almost always retrieval. If the right passage never reaches the context window, no amount of prompt engineering recovers it. The model will answer anyway, confidently, from whatever it did receive. That is the dangerous part: a retrieval failure looks exactly like a correct answer until someone checks.

So the first thing to build is not the generation step. It is a way to measure whether retrieval is working at all.

Chunking decides your ceiling

Chunking is the least glamorous decision in the pipeline and the one that most constrains quality. Split too small and a chunk loses the context that made it meaningful. Split too large and the embedding becomes an average of several ideas, matching everything weakly and nothing strongly.

Fixed-size chunking with overlap is the common default because it is trivial to implement. It is also the reason many systems underperform: it cuts through the middle of tables, splits a heading from the paragraph it introduces, and severs list items from their stem.

Structure-aware chunking — splitting on document structure rather than character count — is more work and usually worth it. Keep a heading with its section. Keep a table intact or convert it to prose before embedding. Attach breadcrumbs so a retrieved chunk carries its own context: the document title, the section it came from, the date it was written.

Hybrid retrieval, not pure vectors

Vector search is good at meaning and bad at specifics. Ask about an exact error code, a part number, a surname, a version string, and semantic similarity will happily return passages that are about the same topic but contain a different identifier.

Keyword search has the opposite profile. It nails exact tokens and misses paraphrase entirely.

Running both and fusing the results covers more ground than either alone. Reciprocal rank fusion is a reasonable default: it needs no score calibration between the two systems, which is useful because their scores are not comparable in any meaningful way.

Then rerank. A cross-encoder over the top candidates is slower per item but operates on the query and passage together rather than on two independent embeddings, and it consistently promotes better passages into the final context. Retrieve broadly, rerank aggressively, pass few.

Measure retrieval separately from generation

Build a small evaluation set before building the product: questions, and for each, the passages that should be retrieved. A few dozen is enough to start. Then track recall at k — of the passages that should have been found, how many were?

This number tells you something the end-to-end output never will. If recall at 10 is poor, the answer quality problem is not a prompting problem, and time spent rewriting the system prompt is wasted.

Only once retrieval is sound does it make sense to grade the generated answer — for faithfulness to the retrieved context, and for whether it actually addressed the question.

Citations are a feature, not decoration

Every claim the system makes should be traceable to a retrieved passage. This is partly about trust: users who can check are more willing to rely on the system. It is also the cheapest debugging tool you will have. When an answer is wrong, the citation shows immediately whether retrieval failed or generation did.

If a model cannot cite a passage for a claim, that claim should not appear. Enforcing this in the output format catches a meaningful share of fabrication before it reaches anyone.

What it costs to run

Cost scales with retrieved context, not with the question. A system that retrieves twenty passages to answer a one-line query is paying for twenty passages on every call. Reranking lets you retrieve widely and send narrowly, which is usually the single biggest lever on cost.

Cache aggressively. Identical and near-identical queries are far more common than people expect, particularly in support and internal-knowledge settings. An embedding cache and a response cache keyed on normalised queries pay for themselves quickly.

Set a hard ceiling per request and per user per day. Without one, a loop in an agent or a pathological document will produce a bill nobody authorised.

Where the data boundary sits

RAG systems tend to accumulate permissions problems. If the index contains documents from several access levels, retrieval must filter by the requesting user's permissions before the model sees anything. Filtering after generation is not filtering — the content already left the boundary.

This is easier to design in at the start than to retrofit. Store the access scope alongside each chunk and make it part of the query, not a post-processing step.

The honest summary

A production RAG system is mostly a search system with a language model attached to the end. Teams that treat it that way — investing in chunking, hybrid retrieval, reranking and measurement — get systems that hold up. Teams that treat it as a prompting exercise get demos.