Retrieval-augmented generation is easy to demo and hard to trust. You embed some documents, drop them in a vector store, stuff the top matches into a prompt, and the model answers. It works on the five questions you tested. Then a user asks something slightly off-distribution and the model fills the gap with a fluent, well-formatted lie.
Hallucination in RAG almost never comes from the language model being bad. It comes from the retrieval half being wrong and the generation half having no permission to say so. Fix those two things and most of the problem disappears. Here is how we approach RAG development when the output actually has to be correct.
Chunk on structure, not on character count
The default advice is to split documents into fixed windows of, say, 500 tokens with some overlap. This is the single biggest source of bad retrieval. A fixed window cuts a table in half, separates a heading from the paragraph it introduces, and strands a definition three chunks away from the term it defines. The embedding of a half-thought retrieves poorly because it means half a thing.
Chunk on the document's own structure instead. Split on headings, list items, table rows, and function or section boundaries. A chunk should be one coherent unit of meaning: a complete clause of a contract, a full API endpoint description, a whole troubleshooting step. Carry the surrounding context with it — prepend the section title and any parent headings so the chunk knows where it lives. When a chunk is self-contained, its embedding is honest, and honest embeddings retrieve the right thing.
For structured or semi-structured sources, keep metadata alongside the text: source document, section path, last-updated date, access level. You will use every one of those fields later for filtering, citation, and freshness checks.
Retrieve wide, then rerank
Vector similarity is a coarse instrument. Cosine distance between embeddings gets you in the neighborhood, but the top result by raw similarity is frequently not the most relevant one — it is just the one that happened to share the most surface vocabulary. Pulling only the top three chunks and trusting them is how you miss the paragraph that actually answers the question.
Retrieve wide — pull the top twenty or thirty candidates — then run them through a reranker: a cross-encoder that scores each chunk against the query directly rather than comparing pre-computed vectors. Reranking is the highest-leverage upgrade in most RAG systems and it is criminally underused. It reorders a mediocre candidate set into a good one, and it gives you a real relevance score per chunk, which you need for the next step. Pair dense vector search with a keyword search like BM25 too; hybrid retrieval catches the exact-match cases — error codes, product SKUs, proper nouns — that embeddings routinely fumble.
Gate on confidence, and let the model refuse
Here is the part almost everyone skips. After reranking, look at the top relevance score. If nothing crosses a threshold you have calibrated, the correct answer is not a best guess — it is 'I don't have that information.' A RAG system that can refuse is worth more than one that always answers, because a wrong answer costs more than no answer.
Put the gate in two places. First in retrieval: if the best reranked chunk scores below your bar, short-circuit and return a refusal before you ever call the model. Second in the prompt: instruct the model explicitly that it may only answer from the provided context, and that if the context does not contain the answer it must say so. Models follow this instruction far better than people expect — but only when the retrieval gate has already stopped feeding them irrelevant chunks that they feel obligated to use.
Calibrate the threshold on real queries, including ones you know the corpus can't answer. The goal is not zero refusals. The goal is that when the system does answer, it is almost always right.
Cite everything, and make citations checkable
Every claim in the output should point back to the chunk it came from. This is not decoration. Citations are how you verify the system in testing, how users build trust, and how you catch the model when it drifts from the source. Because you kept metadata on every chunk, a citation can name the source document and section precisely.
Go one step further and validate that the cited chunk actually supports the sentence it is attached to — a second, cheap model call or a string-overlap check will catch the cases where the model cites a real chunk but says something the chunk doesn't. When citations are verifiable, hallucination stops being an invisible failure and becomes something you can measure and regression-test.
Measure retrieval separately from generation
When a RAG answer is wrong, you need to know which half broke. Build an evaluation set of real questions with known-good source passages, and score retrieval on its own: did the right chunk make it into the top results? A retrieval failure and a generation failure look identical in the final output but have completely different fixes. Teams that only eyeball final answers spend weeks tuning prompts when the actual problem was chunking the whole time.
Once retrieval is solid and the confidence gate is honest, generation quality mostly takes care of itself. The model was never the weak link.