At 9:12 on a Monday, a support assistant answers a refund question by citing a policy page that was replaced 14 months ago. The answer reads cleanly, includes a link, and is wrong. That is the standard RAG failure: retrieval returns something plausible enough for the model to turn into confident prose.

A RAG system is not a chat interface with a vector database behind it. It is an evidence-selection pipeline. Every answer depends on four decisions made before generation begins: which documents entered the corpus, how their text was split, which records were eligible for this user and query, and whether the retrieved passages actually support the claim. Get one wrong and a larger model produces a more articulate error.

Start with a question set

Collect 50 to 100 questions from support tickets, sales-call notes, internal search logs, and the Slack threads where subject-matter experts repeat themselves. For each question, record the approved answer, the exact supporting section, document version, and whether an answer is allowed. This creates a retrieval benchmark before you choose an embedding model, chunk size, or database vendor.

Reserve at least 20% of the set for refusal cases. Include questions about customer-specific pricing, unpublished roadmap items, expired policies, competing products, and facts absent from the corpus. A system that answers every question has no boundary; it is guessing when its index is silent.

Score retrieval separately from answer quality. Measure Recall@10: did one of the top 10 results contain the required evidence? Then measure citation faithfulness: does the cited text entail the answer? These two numbers expose different failures. Low recall points to corpus coverage, filters, chunking, or search. High recall with weak answers points to prompting, context assembly, or citation validation.

Make documents retrievable

Ingestion fails long before embeddings do. PDF extraction commonly duplicates page headers in every chunk, reads two-column layouts across the page instead of down the column, drops table headers, and turns scanned documents into empty strings. Run extraction checks: compare page count, character count, detected headings, and sampled rendered pages against the original file. A 40-page handbook reduced to 900 characters is not a search problem.

Store each chunk with a stable document ID, source URL or file ID, title, heading path, page or section locator, publication date, effective date, product version, tenant, access scope, and content hash. The hash lets you identify duplicates and skip unchanged documents during re-indexing. Without version metadata, search will eventually mix superseded instructions with current ones.

Split on document structure first: heading, subsection, list, table, code block, or FAQ entry. Keep a heading with the material it governs. Start testing around 300 to 600 tokens with 50 to 100 tokens of overlap, then inspect failures rather than treating those values as doctrine. Never split a numbered exception from its condition or a table row from its column headings; that failure can invert a policy answer. For large or rapidly changing corpora, vector search engineering is where these ingestion rules, metadata contracts, and re-indexing jobs belong.

Retrieve before you generate

Apply hard filters before ranking whenever the query supplies a constraint. Tenant ID, user permission, locale, product edition, and effective date are eligibility rules, not hints for an embedding model. A user asking about version 4 must not receive version 2 documentation because both pages describe installation.

Retrieve 20 to 50 candidates, then test hybrid search: lexical retrieval for error code E1042, SKU names, clause numbers, and exact API fields; semantic retrieval for ordinary-language paraphrases. Fuse the result sets and rerank the top candidates with a model that reads the query and passage together. Reranking fixes ordering errors. It cannot retrieve a document that never entered the index or repair text mangled during extraction.

Pass only the strongest, non-duplicate evidence into the generation context. Show source title, section, and link beside the answer. Do not treat a citation as proof: a common failure is citation laundering, where the model cites a nearby paragraph that sounds related but does not support its specific claim. Test whether each answer sentence maps to an excerpt, not merely whether the response contains a link.

Build refusal into the product

Use a three-way decision instead of a single answer path: answer when evidence clears a relevance and support threshold; ask a clarifying question when the requested version, account, or product is ambiguous; refuse when no approved evidence exists. Log which branch fired, the top retrieval score, the selected chunks, and the final sources. Those records turn an incident into a diagnosable retrieval trace.

Prompting should state that supplied evidence is authoritative, unsupported claims must be omitted, and citations must follow the claims they support. Pair that instruction with product controls: minimum reranker score, document recency rules, source allowlists, and human review for legal, financial, medical, or account-specific requests. A support assistant may summarize the returns policy; it must not invent an exception for a customer.

Re-run the benchmark after every parser change, corpus update, embedding migration, ranking adjustment, or prompt revision. Track Recall@10, citation faithfulness, refusal precision, and latency at the 95th percentile. Keep failed traces in the test set until they pass for the right reason. If the product must create tickets, issue refunds, or inspect accounts after answering, isolate those actions from retrieval and use AI agents with explicit tool permissions and confirmation steps.

Questions, answered.

How do I set up a RAG system from scratch?

Begin with 50 to 100 real questions, their approved source passages, and refusal cases. Build structured ingestion and metadata first, test retrieval against that set, then add grounded generation and citation checks.

What database do I need for a RAG system?

Choose a store that supports vector search, keyword search or an adjacent lexical index, metadata filtering, access controls, and incremental updates. The deciding factors are corpus size, filter cardinality, update rate, and operational fit with your existing stack.

Why does my RAG system give wrong answers?

Inspect the retrieval trace before changing models. Frequent causes are stale documents, missing access or version filters, broken PDF extraction, chunks split across exceptions, duplicate results, and citations that do not entail the generated claim.

How do I stop a RAG system from hallucinating?

You cannot eliminate model error, but you can constrain it with approved-source retrieval, claim-level citations, relevance thresholds, refusal cases, and benchmark checks after every corpus or ranking change. Treat unsupported-question accuracy as a release metric.