Ask a Claude-backed support bot whether the Pro plan includes SSO and it says yes, with a citation. The citation is accurate. The quoted line comes from the Enterprise column of a pricing table, and a 512-token chunk boundary sliced off the header that said so. Claude did its job: it quoted exactly what it was given. The retriever gave it the wrong page.

Almost every confident wrong answer from a Claude RAG system looks like that, which is why this guide spends one section on the model and four on what the model reads. This is the build we'd defend in code review: embeddings chosen on purpose, chunks that carry their own context, hybrid search with reranking, a prompt built for quoting, and an eval set that tells you which layer broke.

Claude doesn't do embeddings, so choose them deliberately

Anthropic doesn't ship an embedding model; its docs point to Voyage AI. OpenAI's text-embedding-3, Cohere and self-hosted models like BGE also work. The choice is sticky. The embedding model defines what "similar" means, and swapping it later means re-embedding every chunk and rebuilding the index. Test two candidates on your own questions before committing, check recall on mixed-language documents if your corpus has them, and pin the exact model version in config.

Start with Postgres and pgvector. With an HNSW index it serves a few million chunks at latencies most products never notice, and chunks, metadata and access rules sit in one database you already back up, so a permissions filter is a WHERE clause instead of a sync job. Move to a dedicated vector engine when you've measured a latency or scale problem, not because an architecture diagram had one. If you're starting from zero, our walkthrough of building a simple RAG system covers the minimal version first.

Give every chunk its context back

Chunking strips meaning. "The limit was raised to the higher tier in Q3" is useless alone: which limit, which product, which year? Split a 40-page policy into fixed 512-token windows and you get hundreds of chunks like that. The vector encodes "limit, raised, tier" and nothing about the source, so it ranks alongside every other limit in the corpus.

Anthropic's fix, contextual retrieval, is worth copying exactly. At ingest, send each chunk to Claude with its full source document and ask for 50 to 100 tokens placing it: which document, which section, what the chunk covers. Prepend that, then embed and keyword-index the combined text. Run it on Haiku with prompt caching so each document is billed once, not once per chunk; Anthropic put the cost at roughly $1.02 per million document tokens. In their tests, contextual embeddings alone cut top-20 retrieval failures by 35%, a one-off ingest cost that moves answer quality more than upgrading the answering model.

Search two ways, then rerank

Embeddings handle paraphrase and fumble exact strings. Ask about error E4012 and vector search returns chunks about E4011 and E4102, because they look alike. Run BM25 beside it, Postgres full-text search on a tsvector column is enough, and merge both lists with reciprocal rank fusion, scoring each chunk 1/(60 + rank) per list. It's a few dozen lines of SQL, and it catches the SKU, error-code and clause-number queries where one wrong answer loses a user for good. Adding contextual BM25 took Anthropic's failure reduction from 35% to 49%.

Then rerank. Pull 100 to 150 candidates from the fused list, score each against the question with a cross-encoder such as Cohere Rerank or Voyage's rerankers, and send Claude the top 10 to 20. A cross-encoder reads query and chunk together, so it spots a chunk that names the right product but answers a different question. It's too slow to run over the whole index, which is why it only sees candidates. With reranking added, Anthropic's failure reduction reached 67%. Our vector search work is mostly this layer, because it's where relevance is won or lost.

Shape the prompt for quoting, not guessing

Put retrieved documents at the top of the prompt, each in its own <document> block with a source ID and title, and the question last; Anthropic reports up to 30% better responses with the query after long context. Tell Claude to answer only from those documents and to say "not in the sources" when they don't cover it. That permission matters: a model with an approved way out stops stitching adjacent facts into a plausible answer.

Use the API's citations feature, not "list your sources" in the prompt. Pass chunks as document content blocks with citations enabled, and the response returns cited_text spans pointing at the exact passages used. Your UI can link each sentence to its source, and your logs show which of the 20 chunks Claude relied on. Right chunk retrieved but never cited is a prompt problem; never retrieved is a search problem. We cover that split in RAG that doesn't hallucinate.

Measure retrieval before you tune anything

Write 50 to 100 real questions from support tickets, sales calls or whoever answers them by hand today, each tagged with the chunk IDs that should answer it. Score retrieval alone first: recall@20, the share of questions whose correct chunk made the top 20. A question that fails there can't be fixed with system-prompt edits, and knowing that saves a week of rewording.

Grade answers separately, faithfulness to cited chunks before tone, and rerun the full set in CI on every change to chunk size, embedding model, reranker or prompt. Testing LLM code covers keeping those runs cheap enough that nobody skips them. When recall and faithfulness hold on your real documents instead of a demo folder, you have a system. If you'd rather not build that harness yourself, it's a large part of what our RAG development engagements deliver.

Questions, answered.

Can Claude create embeddings for a RAG system?

No. Anthropic has no embedding model of its own and points to Voyage AI, though OpenAI, Cohere or a self-hosted model like BGE also work. Claude writes chunk context and answers questions; something else produces the vectors.

Which Claude model should I use for RAG?

Haiku for high-volume ingest work like writing chunk context, where you may call it thousands of times per document set. A stronger model for final answers that reason across several sources. Run both against your eval set and pick on measured faithfulness, not price alone.

Do I still need RAG if Claude has a large context window?

Under a few hundred pages, putting everything in the prompt with prompt caching can beat a retrieval pipeline. Past that, or once per-query cost and latency matter, retrieval keeps each request small, fast and grounded in the passages that matter.

How do I stop a Claude RAG system from hallucinating?

Get the right chunks into the prompt with hybrid search and reranking, tell Claude to answer only from the provided documents, and give it explicit permission to say the answer isn't there. Enable citations so every claim maps to a passage you can check.

What is contextual retrieval?

A technique Anthropic published where each chunk gets a 50 to 100 token, model-written note on where it sits in its source document, prepended before embedding and keyword indexing. In Anthropic's tests it cut top-20 retrieval failures by 35%, and by 67% combined with BM25 and reranking.