The demo answers every question perfectly on the twelve PDFs you tested it with, so you point it at the client's actual 40,000 documents and ship it. Within a day someone asks about error code E-4021 and it confidently describes E-4102 instead, cites a page that doesn't contain the claim, and a compliance reviewer flags it. Nothing in the code changed — the volume did, and volume is exactly what the tutorial you copied never had to survive. Local RAG means the embeddings, the vector store, and usually the model live on hardware you control, and people build it for two honest reasons: the documents are sensitive enough that a hosted API is a non-starter, or a cloud vector database sent them a $4,000 monthly bill and they did the arithmetic.
The mechanics are simple, which is the trap: chunk the documents, embed the chunks, store the vectors, embed the question, pull the nearest neighbors, stuff them into a prompt. Forty lines, works on twelve documents, demos beautifully. At 40,000 documents the answers go vague, the wrong passages surface, and the model narrates things that were never in the retrieved context. Every hard problem lives in the parts the forty-line version skips.
The stack that actually works locally
Use sentence-transformers with bge-small or bge-base for embeddings — both run on CPU at a few hundred chunks a second and beat the all-MiniLM everyone still copy-pastes from 2021 by a wide margin on retrieval benchmarks. For storage, start with local Chroma or a plain SQLite table plus sqlite-vec, and only move to Qdrant once you actually need payload filtering and indexed metadata. For generation, Ollama running a quantized Qwen or Llama is the least-effort path: a 7B–8B model at 4-bit fits in roughly 6GB of VRAM and answers grounded questions well enough that the generator is almost never your bottleneck.
The reflex is to run the biggest model your GPU holds. Resist it — retrieval quality dominates. A 70B model handed the wrong three chunks gives you a fluent, well-formatted wrong answer; a 7B model handed the right three chunks gives you a correct one. Get the right chunks in front of the model and upgrading the generator later is a one-line model swap.
Chunking is where most systems quietly fail
Every framework defaults to splitting text into fixed windows of N tokens with overlap. That is fine for flowing prose and destructive for real documents. A 512-token window slices a table across the boundary, strips a heading from the paragraph it introduces, and severs clause (c) from the list it belongs to. Retrieval then surfaces the orphaned half, the model gets a fragment with no anchor, and it fills the hole by guessing. You cannot prompt your way out of that — the damage was baked in at ingest.
Chunk on structure, not token count. Split on headings, keep tables and code blocks whole, and treat a section as the unit even when it runs long. Attach metadata to every chunk at write time — source file, page, section title, last-modified date — because you will filter and cite on all four later, and bolting metadata on after you've embedded 40,000 chunks means re-embedding all 40,000. A weekend on the chunker buys more accuracy than a month of swapping models.
Pure vector search is not enough
Embedding similarity nails semantic closeness and fumbles exact tokens. Ask for error code E-4021 or part number 7815-B and vector search cheerfully returns chunks about similar-sounding errors, because the embedding never learned that the literal string is the whole point. Run BM25 keyword search and vector search in parallel and fuse the two ranked lists with reciprocal rank fusion — a dozen lines. Exact identifiers, proper nouns, and rare terms stop vanishing.
Then add a reranker; this is the single largest quality jump most local systems never make. Retrieve 20 candidates cheaply, run a cross-encoder like bge-reranker over each question-and-chunk pair, reorder, and keep the top 4. The cross-encoder reads the question against each chunk instead of comparing two frozen vectors, so it catches relevance the embeddings miss entirely. We had a system cross 400,000 chunks and start returning mush; the fix was not a bigger model, it was two lines of reranking. It runs on CPU in tens of milliseconds and it is the line between a demo and a tool.
Grounding, citations, and knowing when to say no
A RAG system that never says 'the documents don't cover this' is a liability, because the moments it doesn't know are precisely the moments it invents. Instruct the model to answer only from the provided context and to name the gap when the answer isn't there — then verify it, by checking that each sentence of the answer maps back to a retrieved chunk before you show it. Return a citation with every claim: source document and section. Citations are how a user catches a wrong answer in two seconds instead of carrying it into a decision, and in a regulated environment they are frequently the feature that makes deployment allowable at all.
Measure retrieval before you measure answers. Build 30–50 real questions with their known correct source passages and track how often the right chunk lands in your top results — recall@k, nothing fancier. If the right chunk never gets retrieved, no downstream prompt engineering can rescue the answer. Standing this up on your own hardware is very doable, but the doable part is the plumbing; the value is the chunking, the hybrid retrieval, and the reranking the tutorials leave out.