How to Build a RAG System from Scratch That Handles Real‑World Data

Most RAG demos look impressive because the corpus is a handful of PDFs. When you point the same pipeline at half a million product manuals, the latency spikes, the relevance drops, and the LLM starts hallucinating. The root cause isn’t the model size; it’s the retrieval layer and the way you stitch the results together. In this article I walk through the exact stack we use at Duskel, the failure modes we hit, and the concrete fixes that turned a flaky proof‑of‑concept into a production service. I’ll assume you have a basic LLM API key and a modest cloud budget. The goal is to end up with a system that can answer user queries over a large, heterogeneous document set, keep latency under 500 ms, and let you add new docs without re‑indexing the whole corpus. If you’re looking for a glossy overview, skip ahead – this is the nitty‑gritty you need to ship.

Choose the Right Vector Store and Tune Index Parameters

We started with an off‑the‑shelf FAISS index because it’s easy to spin up. The moment we crossed 100 k vectors the index size ballooned and search time doubled. The fix was to switch to a hybrid architecture: use a managed Milvus cluster for storage and an HNSW index with an M‑parameter tuned to 32. This reduced average query time from 1.2 s to 380 ms on a 500 k‑vector collection. A common failure mode is ignoring the dimensionality of the embedding model. We tried a 1536‑dim sentence‑transformer and hit out‑of‑memory errors on the node. The remedy was to switch to a 768‑dim model (e.g., all‑MiniLM‑L6‑v2) and enable IVF‑PQ compression in Milvus. The trade‑off is a 0.3 % drop in cosine similarity, but the memory savings let us stay under our budget and keep the index in RAM for hot queries.

Rerank, Don’t Just Retrieve

Our first production run returned the top‑10 nearest vectors and fed them directly to the LLM. The answers were often correct but occasionally drifted because the raw similarity score doesn’t capture query intent. Adding a lightweight cross‑encoder reranker (a 2‑layer BERT fine‑tuned on a few hundred relevance labels) cut the hallucination rate from 12 % to 3 %. The reranker adds about 70 ms per query, which is acceptable once you’ve shaved most of the retrieval time. The key is to keep the reranker small and batch it with the LLM call. If you try to run a full‑size cross‑encoder you’ll blow the latency budget and defeat the purpose of a RAG system.

Chunking Strategy and Metadata Enrichment

We originally split PDFs by page, which gave us chunks of 500‑800 tokens. Long chunks cause the LLM to truncate the context and short chunks explode the index size. The sweet spot we found is 200‑300 token chunks with an overlap of 50 tokens. This balances relevance and index size. Metadata is the unsung hero of retrieval. Adding fields like source‑type, publication‑date, and a confidence tag lets you filter results post‑search. In one client we discovered that queries about “2022 safety guidelines” kept surfacing 2018 docs because the filter was missing. Adding a date filter in the Milvus query reduced irrelevant hits by 40 % without any extra model work.

Orchestrate the Pipeline with a Simple MCP Server

All the moving parts—embedding service, vector store, reranker, LLM—need a reliable glue. We built a tiny MCP (Message‑Controlled Protocol) server in FastAPI that exposes a single `/query` endpoint. The server validates payloads, runs the embed‑retrieve‑rerank sequence, and streams the final answer. A failure mode we hit early was token leakage: the server inadvertently passed the entire retrieved chunk list to the LLM, blowing the context window. The fix was to truncate the concatenated chunks to the model’s max tokens minus a safety margin, and to prepend a short system prompt that tells the model to cite the source IDs. This also gave us a clean audit trail for compliance.

Ready to turn your document lake into a reliable RAG service? Let’s talk.

Send the problem. You get one fixed number and a plan back within a business day.