how to build a rag system in python – a step‑by‑step guide that gets you from a handful of docs to a production‑grade pipeline without choking on latency.
Most RAG demos work because the dataset is 12 documents. Ours hit 400k and fell over — the fix wasn’t a bigger model, it was reranking. In this piece I walk through the exact stack we use, the failure modes we hit, and the concrete fixes that turn a flaky prototype into a reliable service.
I built the first version on a laptop with LangChain, a tiny FAISS index and GPT‑3.5. It answered questions fast, but the moment we added a second data source the latency doubled and the answers started hallucinating. The lesson is simple: you need a solid retrieval backbone, a deterministic preprocessing pipeline, and a second‑stage ranker that can prune the noisy results before they hit the LLM.
The first thing that breaks is inconsistent chunk sizes. We started with a naive split on newlines, which left some 10‑token chunks and others over 2,000 tokens. The LLM hit the context window and dropped the tail of the query. The fix is to use a recursive character splitter that respects markdown headings and limits chunks to 500‑800 tokens, then store the original source offsets for traceability.
During ingestion we also discovered that PDF OCR errors were surfacing as invisible Unicode characters. Those characters prevented the vectorizer from matching anything. A quick pass with a Unicode normalizer and a regex that strips control characters eliminated 99 % of missed matches.
FAISS is great for prototypes but it doesn’t handle incremental updates well. After the first 50k embeddings we started re‑indexing the whole collection on every new batch, which caused a 30‑minute pause in the API. Switching to a hybrid solution—Milvus for persistent storage and a small in‑memory FAISS cache for hot queries—solved the problem. The cache is refreshed every 5 minutes with the latest embeddings, keeping latency under 200 ms.
Another hidden failure mode is stale embeddings when the source text changes. We built a simple checksum on the raw chunk and store it alongside the vector. If the checksum differs on the next ingest run, we delete the old vector and replace it. This prevented the infamous “answer from last month’s spec” bug.
With 400k chunks the top‑k retrieval (k=10) often includes unrelated passages. The LLM then tries to synthesize a answer from noise, leading to hallucinations. We added a lightweight cross‑encoder (MiniLM‑v2) that scores the 10 retrieved passages against the query and keeps only the top‑3. The cross‑encoder runs in 30 ms on a single GPU and cuts hallucinations by 70 %.
The cross‑encoder also gives us a confidence score we can surface to the user. When the score falls below 0.4 we fall back to a “no answer” response and log the query for manual review. This pattern turned a flaky chatbot into a trustworthy assistant.
Even with perfect retrieval, a badly phrased prompt will make the model wander. We settled on a two‑part prompt: a system message that defines the role and a user message that injects the retrieved passages with explicit citations. The prompt also includes a short instruction: “If you cannot find a direct answer, say so.” This simple guardrail stopped the model from fabricating references to non‑existent sections.
We also wrapped the LLM call in a timeout and a retry policy. In production we observed occasional 5‑second spikes from the API provider; the retry with exponential backoff and a fallback to a cached answer kept the SLA at 99.5 %.
Finally, you need to know when the system is breaking. We instrumented every stage—ingest, embed, retrieve, rerank, generate—with OpenTelemetry spans and exported metrics to Grafana. The most useful alerts were “retrieval latency > 500 ms” and “cross‑encoder confidence < 0.3”. When either fires we automatically open a Jira ticket with the offending query and the top retrieved chunks.
Continuous improvement comes from feeding those tickets back into the data pipeline: add missing docs, adjust chunk sizes, or fine‑tune the cross‑encoder on the hardest queries. The loop closed the performance gap we initially saw when scaling from 12 to 400k documents.
A software studio that ships and maintains its own products — KeepChats, Gwora and Cairn — and builds the same way for clients. Founded and led by codewithumar.
Talk to the studio →Send the problem. You get one fixed number and a plan back within a business day.
We build software worth keeping — for clients, and for ourselves.
Founded & led by codewithumar