How to build a simple RAG system in about 200 lines — and the three places where simple quietly stops working: chunk boundaries, retrieval that only knows embeddings, and having no eval set.
A working RAG system is a loader, a chunker, an embedding call, a vector index, a retrieval query, and a prompt. That is genuinely it. In Python with pgvector and one of the OpenAI or Voyage embedding models, the whole thing fits in about 200 lines with room for logging. We have shipped versions of this to clients in under a week, and the first one we ever built for a legal-ops team answered questions about 6,000 contracts using code you could read in one sitting.
The trouble is that most people spend their complexity budget in the wrong place. They add a query-rewriting agent, three retrieval strategies behind a router, and a self-critique loop before they have checked whether their chunker is cutting tables in half. Simple is the right starting architecture — but simple has three specific failure points, and every one of them is upstream of the model. Fix those and a plain pipeline beats a clever one that nobody can debug at 2am.
Store documents in Postgres with the pgvector extension. One table for source documents, one for chunks with a foreign key back, a vector column sized to your embedding model, and an HNSW index on it. Skip the dedicated vector database on day one. If your corpus is under a few million chunks, Postgres will serve queries in under 50ms and you get transactions, joins, and backups you already know how to operate. We have migrated exactly two clients off pgvector to a specialist store, both because of multi-tenant filtering at scale, and neither because of raw speed.
Ingestion is a job, not a script you run by hand. Every document gets a content hash. On re-ingest you compare hashes, and only changed documents get re-chunked and re-embedded. This one detail is the difference between a demo and something a client can keep feeding. Without it, someone re-uploads the same 400-page policy PDF, you pay for 900 embeddings again, and you end up with duplicate chunks that crowd out the good answer in top-k.
Retrieval is a single SQL query with a filter clause and an ORDER BY on cosine distance. The prompt takes the retrieved chunks with their source titles and a hard instruction to answer only from the provided context and to say so when the context does not cover it. Return the source IDs alongside the answer. Citations are not a nice-to-have feature you add later — they are the only cheap way anyone can tell whether the retrieval step did its job.
Character-count splitting is the default in every tutorial and it is wrong for most real corpora. Slicing at 1,000 characters cuts through the middle of a table row, orphans a heading from the paragraph it introduces, and separates a clause from the definition that gives it meaning. The retrieval then works perfectly and returns a chunk that is technically relevant and practically useless, and the model, having nothing better, invents the missing half.
Split on structure first, then on size. Parse headings, list items, and table boundaries from the source format, build chunks that respect those boundaries, and only fall back to a token-count split inside a section that is genuinely too long. Around 800 tokens with 100 tokens of overlap is a sane default for prose. Then prepend the document title and heading path to each chunk before embedding — that single line of string concatenation has done more for retrieval quality on our projects than any model upgrade. A chunk that reads 'Refund Policy > EU Customers > 14-day window' embeds very differently from one that starts mid-sentence with 'this period may be extended'.
Go read 30 of your chunks. Not a sample of 3 — 30, picked at random, printed to your terminal. If you cannot answer a real user question from a chunk you are holding in your hand, neither can the model. This takes twenty minutes and catches problems that a week of prompt tuning will not.
Pure embedding search fails on exactly the queries your users care most about: product SKUs, error codes, invoice numbers, proper nouns, and any term that was rare in the embedding model's training data. Someone searches for 'ERR_TLS_4412' and the vector index helpfully returns four chunks about general network troubleshooting. Postgres already ships full-text search, so add a tsvector column, run both queries, and fuse the two ranked lists with reciprocal rank fusion. That is roughly fifteen lines and it fixes an entire class of complaints.
Then rerank. Pull 20 candidates from the fused list, run them through a cross-encoder reranker — Cohere's or a local bge-reranker both work fine — and pass the top 5 to the model. Bi-encoder embeddings are compressed into a single vector and lose the fine-grained interaction between question and passage; a cross-encoder reads the pair together and sorts it properly. On a 400,000-chunk corpus we watched answer accuracy on a held-out question set go from the low sixties to the high eighties from reranking alone. The bigger model would have cost more and fixed nothing, because the right chunk was never in the context window.
Before you tune anything, sit down with whoever owns the content and write 50 real questions with the document and passage that should answer each one. This takes an afternoon and it is the highest-value afternoon in the project. Now you can measure recall at k — how often the correct chunk appears in what you retrieve — which is a number you can compute without an LLM, without a judge model, and without arguing about tone. If recall at 5 is 60%, no amount of prompt engineering will save you, and you now know that before you waste a week on it.
Run that set on every change: a new chunker, a different embedding model, a reranker, a tweak to overlap. Most changes will move the number by a point or two, and some will move it by fifteen. The ones that move it by fifteen are the ones worth keeping. We have killed plenty of interesting ideas this way, including a query-expansion step that looked brilliant in three hand-tested examples and lost four points of recall across the full set.
There are real reasons to outgrow this design, and they are narrower than the internet suggests. Questions that require joining facts across many documents want a graph layer. Corpora where the answer lives in a chart or a scanned diagram want a multimodal pipeline. Workflows where the model must decide what to look up next, and then look up something else based on that, want an agentic loop and the latency budget to pay for it. Tens of millions of chunks with hard per-tenant isolation want a purpose-built vector store.
Everything else — the large majority of internal knowledge bases, support deflection bots, and document Q&A tools we get asked to build — is served well by structural chunking, hybrid retrieval, a reranker, honest citations, and an eval set you actually run. Build that first. It will answer most of your questions, it will tell you clearly which ones it cannot answer, and when you do need something more sophisticated, you will have the measurements to prove which part needs it.
A software studio that ships and maintains its own products — KeepChats, Gwora and Cairn — and builds the same way for clients. Founded and led by codewithumar.
Talk to the studio →Send the problem. You get one fixed number and a plan back within a business day.
We build software worth keeping — for clients, and for ourselves.
Founded & led by codewithumar