How to build a graph RAG system (and when to skip it)

The question that broke our first vector RAG deployment was twelve words long: which of our vendors share a parent company with anyone restricted? The index held 1,900 supplier documents and returned five passages — each naming one vendor, none naming two. Ask the same system what the MSA says about termination and it is perfect, because that answer sits inside one chunk and cosine similarity is excellent at finding one chunk. The parent-company answer was never in a chunk. It lived across three of them, and embeddings store proximity, not relationships: "Acme acquired Beta" and "Beta acquired Acme" land within about 0.02 cosine of each other.

Graph RAG pulls entities and the edges between them into an actual graph and retrieves subgraphs instead of paragraphs. It is more capable and more expensive, and most teams fail at the identical step. They spend a week choosing between Neo4j and a property graph in Postgres, write a two-hop traversal, and discover their graph holds four nodes named Acme and the same relationship labelled acquired, bought, took over and purchased a majority stake in. The store is the easy part. The extraction pipeline is the product, and every dollar and every wrong answer traces back to it.

What the graph buys you that top-k never will

Two question shapes justify the build. Multi-hop: a subsidiary named in a filing, a contract naming the subsidiary, an incident report naming the contract. No chunk holds the path, and raising k does not help — we pushed k to 100 and recall on those chains stayed at zero, because the connecting document shares no vocabulary with the question. Aggregate: "which suppliers appear in more than three open disputes" requires counting, and twenty chunks drawn from 400,000 cannot count, only sample. If most of your query log is single-fact lookup, add a cross-encoder reranker instead. Ours bought 11 accuracy points for two days of work.

The unit of retrieval changes as well. You return an entity, its neighbours one or two hops out, the labelled edges, and the source passage behind each edge. The model reasons over a small explicitly connected context instead of a pile of loosely related paragraphs. On a 400-document supplier corpus we finished with roughly 12,000 entities and 30,000 edges, and the assembled context for most questions came in under 2,000 tokens — against about 8,000 for a top-20 chunk dump that answered fewer of them.

Extraction and entity resolution are the whole job

Extraction is an LLM pass over every chunk, and the first decision is open schema versus typed. Open extraction produces a graph where traversal by edge type is worthless, because four labels mean one relationship. Fix the entity types and the relation vocabulary up front — we settled on 14 and 21 — make the model pick from that list, and give it an OTHER bucket that records the verbatim phrase, so you can promote whatever shows up 200 times into the vocabulary next run. Every edge carries its chunk ID, document ID and character offsets. That provenance is how you cite an answer and how you debug the graph six weeks later.

Then entity resolution, where naive builds die. "Acme Corp", "Acme Corporation", "ACME Corp." and "Acme" are four nodes until merged, and a fragmented graph fails silently: the traversal returns an empty set, not an error. Block candidates on trigram similarity and embedding distance, auto-merge the confident band, and send only the ambiguous middle to an LLM with surrounding context — "Apple" the fruit supplier and "Apple" the hardware vendor must stay apart. Over-merging is the worse failure, since it invents edges and cites them. Extraction plus resolution cost us about $380 and six hours wall clock, and we reran it nine times while tuning the vocabulary.

Community summaries for questions with no local answer

Global questions — "what are the recurring themes in customer complaints this quarter" — have no entry point, because the answer is a property of the corpus rather than any entity. Run Leiden over the graph to cluster densely connected entities, summarise each cluster, then summarise clusters of clusters. Our 12,000 entities produced 340 leaf communities, 61 at the middle level and 9 at the top. A global query becomes map-reduce across summaries at one level instead of a traversal, and the answer reflects the corpus rather than twenty lucky chunks.

Price this honestly. A map-reduce across all 340 leaf summaries is 340 model calls, roughly a dollar and 40 seconds per question — fine for a weekly report, ruinous inside a chat box. Cache by question embedding, start at the 9-community top level and descend only when the answer reads vague, and hard-cap how many summaries one query may touch. Communities drift as documents arrive; we re-cluster on a weekly job rather than per ingest and accept slightly stale summaries between runs.

Route the query instead of graphing everything

In production most questions stay boring lookups, and routing them through traversal makes them slower and pricier without making them righter. Put a cheap classifier in front picking one of three paths: hybrid retrieval for single facts, local graph search for multi-hop, community search for aggregates and themes. Train it on 200 real questions pulled from your logs — that beats any prompt written from imagination — and log every routing decision so misroutes are auditable. Ours lands around 94%. When confidence is low, run the cheap path and escalate only if the answer comes back thin.

Local graph search should still start with vectors: embed the question, take the five closest entity nodes as seeds, expand one or two hops, then rerank the resulting edges and source passages before assembling context. Cap fan-out per node — one hub entity like the client's own company name connects to 4,000 others and will drag the entire graph into a two-hop expansion. Pure traversal misses paraphrase, pure vector search misses structure, and the reranker at the end is what stops a three-hop walk from filling the window with noise.

Incremental updates, and the honest case for skipping this

A graph you rebuild from scratch every ingest is a graph you stop updating by month two. Design for deltas immediately: extract only new and changed chunks, resolve their entities against the live graph rather than a fresh one, upsert edges keyed on source chunk so a reprocessed document replaces its own facts instead of duplicating them, and flag affected communities dirty for the next clustering run. Deletions matter more than teams expect — when a contract is terminated and its edges survive, the system cites it with full provenance and total confidence.

And the part vendors skip: if your corpus is under a few thousand documents and your users ask lookup questions, this is the wrong build. BM25 plus hybrid search, deliberate chunking and a cross-encoder reranker reach most of the accuracy for a fraction of the indexing bill and a tenth of the maintenance surface. A graph earns its keep when questions sit structurally beyond retrieval: connections spanning documents, counts spanning a corpus, the same entity wearing four names across three systems. Build it for those, and route everything else somewhere cheaper.

If the answers your team needs live between documents rather than inside them, tell us about your corpus and we'll tell you whether a graph is worth building.