How to build an agentic RAG system without burning your latency budget

The query was "which of our MSAs have uncapped indemnity and expire before June." Top-k returned five agreements that discussed indemnity in gorgeous prose and not one that contained a termination date. Every chunk scored well — they were topically perfect. Cosine similarity has no representation of "before June," and the date lived in a signature block the chunker had split off two pages earlier. Retrieval didn't fail. The question was never a similarity problem, and no amount of reranking turns it into one.

The loop that fixes this is about forty lines: plan, retrieve, inspect what came back, retrieve again, answer. Writing it takes an afternoon. Our first version on a 400k-document contract corpus beat the single-shot baseline on 12% of a 150-question eval set, tied on 61%, and lost the rest — same answers at 3x the latency and 8x the cost, plus a brand-new failure mode where the agent stitched an indemnity clause from one MSA onto an expiry date from another and cited both. The loop wasn't the upgrade. The router, the structured tool, and the hard budget were.

Route before you loop

Most traffic into a production RAG system is a single-hop lookup: the refund window, the SSO setup steps, section 7 of one named contract. Running a planner over those pays agent prices for a database read. Put a classifier in front with four labels — single-hop lookup, multi-hop join, aggregation or count, out of scope. A small model with twenty few-shot examples hit 94% on our hand-labeled set, costs under a cent per thousand calls, and adds about 180ms. It sent 71% of traffic down the cheap path and moved p95 from 14 seconds to 2.1.

The remaining 6% is where you need an escape hatch, not a better prompt. After single-shot retrieval and before generation, run one check against text you already paid to retrieve: does every named entity and every constraint in the question appear in at least one chunk? Question says "before June" and nothing retrieved contains a date — escalate to the agent. That's one small-model call over 2k tokens, and it catches the routing errors that actually hurt: the ones where the fast path answers confidently from documents that never addressed the constraint at all.

Three tools, not thirty

Tool sprawl is the most common way this goes sideways. Someone exposes search_contracts, search_emails, search_tickets, search_wiki, and search_slack, and the model starts spending hops guessing which haystack holds the needle — we watched one run burn four of five hops on the ticket index because the word "ticket" appeared in the question. Collapse to three. One search tool taking a query string plus typed source and metadata filters. One fetch tool that pulls a document or named section by ID, because chunk boundaries cut mid-clause and agents constantly need the paragraph around their hit. One structured query tool over your metadata table.

That third tool produces most of the wins, and teams skip it because it feels like cheating. "Expires before June" is a WHERE clause. "How many vendors are on net-60" is a COUNT — ask a vector index and it returns k chunks and lets the model tally them, which is how you get "approximately 12" when the answer is 47. Let the agent filter on fields you extracted at ingest: dates, parties, amounts, status, document type. Our accuracy moved more from fixing a date parser that silently dropped DD/MM/YYYY rows than from any reranker swap.

The stop condition is the whole design

An agent without an explicit stop condition fails in two directions. It quits after one hop because the first result looked plausible, or it grinds through eleven near-identical queries and hits the wall clock. Both come from the same gap: nothing tracks what the question still needs. Decompose into sub-questions up front and keep a ledger. Each sub-question stays open until a retrieved chunk is cited against it. The loop ends when the ledger closes or the budget dies — five hops, a token ceiling, and a 20-second wall clock that fires mid-tool-call.

Normalize and cache every query the agent issues. Agents re-ask the same thing with fresh adjectives constantly, and in our traces a repeated normalized query was the single best predictor of a run that would exhaust its budget. Treat the second repeat as a signal: force a different tool or terminate. When the budget dies with the ledger open, say so. "I found indemnity terms for four of six MSAs and no expiry date for two" is a useful answer. A confident join across two contracts that never mentioned each other is the one people remember.

Evaluate the trace, not the answer

Answer-level scoring tells you a number dropped and nothing about which component moved it. Log every span: the plan, each query issued, the chunk IDs returned, which sub-question each citation closed. Then track four things. Sub-question recall — did retrieval ever surface the gold chunk, regardless of what the model wrote with it. Hop-count distribution, where mass piling up at your cap means the stop logic is losing. Tool-selection accuracy. And the share of runs that ended on budget rather than completion. Quality regressions show up in one of those four.

Freeze retrieval traces so you can iterate deterministically. Take 150 real questions with gold chunk IDs, record the retrieval results once, then replay them while you tune the planner prompt — that loop runs in 40 seconds instead of 20 minutes, and it doesn't move under you. Tune against live retrieval and every prompt tweak is measured against fluctuating ANN recall and a shifting index; you're reading noise. Re-run live nightly to catch index drift, and keep the replay loop for everything else.

When to skip it

Agentic RAG earns its cost when questions span sources, need filters, or ask for comparisons and counts. It does not earn its cost on one homogeneous corpus answering single-hop questions under a two-second SLA, and shipping it there buys you a system that is slower, pricier, and no more accurate. Budget honestly: 6 to 10 seconds end to end on multi-hop queries, 20 to 40 cents per query at current frontier pricing. Stream the plan and sub-questions as they're written — our abandonment on 8-second queries fell from 31% to 9% once the wait rendered as visible work.

Before you build anything, classify a week of production queries by how many hops they truly need. Under a fifth multi-hop, spend the money on chunking, a reranker, and metadata extraction — the boring path, and usually the right one. A third or more needing joins and filters, build the router, the three tools, and the ledger, in that order. Tune the planner prompt last. It's the knob with the best feel and the smallest effect, and it will absorb three weeks if you let it.

If your retrieval keeps missing the multi-hop questions, tell us what your users are asking and we'll show you where the loop pays for itself.

Send the problem. You get one fixed number and a plan back within a business day.