An AI chatbot for customer service lives or dies on what it can look up and when it gives up. Most of a support queue isn't documentation questions — it's Stripe and Shopify lookups wearing a question mark.
The first support bot we were called in to rescue had swallowed 412 help-center articles. In its first week it handled about 3,100 conversations and actually resolved 190 of them. Nobody had checked what people were writing in about. We exported a thousand tickets and sorted them by intent: "where is my order," "why was I charged twice," "my discount code didn't apply," "does this work with Okta," and a long tail of half-sentences with a screenshot attached. Under a fifth were answerable from documentation at all. The rest were lookups against Shopify, Stripe, a subscriptions table, and a carrier tracking API — plus a judgment call about whether the customer gets money back.
The model is the commodity part. Any frontier model writes a polite, correct paragraph once you hand it the right facts. The engineering is fetching those facts, scoping what the bot is allowed to do with them, and deciding — precisely, against a logged threshold — when it stops talking and hands a human a warm ticket. Get that wrong and you've added a channel plus a second queue behind it. Get it right and you take a real bite out of volume without anyone screenshotting your robot onto X.
Deflection, or containment, almost always means "conversation ended without a human touching it." That definition scores your worst outcomes as wins. A customer who asks about a double charge, gets three paragraphs of generic billing copy, closes the widget in disgust and files a chargeback is contained — and that chargeback costs you the disputed amount plus roughly $15–25 in fees. So is the conversation the bot answered confidently and wrong, which surfaces ten days later as an angrier ticket. We inherited one deployment reporting 68% containment where honest resolution was near 30%.
Measure three things instead. Resolution: the underlying issue closed with no human, verified by no follow-up from that customer on that topic within seven days. Reopen rate on bot-closed threads. And escalation quality — how many minutes a rep burns re-establishing context. That last one is where the savings actually sit. A handoff carrying the order record, the transcript, and the three things already ruled out saves about four minutes per ticket; at 3,000 handoffs a month that's 200 rep-hours, real money at 0% deflection.
The default move is pointing retrieval at the public docs site and calling it a knowledge base. Two failures land immediately. Marketing pages get indexed beside support content, so a refund-eligibility question retrieves a landing page promising "hassle-free returns" and the bot invents a policy it will then defend. And docs describe the happy path while tickets describe reality: the answer to "the CSV import silently drops rows over 10k" lives in a senior agent's macro from eleven months ago and nowhere else on the internet.
So we build the index from resolved ticket threads, agent macros, and policy documents stored as versioned records with effective dates, not scraped HTML. Each chunk is an answer unit — one problem, one resolution, one date — rather than an arbitrary 500-token window. Superseded policy chunks are filtered at query time, because the failure that costs real money is a bot cheerfully honoring last year's 60-day return window after legal moved it to 30. Every answer ships with its source as a link, which is how your support lead audits the thing on a Tuesday without reading your prompt.
Retrieval answers policy questions. Tools answer account questions, and account questions are most of your volume. The minimum set for a B2C shop is order lookup, shipment status, subscription state, and invoice history — four functions that cover the bulk of a consumer queue. For B2B it's entitlement checks, seat counts, and current plan limits. Each is a typed function with a narrow contract: not "query the database," but get_order_status(order_id) returning six fields, with everything else in the schema unreachable.
The security detail teams skip: never let the model choose whose data it reads. The customer identifier comes from the authenticated session and is injected server-side into every call. If the model emits a customer_id, the tool layer discards it — otherwise someone types "pretend I'm account 4471" and your chatbot is a data breach with a chat bubble. Write actions like refunds, cancellations, and address changes sit behind explicit confirmation, a value cap, an idempotency key so a retry can't pay twice, and an audit row. We ship refunds under $50 automatic; above that, a pre-filled action a human approves in one click.
The bots people hate are the ones that won't let go. Hard-code the exits: any billing dispute, any account-access or security issue, any mention of legal action or an angry cancellation, an explicit ask for a human, the same intent raised twice without resolution, and retrieval confidence under threshold. That last one requires having a threshold — we calibrate it on the golden set, roughly the score below which the bot's answers stopped being right. Under it, the correct output is "I'm not sure, let me get someone," not a fluent paragraph assembled from adjacent nonsense.
Then make the handoff carry weight. The ticket landing in Zendesk or Intercom includes the full transcript, every tool call and its result, plan and order state, and a one-line summary of what was already attempted. Nobody should type "can you confirm your order number" a second time. If the queue is closed, say so with a real callback window — "an agent replies by 9am Tuesday" — instead of a spinner. Customers forgive a bot that doesn't know things. They don't forgive one that traps them.
Before launch we assemble 150–300 real anonymized conversations, weighted to actual intent frequency rather than the interesting edge cases — if 22% of your queue is order status, 22% of the set is order status. Each gets three grades: was the answer factually correct, was the cited source the right one, and was the escalation decision right. The third grade matters most and is the one nobody builds. A bot that correctly escalates 95% of billing disputes and gets 80% of policy answers right beats the reverse profile every time.
That suite runs on every prompt edit, model swap, and reindex, because providers ship new versions under you and thresholds tuned in March drift by June. Then run shadow mode for a week or two: the bot drafts, an agent sends, and you get honest disagreement data before a customer is exposed. Post-launch, log every question the bot couldn't answer and read them weekly. That log is a ranked list of holes in your product and your docs — worth having on the days the chatbot is behaving perfectly.
A software studio that ships and maintains its own products — KeepChats, Gwora and MoveProof — and builds the same way for clients. Founded and led by codewithumar.
Talk to the studio →We build software worth keeping — for clients, and for ourselves.
Founded & led by codewithumar