A website chatbot takes three days to demo and three months to trust. The failures are always the same four — boilerplate-poisoned chunks, a six-second first token, invented pricing, and an injected review page — and each has a fix that survives real traffic.
Nine days after launch, a client's widget offered a visitor a 40 percent volume discount. The discount had never existed. The retriever had handed the model four thin chunks of a 2019 case study in which one customer negotiated one, and the model did what models do with weak context: it wrote a fluent sentence. Nobody had touched the prompt or the model since the demo everyone loved.
Every one of these demos well. Crawl, embed, wire to a model, and by Thursday it answers questions about your company with eerie confidence. Real traffic breaks it inside a week, and never where you were looking. We have built these for B2B software teams, a logistics firm with 900 pages of shipping policy, and a medical device distributor where a wrong compatibility answer is a reportable event. Across all three, swapping the frontier model at the end of the pipe moved our eval score two to three points. Fixing what goes into the context window moved it thirty.
Point a crawler at a marketing site and every chunk carries the same nav menu, the same footer, the same cookie banner, the same 40-word company blurb. On the sites we have measured, that chrome is 60 to 80 percent of the tokens on a page. Embed it and the index collapses toward a single point — we logged 0.85 cosine similarity between a pricing page and an about page whose only shared text was a header. The symptom is absurd: the top hit for "how do I cancel" came back as the careers page, because careers and billing share a 180-word footer. The retriever isn't confused. You gave it near-identical text.
The fixes are unglamorous. Strip chrome with readability-style extraction before you chunk, not after. Split on heading boundaries instead of a fixed 512-token window, and carry the heading path and canonical URL as metadata so every answer can cite its source. Deduplicate with SimHash over normalized text; on a 1,200-page site that deleted 40 percent of the index. Then accept that some answers live on no page at all — enterprise pricing, SLA terms, return-policy edge cases. Write 30 to 60 hand-authored question-and-answer pairs for those and index them beside the crawl. That one file beats every clever chunking scheme we have tried.
A chat widget competes with the back button. Our budget is first token under 900 milliseconds, and the fashionable architecture blows it before you notice: rewrite the query with an LLM, embed, retrieve, rerank with a second LLM call, then generate. Four round trips before one character reaches the visitor. We inherited an agency-built widget that took 6.2 seconds to start talking, two of those a cold serverless start on the embedding endpoint. Thirty-eight percent of sessions closed it before the first token landed.
Keep the hot path short and boring. Embed the query with a small model you keep warm, pull top 30 from a vector index in the same region as your API — a cross-region hop costs 100 to 200 milliseconds for nothing — rerank those 30 with a cross-encoder in roughly 80 milliseconds, and stream from the first token. Skip LLM query rewriting until your eval set proves it earns its 400 milliseconds; for follow-up turns, heuristic coreference against the previous user message resolves most pronouns. Then cache: on every site we have instrumented, eight questions are 30 to 40 percent of traffic, and a semantic cache turns those from a two-second answer into a 40-millisecond one.
Nobody gets fired because the bot said it didn't know. They get fired when it quotes a discount that doesn't exist, or tells a hospital buyer a pump is compatible with a controller it was never tested against. A model handed four weak passages still writes a confident paragraph — that is the entire job description. You have to build the exit and make taking it cheap.
Put a floor on the reranker score, tuned against your eval set, and short-circuit below it: no generation call at all, just a scripted line and a route to a human. Instruct the model to answer only from the delimited passages and to emit the source URL, then render those citations as clickable links — visible sources change what visitors ask next and make your own QA tractable. For pricing, contract terms, and clinical or financial claims, route to deterministic responses or a person instead of pretending the model is the right tool. On the device distributor build, 11 percent of questions hit a hard-refusal rule. That 11 percent is why legal signed the launch.
A public chatbot is an open, metered endpoint on your infrastructure with a language model behind it. Visitors will paste "ignore your previous instructions" and forty variants of it, and enough of them land that you should treat your system prompt as public: no API keys, no internal hostnames, no unreleased roadmap in it. If you indexed reviews, forum threads, or support comments, that text can carry instructions straight into your context window. Keep retrieved passages inside a delimited block and tell the model they are data, never commands.
The boring controls do more than the clever ones. Proxy every model call through your own server so no key touches the browser. Rate limit per session and per IP, cap turns and total tokens per session, and set a hard monthly spend ceiling with an alert — one scripted abuser burned four figures of inference on a client account overnight before that ceiling existed. If the bot can act, booking a demo or opening a ticket, put it behind an explicit confirmation and a scoped service account, not the credentials your admin panel uses.
The job is not answering everything. It is answering the repetitive 70 percent instantly and handing the rest to a person with context attached: transcript, the pages the visitor read, the email you captured, pushed into the helpdesk or CRM as a real ticket rather than a Slack alert someone scrolls past. A bot that answers well and dead-ends a qualified buyer is worse than no bot, because it consumed the one moment they wanted to talk.
Then instrument it. Log every conversation where the relevance floor fired or the visitor rephrased the same question twice — that log is the highest-signal content backlog your marketing team will ever get. Track containment, escalation, and cost per conversation, which sits between two and six cents with a small model and a warm cache and multiplies without either. Run a fixed set of 100 real questions with expected answers on every content or prompt change, because the failure mode here is silent regression: someone edits one line of the system prompt to fix a complaint and quietly breaks nine answers that worked yesterday.
A software studio that ships and maintains its own products — KeepChats, Gwora and MoveProof — and builds the same way for clients. Founded and led by codewithumar.
Talk to the studio →We build software worth keeping — for clients, and for ourselves.
Founded & led by codewithumar