Building an AI chatbot with a custom knowledge base fails at ingestion and retrieval long before the model matters. Here are the sync, chunking, permission and evaluation decisions that decide whether it survives past week three.
Ask ten teams how they'd build an AI chatbot on their own data and nine describe the same thing: dump the PDFs into a vector store, point a model at it, ship. That demo works. It works because you tested it with questions you already knew the answers to, against forty documents that happened to be current on the day you indexed them. Week three is when it breaks — someone asks about the refund window and gets the 2023 policy, because the new one lives in Notion and nobody ever deleted the old PDF.
The model is the cheapest, most replaceable part of this system. What you're actually building is a data pipeline with a freshness guarantee, a permission model, and a test harness — and then a fairly boring prompt sitting on top of it. Every expensive failure we've been called in to fix on a custom chatbot lived upstream of the prompt. Below is what we build instead, and why each piece exists.
A one-time upload is a snapshot of a corpus that has already started rotting. Real ingestion is a connector per source — Zendesk, Confluence, Notion, a Postgres table, a Drive folder — that runs on a schedule and pulls only what changed since the last cursor. Store the source's updated_at and a content hash per document; if the hash matches, skip re-embedding entirely, which keeps a nightly sync over 80k documents costing a few dollars instead of a few hundred. The part everyone skips is deletion. Notion soft-deletes pages, Confluence archives them, and Drive moves them to a trash folder that your service account can still read — all three will keep answering questions for months unless you explicitly reconcile the ID set and tombstone what's gone.
Different sources rot at different speeds, so track freshness per source rather than globally. Support macros change weekly, pricing changes quarterly, a signed MSA changes never. We attach source_updated_at to every chunk as metadata and surface it in the citation, so an answer reads "per the Returns Policy, updated 14 Aug 2026" instead of asserting a fact with no provenance. That one line does more for user trust than any amount of prompt tuning, and it makes stale content visible to the humans who own it.
The default 512-token sliding window is what turns a good corpus into a bad one. It splits a numbered procedure across two chunks so step 4 arrives without steps 1 through 3. It cuts a pricing table in half, so the tier names land in one chunk and the numbers in another, and the bot confidently pairs the wrong ones. Chunk on the document's own structure instead: split at heading boundaries, keep tables and list items whole even when they run long, and prepend the full heading path to every chunk so "Returns > International > Exceptions" travels with the text.
The other structural fix is cheap and underused. Half the chunks in any real corpus start with an orphaned pronoun — "It must be returned within 30 days in original packaging" — which embeds into a vector that means almost nothing. Run one small-model call per chunk at index time to write a single sentence of surrounding context and prepend it. It costs single-digit dollars per hundred thousand chunks, runs once, and in our benchmarks moves recall@10 more than swapping to a larger embedding model does.
Dense embeddings are good at paraphrase and bad at identifiers. A user typing "error PGX-4471" or "SKU 88231-B" or a customer's company name gets fuzzy nearest-neighbours that share a topic but not the token that actually mattered. BM25 nails those and is terrible at "how do I get my money back." Run both, fuse the result lists with reciprocal rank fusion, and you stop arguing about which retriever is better. Then over-fetch — top 50 — and put a cross-encoder reranker in front of it to cut down to the six chunks that reach the model. Retrieval quality is the single biggest lever on answer quality, and reranking is the cheapest way to pull it.
Multi-turn is where otherwise-fine systems fall apart. "What about for enterprise?" is not a searchable string; embedded on its own it retrieves noise. Rewrite the query against the last few turns into a standalone question before it ever touches the index, and log both the raw and rewritten forms so you can debug the rewriter separately from the retriever when someone reports a bad answer.
Telling the model "only answer using documents the user is allowed to see" is not access control, it's a suggestion. Every chunk carries the group IDs that can read its source document, and the vector search runs with a metadata filter built from the caller's resolved groups. Nothing the user isn't cleared for is ever retrieved, so nothing can leak through a clever prompt or an over-eager summary. For genuinely separate tenants, go further and give each one its own namespace or index — filters are one config mistake away from cross-tenant retrieval, and namespaces fail closed.
Denormalized ACLs go stale the moment someone changes teams, so resolve the caller's groups per request from your identity provider rather than caching them on a session, and re-sync document-level permissions whenever the source reports a change. This is also the part that decides whether the chatbot can touch HR, legal or finance content at all. Get it right early and those corpora become available later; bolt it on afterwards and you're reindexing everything.
Before launch, pull 60 to 100 real questions out of the support inbox or sales call notes, and for each one record both the correct answer and the document that contains it. Now you can measure the two things separately: did retrieval surface the right source in the top k, and did the model use it correctly. Nearly every complaint that arrives as "the AI is dumb" turns out to be recall — the right chunk was never in the context window, and no model can reason its way out of that. Run the set in CI on every prompt, chunking or embedding change, because all three are silent regressions otherwise.
Include questions your corpus genuinely cannot answer, and score a clean refusal with a pointer to a human as correct. A chatbot that says "I don't have that" ten percent of the time is worth deploying; one that never refuses is inventing answers and you just haven't caught it yet. Log every low-confidence and refused query into a weekly review — that list is the highest-signal content roadmap your documentation team will ever get, and it's the mechanism that makes the thing improve after launch instead of decaying.
A software studio that ships and maintains its own products — KeepChats, Gwora and Cairn — and builds the same way for clients. Founded and led by codewithumar.
Talk to the studio →Send the problem. You get one fixed number and a plan back within a business day.
We build software worth keeping — for clients, and for ourselves.
Founded & led by codewithumar