A client shipped semantic search over 2 million support tickets. It passed every test. Three days after launch, a tenant with 40 documents opened a ticket saying search returned nothing — not bad results, nothing, an empty page, every query. The index was healthy. Recall on the eval set was 0.97. The bug was that their 40 documents never made it into the global top 100 before the `tenant_id` filter ran in Python, so the filter had nothing left to keep. One line of post-filtering, invisible for the 4,960 tenants big enough to hide it.
That's the shape of every vector search failure worth writing about: it doesn't error, it doesn't spike a dashboard, and the demo corpus can't produce it. Forty vectors and a cosine loop always works, because forty vectors have no failure modes yet. The interesting part starts at a million, when the brute-force scan takes 800ms, the HNSW index you swapped in returns results that are subtly worse, and someone asks why searching "invoice not paid" returns the paid invoices. Every one of those has a known fix, and almost none of them are about the embedding model. Here's what a Python vector search implementation looks like when it has to hold up.
Pick the distance metric your model was trained for
The first bug most teams ship is a metric mismatch. You use an embedding model trained with cosine similarity, then hand the vectors to an index configured for L2 because that was the default in the example code. It doesn't error. It returns results, they're just quietly wrong in a way you'll blame on the model. If your vectors are L2-normalised — and most sentence-transformers checkpoints normalise in the pooling layer — then cosine, dot product and euclidean rank identically, so the mismatch hides in every test you wrote. Swap in a model that doesn't normalise, like OpenAI's `text-embedding-3-large` truncated to 256 dims via Matryoshka, and the rankings diverge with no stack trace and no deploy to blame.
Normalise explicitly and pick deliberately. In numpy that's `v / np.linalg.norm(v, axis=1, keepdims=True)`, done once at write time, never in a query-path loop. Then use inner product, because on normalised vectors it is cosine and it's the fastest path in every index — FAISS `IndexFlatIP`, pgvector's `vector_ip_ops`, Qdrant's `Distance.DOT`. Then write the four-line test: embed "the invoice is overdue" and "payment is late", assert similarity > 0.7, assert `index.d == model.get_sentence_embedding_dimension()`. Run it in CI. That test catches metric mismatches, dimension mismatches, and the afternoon someone bumps the model in `requirements.txt` without telling you.
HNSW is a recall/latency dial, not a checkbox
Once brute force is too slow you reach for an approximate index, and that's HNSW nearly every time — pgvector, Qdrant, Weaviate and FAISS all default to it. But HNSW isn't a mode you turn on, it's three numbers, and the defaults are tuned for SIFT1M and GIST1M, not your corpus. `m` sets how many neighbours each node keeps: 16 is the usual default, 32 buys recall on high-dimensional text embeddings and roughly doubles the graph's RAM footprint. `ef_construction` sets how hard the build works: going from 64 to 200 on 2M vectors turned a 25-minute ingest into 90 minutes and moved recall@10 by about a point. `ef_search` is the one that matters, because it's the runtime dial and you can change it per query without rebuilding anything.
Measure it instead of arguing about it. Take 1,000 real queries, compute the true top-10 by brute force with `IndexFlatIP`, then sweep `ef_search` from 40 to 400 and plot recall@10 against p95 latency. There's a knee, usually between 100 and 200, where recall sits at 0.98 and p95 is still in double-digit milliseconds. Past it you're paying 3x the latency for 0.4% recall. That plot is a twenty-minute script and it's the most useful artifact in the project, because it turns an argument about vibes into a number someone can put in a ticket.
Filtered search is where recall goes to die
Back to the 2 million documents across 5,000 tenants. The naive implementation searches the whole index for the top 100, then filters by `tenant_id` in Python. For the tenant with 300k documents it's fine. For the tenant with 40, it returns zero rows, because none of their documents cracked the global top 100 — the filter is applied to a candidate list that was chosen without knowing the filter exists. This is post-filtering, and it fails hardest exactly when the filter is most selective, which is exactly when you need it. The user reports it as "search is broken for us," which is accurate.
The fix depends on your engine, so know which one you have. Qdrant and Weaviate do filtered HNSW traversal — the filter is evaluated during the graph walk, so pass it to the engine instead of the list comprehension. pgvector needs help: with an HNSW index and `WHERE tenant_id = $1`, the planner scans the index and throws away candidates that miss the predicate, so you either push `hnsw.ef_search` to 400+ or build partial indexes for your dozen largest tenants. And when a filter cuts to under a few thousand rows, skip the ANN index entirely and brute-force the subset — 2,000 normalised vectors through a numpy matmul is well under a millisecond and gives perfect recall. Route on cardinality: read the tenant's document count, take one branch above 10k and the other below. One code path does not serve every query.
Embeddings retrieve, rerankers decide
A bi-encoder compresses an entire document into one 768-dim vector before it has ever seen your query. That compression is why search is fast, and it's also why the top result is so often almost right. Embeddings are bad at negation: "invoice not paid" and "invoice paid" sit at cosine 0.94 on most models, because the encoder weights topic far above polarity. They're bad at exact identifiers — search `E4021` and you get documents about error codes generally, since the token was never in the vocabulary and got shredded into subwords. And they're bad at rare terms, which is precisely where a BM25 pass still beats a current embedding model outright.
So build two stages. Retrieve wide and cheap: 50 candidates from the vector index, 50 from BM25, fused with reciprocal rank fusion — eight lines of Python, `1/(60+rank)` summed per document, no score calibration needed because it only reads ranks. Then rerank the union with a cross-encoder, bge-reranker-v2-m3 or Cohere Rerank, which reads query and document in the same forward pass and resolves the negation the bi-encoder smeared. Reranking 100 candidates costs 50-150ms. On that 400k-ticket corpus, moving to a bigger, slower embedding model bought about two points of nDCG@10. Bolting a reranker onto the cheap model bought eleven.
Measure retrieval before you blame the LLM
Most broken RAG systems are broken because nobody has a number for retrieval quality. The LLM is downstream and it's what the user sees, so it takes the blame, and the team burns a month on prompt engineering to fix a problem where the right chunk was never in the context window. Build the golden set first: 100 real queries from your logs, each labelled with the correct document IDs by someone who knows the domain. It takes an afternoon, it's tedious, and it is not optional — every hour you skip here you spend fivefold guessing later.
Then track two numbers in CI: recall@50 for retrieval, nDCG@10 for the reranked list. They fail differently. If recall@50 is 0.6, the answer isn't in the candidates and no reranker or prompt will conjure it — go fix chunking, filters or `ef_search`. If recall@50 is 0.95 and nDCG@10 is 0.6, retrieval is fine and ranking is the problem — go fix the reranker. That split is what turns "search feels bad" into a bug with an owner and a next action, and it's the whole distance between a demo and a system.