Most vector search projects start with the wrong question. Teams spend a week comparing Pinecone, Qdrant, Weaviate and pgvector, pick one, embed every document, and then watch it return a confident list of almost-relevant passages. The database was never the problem. Picking one takes an afternoon. What separates search that works from search that only looks like it works is a handful of quieter decisions about what you embed, how you combine it with keyword search, and how you prove it improved.

This is a guide to those decisions, written for developers who have already got a toy version running and now have to make it hold up on real data. If you want the code-level walkthrough first, our Python vector search implementation guide covers the mechanics. This piece covers the judgment calls that come after it.

Build the evaluation set before the index

Before you embed anything, write down 50 to 100 real queries and, for each one, the passage or document that should come back. Pull them from support tickets, search logs or the people who will actually use the system. Don't write them yourself from the documents. Queries written by someone looking at the answer share its vocabulary, and that makes any retriever look brilliant. Real users ask "why did my invoice double", not "billing duplication policy".

With that set in hand, every later choice becomes a measurement instead of an argument. Track recall at k: for what share of queries does the right passage appear in the top five or top ten results? Changing the chunk size, swapping embedding models or adding a re-ranker either moves that number or it doesn't. Without it, teams tune by vibes, ship the version that worked on the three queries someone tried in a meeting, and find out in production.

Chunk for the question, not the file

An embedding is a summary of meaning, and a chunk that covers four topics gets an embedding that is close to none of them. Fixed 1,000-token windows are the default because they are easy, not because they suit your documents. Split on the structure the documents already have, such as headings, clauses, FAQ entries or table rows, and keep each chunk about one thing. Prepend the document title and section heading to the chunk text before embedding so that a paragraph saying "this applies after 30 days" still knows what "this" is.

Store chunk-to-parent links too. Retrieval works best on small, focused units, but the model or the user usually needs the surrounding context to act on a result. Matching on the small chunk and returning the parent section gives you precision and context without forcing one chunk size to do both jobs.

Pure vector search loses to keywords on the queries that matter

Embeddings are good at paraphrase and bad at exact strings. Product codes, error messages, invoice numbers, people's names and version strings all get smeared into nearby meaning, so a search for "ERR_4012" returns passages about errors in general. BM25 keyword search handles these perfectly and fails at everything embeddings are good at. Run both and merge the results with reciprocal rank fusion, which combines ranks rather than raw scores and so needs no tuning to reconcile two scales that were never comparable.

Filters are the other trap. Approximate nearest neighbour indexes like HNSW find the closest vectors first, so if you filter by tenant, language or date afterwards, a restrictive filter can leave you with two results out of ten, or none. Check whether your database filters during the graph search or after it, and test with your most selective real filter, not the demo one. In a multi-tenant product, a tenant filter applied after retrieval is not just a relevance bug. It is how one customer's documents end up in another customer's results.

Re-rank, then plan for re-embedding

The first-stage retriever's job is to get the right passage somewhere into the top 50. A cross-encoder re-ranker then reads each query and passage pair together and reorders them, and that usually beats any amount of fiddling with the embedding model. It adds latency, so re-rank a shortlist, not the whole corpus. If your search feeds an LLM, this step does more for answer quality than prompt engineering, which is most of what we cover in building RAG that doesn't hallucinate.

Finally, assume you will change embedding models. Vectors from different models live in different spaces and cannot be mixed in one index, so store the model name and version next to every vector, keep the source text, and make re-embedding a script you can run rather than a migration you have to invent. Handle deletes and updates from day one as well. A vector for a document that was edited last month will keep answering with last month's facts until something removes it.

Where pgvector is enough

For most B2B products, the corpus is thousands to a few million chunks, and Postgres with pgvector handles that comfortably while keeping your filters, permissions and joins in one transactional database. A dedicated vector store earns its place when you have very large collections, heavy concurrent query load or a need for features like built-in quantization at scale. Start where your data already lives, measure against the evaluation set, and move only when a number tells you to.

If you are wiring this into a retrieval pipeline end to end, our guide to setting up a RAG system that finds the right evidence picks up from here. If you would rather have someone build and measure it against your own documents, that is what our vector search work is for.

Questions, answered.

What is the best database for vector search implementation?

For most products with up to a few million chunks, Postgres with pgvector is enough and keeps filters and permissions in one place. Dedicated vector databases help at very large scale or high query load. The database matters less than chunking, hybrid search and evaluation.

Should I use hybrid search or pure vector search?

Use hybrid search for almost any real corpus. Embeddings handle paraphrase but miss exact strings like product codes and error messages, which keyword search catches. Reciprocal rank fusion merges the two result lists without score tuning.

How do I measure if my vector search is working?

Build a set of 50 to 100 real user queries, each paired with the passage that should be returned. Measure recall at k, meaning how often the right passage appears in the top results. Re-run it after every change to chunking, models or ranking.

What chunk size should I use for embeddings?

There is no universal number. Split along the structure your documents already have so each chunk covers one topic, and add the title and section heading to the chunk text. Then test chunk sizes against your evaluation set instead of guessing.

Do I need to re-embed everything if I change embedding models?

Yes. Vectors from different models are not comparable and cannot share an index. Store the model version with each vector and keep the source text so re-embedding is a repeatable script.