Charts, tables and scanned pages don't go missing because the model is too small. They go missing because the page was flattened into undifferentiated text before anything was embedded — and nothing in the pipeline threw an error.
A pilot answered nine of ten questions about a 30-slide deck. Pointed at the real corpus — 40,000 pages of scanned service manuals, exploded-view drawings and spreadsheets printed to PDF — recall on figure-grounded questions came in near 30 percent. The instinct was a bigger vision model. The defect was upstream, in the ingest job: roughly 6,000 image-only pages had extracted to the empty string, been chunked into nothing, and been written to the index as zero-length documents. Exit code 0, no warning, no dead-letter queue. The model wasn't hallucinating the chart. It had never been shown one.
A multimodal RAG system is a document-parsing pipeline with a retriever bolted on the end. Four decisions decide whether it works: how you segment a page, what you store for each non-text region, what the retriever actually compares the query against, and what you hand the answering model at generation time. Get those right and a mid-tier vision model clears the bar. Get them wrong and the frontier model fabricates with better grammar, because the figure holding the answer never entered the index in the first place.
Run `page.get_text()` over a two-column datasheet and you get spans in insertion order, not reading order: line one of the left column, then line one of the right, interleaved down the page into text that embeds as noise. Tables arrive as space-separated fragments with row labels drifted three columns from their values, so "Flange, 3 in" and "300 psi" end up in different chunks. Image-only pages return an empty string and raise nothing. The chunk is written, the vector is stored, and six months later someone asks why the system has never once cited the 1990s maintenance archive.
Do layout detection before text extraction. Segment each page into regions, sort reading order by column block rather than raw y-coordinate, then route by type: prose to the text path, tables serialized to markdown or HTML with row and column structure intact, figures cropped to disk with their bounding box, page number and caption. Then add the guardrail that catches the silent case — if extracted characters per page fall under about 50, force the page down the OCR or vision path and log it, rather than accepting an empty chunk. On every inherited pipeline we've audited, silently empty pages cost more recall than chunking strategy and embedding choice combined.
The obvious move is to embed figures with CLIP or SigLIP into a shared space and search both modalities at once. It underdelivers on document work for a structural reason: those models were trained against alt-text-style captions and truncate text input at 77 tokens. They are excellent at matching "a dog on a beach" to a photo of a dog on a beach. "What's the pressure rating for the three-inch flange in the exploded view" is not a caption, and it lands nowhere near your CAD drawing. The tell is the score distribution — top-1 and top-50 separated by 0.03 cosine, so no threshold distinguishes a hit from a miss.
Two approaches survive contact with a real corpus. First: at index time, send each figure and table to a vision model with its caption, section heading and adjacent paragraph, and store the generated description as text. One embedder, one vector space, BM25 still working over part numbers and serials, and failures you can read — you can open the description and see what the system believes the figure says. Second: late-interaction page embeddings in the ColPali family, which encode the rendered page as roughly a thousand patch vectors scored against query tokens by MaxSim. Retrieval is better and storage is about two orders of magnitude larger, and you need a multi-vector store. Start with descriptions. Move to late interaction after you've measured that descriptions, not the parser, are the ceiling.
The unit you retrieve on and the unit you generate from should not be the same object. A description is lossy on purpose: it says "bar chart of quarterly revenue by region" and drops that Q3 EMEA was 4.2 million, which is the number the user asked for. So store a pointer beside every description — source file, page number, bounding box — and at generation time send the pixels: the cropped region, or the full page rendered around 150 DPI when the surrounding table headers matter. Retrieve on text, answer from the image.
That pointer is also the citation. A multimodal answer with no visual citation is unverifiable, and in regulated work unverifiable means unusable regardless of accuracy — the reviewer has to click the footnote and land on the highlighted box on page 214. Deduplicate at the same time. The same schematic reproduced across four appendices will otherwise fill top-k with near-identical copies; collapse crops above about 0.98 similarity into one hit with multiple locations, or the single revised drawing that actually differs gets pushed off the end of the list.
Index-time captioning is a line item, not a rounding error. At roughly 1,200 input and 800 output tokens per figure, a 40,000-page corpus averaging two figures a page is 80,000 vision calls — and it isn't one-time, because you re-run the whole thing every time you change the description prompt or the model behind it. Query time punishes carelessness harder: a full page at high detail costs 1,500 to 2,500 tokens, so eight pages is a 20,000-token request that pushes time-to-first-token past three seconds. Retrieve 20 candidates, rerank on text, send images for the top three.
Then measure retrieval and generation separately, or you'll debug the wrong half. Build about 120 real questions whose answers live only in a figure or a table, never in the prose, and score "was the correct region in the retrieved context" apart from "did the model read it correctly." At 60 percent recall, no model swap saves you and the work is back in the parser. Reserve a fifth of the set for questions with no answer in the corpus: a system that invents an axis value is worse than one that says the chart isn't there.
A software studio that ships and maintains its own products — KeepChats, Gwora and MoveProof — and builds the same way for clients. Founded and led by codewithumar.
Talk to the studio →We build software worth keeping — for clients, and for ourselves.
Founded & led by codewithumar