Why does my document search miss the answer that is in a chart or table?
Because most document search indexes the text layer of a file, and the answer you are looking for is not in the text layer. A chart is a drawing or an embedded image: its bars, lines and axis labels reach the index as a caption at best and as nothing at worst. A table survives extraction as a stream of cell values with the row and column structure stripped out, so "Q3 revenue for the EMEA region" becomes a number floating next to other numbers with no way to tell which row it belonged to. The search engine did its job on the words it was given. The words it was given did not contain the answer.
The fix is to index what the document shows as well as what it says. That means one of three things, and often two of them together: extract charts and tables into structured text before indexing, index page images directly with a visual document retrieval model, or have a vision-language model describe each figure and index the description. The rest of this guide explains how each one works, what it costs, and how to find out which one your corpus needs.
Where does the answer go when a PDF is turned into text?
A PDF stores drawing instructions, embedded images and text runs positioned on a page. A text extractor keeps the text runs and discards the rest. What happens to a chart or a table depends on how it was produced:
| How the figure was made | What the text extractor sees | What is lost |
| Chart drawn as vector graphics (most spreadsheet exports) | The axis tick labels and the legend, as loose words | Every bar, line and point value; which label belongs to which series |
| Chart pasted as an image (screenshots, slides) | Nothing, or an alt text if one was set | The whole chart |
| Table drawn with ruled lines and positioned text | The cell strings in reading order, one line at a time | Row and column membership, merged headers, units in the header row |
| Table in a scanned page | Nothing until OCR runs; then the same loss as above | Structure, plus any OCR errors on digits |
| Number quoted in the body text | The number and its sentence | Nothing, which is why body text is findable and figures are not |
How do I check whether my own corpus has this problem?
Do this before choosing a fix, because the fix depends on how much of the answer surface is visual.
1. Take twenty questions that were answered from figures. Pull them from support tickets, analyst requests or your own evaluation set. The question should be answerable only by reading a chart or a table. 2. Run them through your current search. Record whether the right page ranks in the top five, and whether the returned passage contains the value. 3. Look at the extracted text for the failing pages. Most extractors can dump their output. If the chart's values are absent and the table has collapsed into one line per cell, you have found the gap and you know which shape it takes. 4. Count figures per page across the corpus. Financial reports, scientific papers, engineering documents and slide decks are figure-heavy; contracts and correspondence are not. The count tells you whether the fix is worth running on every page or only on the pages that have figures.
If the twenty questions mostly fail at step 2 and the extracted text at step 3 is missing the values, your search is not broken. Its input is.
What are the three ways to make charts and tables searchable?
Extract the figure into structured text first
Document parsers such as Docling, PaddleOCR-VL and DeepSeek-OCR detect tables and reconstruct them as rows and columns, and the better ones emit charts as data tables or Markdown as well. The output is text, so it drops straight into the search stack you already have, and a table cell keeps its row header and column header attached, which is exactly what makes "Q3 revenue for EMEA" findable. The cost is an extra parsing pass per page and a dependence on the parser getting the layout right; complex merged headers and low-resolution scans are where it fails. The Docling technical report describes the layout and table-structure models involved, and How OCR Actually Works covers the detection, recognition and reading-order steps underneath.
Index the page as an image
Visual document retrieval models such as ColPali and ColQwen embed the rendered page image, not its text, so a bar chart and a table are searchable as what they look like. A query such as "which region had the highest Q3 revenue" is embedded in the same space and matched against page patches, which means the chart participates in retrieval without ever being converted to words. The ColPali paper reports strong results on the ViDoRe benchmark of visually rich documents, and the models are open weights: ColPali v1.3 and ColQwen2.5. The trade-offs are storage, since each page becomes many vectors, and the fact that retrieval finds the page while a second step still has to read the value off it. Visual Document Retrieval: How AI Agents Search Documents Without OCR covers the mechanism and the compression options.
Have a vision-language model describe the figure
A vision-language model can be pointed at each chart or table crop and asked to write out what it shows: the series, the axis units, the values, the trend. Index that description alongside the body text. This catches the cases parsers get wrong, such as a chart with no underlying data table, an annotated diagram, or a photograph of a whiteboard. It is the most expensive option per page, its output should be checked for invented numbers, and it works best when the model is also asked to emit the values as a table, since prose descriptions are harder to match against a numeric query. Benchmarks such as ChartQA and DocVQA measure exactly this skill, and Optical Context Compression explains why reading a page as pixels can be cheaper than it sounds.
Which one should I use?
| Approach | Best when | Cost per page | Main limitation |
| Structured extraction (Docling, PaddleOCR-VL, DeepSeek-OCR) | Tables dominate; you need exact cell values; your search stack is text-only | One parsing pass, CPU or small GPU | Complex layouts and poor scans break table structure |
| Page-image retrieval (ColPali, ColQwen) | Figures dominate; questions are about what a page shows; you can add a vector index | Multi-vector embedding per page, more storage | Finds the page, does not read the value out |
| VLM figure descriptions | Mixed corpora; charts without data tables; diagrams and photos | One model call per figure, the highest of the three | Can invent values; needs a check |
How should I chunk a document that has figures?
Keep each figure with its caption and the paragraph that refers to it, in one chunk. Splitting on a fixed token count will put the caption in one chunk and the table body in the next, which recreates the wrong-part problem after you have paid to extract the table. A table should be one chunk with its header row repeated if it has to be split; a chart description should carry the figure number and the section heading. Multimodal Chunking Strategies goes through the rules for each document type.
How does this work in Mixpeek?
Mixpeek runs the three approaches as extractors over the same documents in your object storage, so you do not choose between them at ingestion. A document collection can run a parser that emits tables as structured text, a visual document retrieval model that embeds each page image, and a vision-language description of detected figures, with each output stored as a searchable field on the same document. A retriever then combines them: lexical search over the extracted table text for exact values, a feature search over page embeddings for questions about what a page shows, and a rerank stage across both. The result set carries the page number and the figure crop, so the reader or the agent sees the chart the answer came from.
If you want to see the models involved, DeepSeek-OCR-2, Granite Docling and PaddleOCR-VL 1.6 are the parsing side, and ColPali and ColQwen2.5 are the page-image side. The comparisons of hosted options are in best document parsing tools, best PDF extraction tools and best OCR APIs.
Frequently asked questions
Does OCR fix this?
OCR turns pixels into characters. It recovers the words on a scanned page, including the numbers inside a chart or table, but it does not recover the structure that gives those numbers meaning. A table needs a table-structure model on top of OCR; a chart needs either a chart-to-table step or a visual index. Treat OCR as the first step, not the fix.
Do I need a vision-language model for every page?
No. Run the cheap parser everywhere, and reserve the model for pages where the parser reports a figure it could not structure. On most corpora that is a small fraction of pages, and the model cost scales with the number of charts, whatever the number of documents.
Will a bigger embedding model solve it?
Not on its own. A text embedding model can only embed the text it is given, and the text is missing the values. Changing the extractor changes what the embedding model sees; changing the embedding model does not put the chart back.
How do I know the extracted table is correct?
Spot-check against the source. Pick tables with merged headers, footnotes and units in the header row, since those are the shapes parsers get wrong, and compare a handful of cells by hand. Parsers that emit a confidence score per table let you route the low-confidence ones to a vision-language model or to a person.