The Short Answer
Query planning is deciding the order your retrieval stages run in, how many candidates each one hands to the next, and which work can be pushed down into the index instead of done afterwards in your application. A multimodal pipeline typically chains a filter, a vector search, sometimes a keyword search, a fusion step, and a reranker. The same five stages can differ by two orders of magnitude in latency and cost depending only on their order and their candidate budgets, with identical results at the top of the ranking.
The three decisions that matter most are: filter before the vector search rather than after it whenever the filter is selective and the index supports it, keep the candidate budget as small as the reranker will tolerate because reranking dominates cost, and stop early on queries where the score gap already decided the answer. Everything else in this guide is detail underneath those three.
Why A Multimodal Pipeline Needs Planning At All
A single vector search does not need a plan. You embed the query, you search, you return the top k. The moment you add a second retrieval path, a metadata filter, or a reranker, you have created a small query optimization problem, and the naive order is usually the expensive one.
The reason is that the stages have wildly different unit costs. Approximate nearest neighbor search over a few million vectors is cheap, on the order of single-digit milliseconds, because the whole point of an ANN index is to avoid looking at most of the data. A cross-encoder reranker is not cheap. It runs a transformer forward pass per candidate, so reranking 500 candidates costs roughly 500 times reranking one. Between those two sits metadata filtering, which is nearly free when the field is indexed and brutally expensive when it is not, because an unindexed filter degenerates into a scan.
Put those together and the cost of a pipeline is dominated by whichever stage sees the most candidates. Planning is mostly the discipline of making sure the expensive stages see the fewest.
Multimodal makes this sharper for a reason that has nothing to do with the algorithms. Video and audio explode the row count. One hour of video segmented into scenes might produce two thousand documents, each with its own embedding, and a library of ten thousand hours is twenty million rows before you have indexed a single frame-level feature. The same pipeline that felt instant on a hundred thousand text chunks is a different problem at that scale, and the stage ordering that was irrelevant becomes the difference between a product and a timeout.
Filter First Or Filter After? The Pre-Filter Versus Post-Filter Decision
This is the first real decision and the one people get wrong most often. It is also the decision that multi-stage retrieval leaves open.
Post-filtering runs the vector search first, takes the top k, then discards the results that fail your metadata predicate. It is simple and it is what you get by default if you filter in application code. It has a failure mode that is easy to miss in testing: if your filter is selective, most of the top k gets thrown away and you are left with far fewer results than you asked for. Search for the 100 nearest neighbours, keep only those from one brand, and you may end up with four. The fix people reach for is to over-fetch, searching for 1000 to survive the filter, which multiplies the work and still gives no guarantee.
Pre-filtering applies the predicate first and searches only the surviving subset. You always get k results if k exist. The cost is that it constrains the index traversal, and how well that goes depends entirely on the implementation.
The naive version of pre-filtering is a disaster on a graph index. HNSW works by greedy traversal through a proximity graph, and if you mask out most of the nodes, the graph fragments. The walk gets stranded in a region with no permitted neighbours and recall collapses, often silently, because the search still returns something. This is the trap: a filtered HNSW query that returns plausible results with terrible recall looks exactly like a filtered query that worked.
Production systems handle this in one of three ways, and it is worth knowing which one you are using:
The rule that falls out: pre-filter when the predicate is selective and indexed, post-filter when the predicate matches most of the corpus. A filter that keeps 90% of your data is not worth pushing down, because the traversal constraint costs more than the candidates saved.
Selectivity, And Why Your Planner Is Probably Guessing
Every decision above depends on one number: what fraction of the corpus survives the filter. A real database keeps histograms and cardinality statistics so the planner can estimate that before running anything. Most vector stores keep almost nothing.
That gap is why vector search "query planning" today is mostly conventions and hand tuning rather than a cost-based optimizer. You are the planner. The practical substitute is to measure the selectivity of your common predicates once, write the numbers down, and let them drive the pipeline shape:
Those boundaries are not universal. They shift with index type, dimensionality and how correlated the filter is with the query semantics. The point is to have measured boundaries at all, because the alternative is a pipeline whose shape was chosen by whoever wrote it first.
Correlation deserves its own warning. Selectivity estimates assume the filter is independent of the query, and in multimodal retrieval it frequently is not. Filtering to one content type and searching for a concept that only appears in that content type is a different distribution from the corpus average, and an estimate drawn from global statistics will be wrong in the direction that hurts.
Candidate Budgets: The Number That Decides Your Bill
Between every pair of stages there is a number: how many candidates the first hands to the second. Those numbers are the single biggest lever on cost, and they are usually set once and never revisited.
Work backwards from the reranker, because it is the expensive stage. A cross-encoder scoring 100 candidates at, say, 4ms each is 400ms of GPU time per query. At 200 candidates it is 800ms. The retrieval in front of it might be 8ms. So the budget handed to the reranker is not one parameter among many; it is approximately the whole cost curve.
The question is how small that budget can get before quality drops, and that has an answer you can measure rather than guess. Take a query set with known relevant documents. Retrieve a deliberately large candidate pool, say 1000. Compute recall at each cut point: how often does the true best result appear in the top 50, the top 100, the top 200? That curve almost always flattens, and where it flattens is your budget. Spending beyond the flattening point buys nothing, and most pipelines are well past it because nobody measured.
Two refinements worth knowing:
Score-gap early termination. If the top candidate's score is far above the second, reranking will not change the order. Some pipelines skip the reranker entirely when the gap exceeds a threshold, which cuts cost on exactly the easy queries that make up most traffic. The risk is that raw ANN scores are not calibrated, so a "large gap" on one query is not comparable to a large gap on another. Calibrate before you trust a threshold.
Adaptive budgets. Rather than a fixed 100 for every query, spend more on queries that look hard. Ambiguous or short queries get a bigger pool, precise ones get a smaller one. This is genuinely effective and genuinely fiddly, and it is worth doing only after the fixed budget has been tuned properly.
Stage Ordering Beyond The Filter
Once filtering is settled, a few ordering rules generalize.
Cheap and selective goes first. This is the oldest rule in query optimization and it survives contact with vectors intact. A stage that eliminates candidates for less than it costs to run should run as early as possible.
Expensive and non-selective goes last. Rerankers reorder, they rarely eliminate. Putting a reranker anywhere but at the end means paying to reorder candidates you are about to throw away.
Fusion goes after retrieval and before reranking. Merging a keyword ranking and a vector ranking with Reciprocal Rank Fusion is arithmetic on ranks, so it is nearly free, and doing it first means the reranker sees one merged pool rather than two.
Deduplicate before you rerank, not after. In multimodal corpora near-duplicates are everywhere: the same scene from two encodes, the same product shot on three pages. Reranking twenty copies of one clip spends twenty forward passes to produce one useful result. Collapsing on a content hash or a grouping key before the reranker is one of the highest-return changes available, and it also fixes the user-visible problem of a results page that shows the same thing repeatedly.
Grouping is worth calling out because it does double duty. A reduce stage that collapses results by a content key gives you diversity in the output and a smaller pool for the expensive stage at the same time. That is rare; most quality improvements cost latency.
What Changes When The Corpus Is Video, Audio, Or Images
Text pipelines and multimodal pipelines share the algorithms and differ in the constants, and the constants change the plan.
Granularity is a planning decision, not just a modelling one. You can index a video at the clip, scene, shot, or frame level. Finer granularity gives better localization and multiplies your row count. A plan that works at scene level may be untenable at frame level purely on candidate volume, so pick the granularity your latency budget can support and use a coarse-to-fine strategy if you need frame precision: retrieve at scene level, then search within the winning scenes.
Multiple vectors per document break the top-k assumption. If a scene has separate visual, audio and text embeddings, "the top 100 documents" is ambiguous. You can search each space and fuse, or search one and rerank with the others. Fusing three rankings of 100 gives you up to 300 distinct documents to rerank, which is the arithmetic behind budget-aware multi-vector retrieval, so the budget arithmetic has to account for the union rather than the per-space k.
Time ranges are a filter you can push down. Restricting to a time window, a speaker, or a shot boundary is a metadata predicate like any other, and the same pre-filter rules apply. Media queries are often much more selective than text queries because the metadata is richer, which is exactly the regime where pushing the filter down pays.
Embedding extraction can dominate the query too. For text the query embedding is microseconds. For an image-by-example or audio query it is a model forward pass, and it happens before retrieval starts. If your product supports search-by-image, that cost is in every query's critical path, so cache aggressively on repeated inputs.
How To Tell Whether Your Plan Is Actually Working
Three measurements, and the third is the one people skip.
Per-stage latency. Not total latency. A pipeline at 900ms tells you nothing; the same pipeline broken into 6ms retrieval, 40ms filter and 850ms rerank tells you exactly what to fix. If your retrieval layer does not report per-stage timings, that is the first thing to add. See debugging retrieval results for how to read them.
Candidate counts in and out of every stage. A stage that takes 500 candidates and returns 498 is not filtering, it is charging you. A stage that takes 500 and returns 3 was probably the wrong stage to run last.
Recall against an exact baseline. This is the one that gets skipped, and it is the one that catches the silent failures. Run your query set against exact brute-force search, no ANN and no filters, and treat that as ground truth. Evaluating multimodal retrieval covers how to build that query set. Then measure what fraction your production plan recovers. This is how you find a filtered traversal that fragmented the graph, because that failure produces confident, plausible, wrong results and no error anywhere.
Run that baseline again whenever you change the plan. Recall regressions from a stage reorder do not announce themselves.
When Query Planning Is The Wrong Thing To Work On
Planning tunes constants. It cannot fix a pipeline that is retrieving the wrong things.
If recall against the exact baseline is already poor, your problem is the embedding model, the chunking, or the granularity, and no amount of reordering will recover a document the retriever never surfaces. Fix retrieval quality first and optimize the plan after, or you will carefully optimize your way to a fast wrong answer.
If your corpus is small, skip all of this. Under roughly a hundred thousand vectors, exact search is fast enough that ANN indexes and their filtering pathologies are complexity you are buying for nothing. Do you need a vector database? works through that threshold. The threshold is higher than most people assume.
And if latency is dominated by something outside retrieval, network round trips, a slow LLM call at the end, cold model loading, then per-stage retrieval tuning is rearranging a small fraction of the total. Measure end to end before you optimize a part.
How This Maps To Mixpeek
Mixpeek retrievers are explicit stage pipelines, so the plan is a data structure you write rather than a behaviour you infer. A pipeline is a list of stages with a
stage_type of filter, sort, reduce, apply or enrich, executed in order, and the candidate budget between them is the final_top_k on each stage.That makes the decisions in this guide concrete. Pre-filtering is a
pre_filters block on the feature_search stage, which applies the predicate as part of the search rather than after it. Deduplication before reranking is a reduce stage with stage_id: "group_by", using group_by_field and max_per_group to collapse near-duplicates into one representative each. Candidate budgets are the top_k on each search and the final_top_k on the stage.Two things make the plan easier to debug than it used to be. Stages validate loudly: an unrecognized
stage_id returns a 422 that lists the available stages rather than being ignored. And as of August 2026 a stage that runs but cannot do its job explains itself in the response warnings array, so a group_by that collapsed nothing tells you the field was absent instead of handing back a flat list you have to interpret. When a Mixpeek response is surprising, read warnings before you read your data.For the storage side of the same problem, Mixpeek's vector store keeps the index on object storage rather than in a separate database, which is what makes the filter-pushdown and tiering decisions in this guide a property of the system rather than something you assemble yourself. The architecture is covered in running a vector database on S3 object storage, and the cost side in what it costs to make a media library searchable.
Frequently Asked Questions
Should I filter before or after a vector search?
Filter before the vector search when the predicate is selective and the field is indexed, and filter after when the predicate keeps most of the corpus. Pre-filtering guarantees you get k results, because the search only ever looks at rows that pass. Post-filtering searches first and discards afterwards, so a selective filter can leave you with a handful of results when you asked for a hundred. The catch with pre-filtering is that a naive implementation fragments a graph index like HNSW, stranding the traversal in regions with no permitted neighbours and quietly destroying recall. Check whether your index does filtered graph traversal, partitions on the filter field, or falls back to an exact scan when the surviving set is small.
How many candidates should I pass to a reranker?
Measure it rather than guessing, because the answer is usually far smaller than the default. Take a query set with known relevant results, retrieve a deliberately large pool such as 1000, and compute recall at each cut point: how often the true best result appears in the top 50, top 100, top 200. That curve flattens, and the flattening point is your budget. Anything past it is money for nothing. This matters more than any other parameter because a cross-encoder costs one transformer forward pass per candidate, so the reranker budget is approximately the whole cost curve of the pipeline.
Why is my filtered vector search returning bad results with no error?
Almost certainly because the filter fragmented the index traversal. A graph-based ANN index walks from node to neighbouring node, and masking out most of the nodes leaves the walk stranded in a region where nothing is permitted. The search still returns something, so nothing fails and nothing logs. The only way to catch it is to compare against exact brute-force search over the same filtered set and measure what fraction of the true nearest neighbours your production query recovers. Confident, plausible, wrong output with no error is the signature of this failure, and it is why a recall baseline belongs in your test suite rather than in a one-off investigation.
Does query planning matter if my corpus is small?
Under roughly a hundred thousand vectors, no. Exact search at that scale is fast enough that an ANN index buys you little and costs you the filtering pathologies described above. The threshold is higher than most teams assume, and a lot of pipeline complexity gets adopted well before it earns its place. Multimodal corpora cross the threshold faster than text ones, because one hour of segmented video can produce a couple of thousand rows, so ten thousand hours is twenty million rows before any frame-level features exist.
What is the difference between query planning and query optimization in vector search?
In a relational database, query optimization is automatic: the planner reads table statistics, estimates selectivity, and picks a plan without being asked. Vector stores mostly do not keep those statistics, so there is no cost-based optimizer to consult and the plan is whatever you wrote. That makes "query planning" in vector search a design activity rather than a runtime one. You measure the selectivity of your common predicates once, you decide the stage order and the candidate budgets from those measurements, and you re-check them when the corpus shape changes.
How do I deduplicate results before reranking?
Collapse on a content key with a grouping or reduce stage placed before the reranker. Near-duplicates are endemic in multimodal corpora, where the same scene appears across two encodes and the same product shot across three pages, and reranking twenty copies of one clip spends twenty forward passes to produce one useful result. Grouping before the expensive stage does two jobs at once: it shrinks the candidate pool you pay to rerank, and it gives the user a results page that does not repeat itself. Very few quality improvements also reduce cost, which makes this one worth doing early.