The Short Answer
Bad retrieval almost always fails in one of five layers — the corpus, the embedding space, the query, the filters, or the fusion/rerank stage — and the fastest way to find which one is to trace a single failing query through each layer with its intermediate outputs, rather than tweaking prompts or swapping models blind. Similarity scores alone will not tell you: a cosine score is only meaningful relative to the query's own score distribution, so debugging is about comparing candidates within a query, checking what each pipeline stage received and dropped, and probing the embedding space directly.
Which layer is failing? The five-layer checklist
| Layer | Symptom | First check |
| Corpus / ingest | The right document can never be found | Is it indexed at all? Was its text/frames actually extracted, or did ingest silently skip it? |
| Embedding space | Results are topically adjacent but wrong | Embed the query and the expected document; check the expected doc's rank among the query's neighbors |
| Query formulation | Works when rephrased | Compare retrieval for the raw query vs a rewritten/expanded one |
| Filters | Fewer results than expected, or none | Count candidates before and after each filter; check for silent drops on missing fields |
| Fusion / rerank | Right docs retrieved, wrong order | Inspect per-source contributions before and after fusion |
Why can't I just read the similarity scores?
Because raw scores are not calibrated. Cosine similarities live in model-specific bands (a SigLIP 0.3 can mean a strong match while an OpenAI-embedding 0.3 is noise), scores drift across model versions, and cross-modal scores sit in systematically different ranges than same-modality ones — the modality gap. Two rules of thumb: compare scores only WITHIN a single query's result list, never across queries or models; and look at the gap structure (a cliff after result 3 usually means the tail is noise; a flat band means the query didn't discriminate at all). If you need scores that behave like probabilities, that is a separate engineering step — calibrating similarity scores.
How do I trace a query through a multi-stage pipeline?
The single most effective technique: capture the document COUNT and a few document IDs at every stage boundary, then find the boundary where the expected document disappears. Real production example from our own incident log: a two-stage search-then-summarize pipeline where search returned 20 documents and the summarize stage received 1 — an execute-level filter was being re-applied to every stage, and the intermediate documents did not carry the filtered field, so 19 of 20 were silently dropped between stages. No error, no warning; the answer just looked mysteriously thin. The generalizable checks at each boundary: how many documents entered and left, which fields they actually carried (payload-lean intermediates are a classic silent killer), and whether the stage served a cached result or computed fresh — stale caches make you debug a pipeline that is not the one running.
What belongs in a retrieval explain plan?
When you formalize the trace above, you get retrieval's equivalent of a database explain plan. A useful one answers five questions per stage: how many candidates entered and exited (and why the delta — filter conditions with hit/drop counts); which features or sources contributed each surviving document, with per-source scores before fusion; what the stage cost (latency, and money for LLM stages); whether the output was cached and from when; and what the final ordering decision was (fusion weights or rerank scores). If your stack cannot answer those, you are debugging blind — instrument before you tune.
How does this work in Mixpeek?
Every retriever execution returns per-stage telemetry:
stage_statistics carries duration and a cache_hit flag per stage (cache provenance is stamped at every level, so a served-from-cache answer is always distinguishable from a fresh one), and a budget block accounts for what the execution cost. Filters bind only to filter-type stages — the silent inter-stage decimation above is a class of bug the pipeline contract now rules out — and each stage's inputs and outputs are inspectable in Studio's retriever workbench, where you can run one query and read the stage-by-stage trace instead of reconstructing it from logs. Ranking-quality problems that survive tracing usually belong to the ordering layer: see two-stage retrieval and reranking for the rerank half, and retrieval feedback loops for turning debugged failures into training signal so the same miss stops recurring. If you are still choosing the stack underneath, start at MVS or the retriever stage catalog; rate economics live on pricing.