Why do my search results get worse as I add more data?
Because the numbers that decide how your search behaves were chosen when the corpus was small, and almost all of them are counts rather than fractions. An approximate index explores a fixed number of candidates per query, so the same budget covers a shrinking share of a growing corpus. Near-duplicate content accumulates faster than original content and competes for the same ten slots. A similarity floor that rejected weak matches at ten thousand items admits them at ten million, because every score band holds more items than it used to.
Five mechanisms account for most instances of this, and they have different fixes, so the first job is finding out which one you have. Each test below is one run over a fixed query set, and the cheapest one comes first.
Is it really getting worse, or does it only feel worse?
The complaint usually arrives as memory. Somebody says search used to find this, and nobody recorded what it used to return. Build the instrument before changing any setting, because every fix below trades something away and you cannot price the trade without a baseline.
The control is the measurement most teams skip, and it splits the problem in one run. When exhaustive search finds the right answer and your index does not, the index or its settings are at fault. When exhaustive search misses it too, the embedding, the chunking or the extraction is at fault, and index tuning cannot reach that.
One more cut, free: group the failing queries by the ingest date of the document they were looking for. Recent documents failing points at the ingest path. Documents failing that used to work points at the index.
The five reasons, and the test for each
Your index explores a fixed number of candidates and the haystack grew
Every approximate index bounds the work it does per query, and the bound is a count. In an HNSW graph it is
ef_search, the size of the candidate list kept
during traversal. In an IVF index it is nprobe, the number of cells opened.
Neither is expressed as a share of the corpus, so both cover less of it as it
grows, and the chance that the true nearest neighbour sits outside the explored
region rises with the corpus size.The original HNSW paper frames its guarantees as scaling behaviour rather than exactness (Malkov and Yashunin, 2016), and the recall-against-latency curves published by ANN-Benchmarks shift with the dataset, which is the measured form of the same point.
Test: multiply
ef_search or nprobe by four and re-run the query set. If
recall recovers, this is your cause, and the same run has just priced the fix in
latency.Fix: derive the search budget from the corpus you have now, off your own recall and latency curve, and re-derive it when the corpus doubles. The teams who hit this are the ones who set the budget once at launch.
Near-duplicates are taking your top ten
A re-encode, a crop, a re-upload at a different resolution and a platform-specific render are all genuinely close to the query, because they are genuinely close to each other. The ranking keeps working correctly while the result set gets worse: ten slots now carry three distinct answers and seven copies.
Growth makes this worse for a structural reason. Original content arrives at the rate your business produces it. Copies arrive at the rate your pipelines, your vendors and your own re-publishing produce them, which is usually faster.
Test: for each failing query, count distinct sources in the top 20 instead of reading the list. A top 20 holding 6 distinct assets has this problem whatever the relevance numbers say.
Fix: two places. At ingest, drop exact copies on a content hash and near-copies on a perceptual hash or an embedding distance. At query time, collapse whatever got through, by field or by similarity. Perceptual Image Hashing covers how a near-copy is recognized; Diversity-Aware Retrieval covers the query-time half.
The coarse structure was trained on data that no longer looks like your data
IVF and product quantization both learn from a sample. Centroids are fit once, usually against whatever was in the index the day it was built. Data arriving later lands unevenly against those centroids, posting lists skew, and opening
nprobe of
them covers less of the relevant space than the arithmetic assumed. Quantization
codebooks go stale the same way: the residuals they were fit to stop resembling the
residuals they are asked to encode, so the reconstruction error that was acceptable
at build time is not the error you have now.Test: look at posting list sizes. Roughly balanced lists mean the partition still fits the data. A handful of lists holding most of the vectors means it does not.
Fix: retrain on a current sample and rebuild. Graph indexes such as HNSW have no equivalent of this failure, which is part of what their memory cost buys.
A similarity floor that was right at 10,000 items is wrong at 10 million
Score cutoffs get set by looking at a handful of queries and picking a number that looked clean. Growth moves the distribution underneath that number, because more items means more items inside every band, so a floor that admitted two weak results admits forty.
There is a sharper version of this trap and we have measured it. On a 119-page corpus, questions the index could NOT answer scored a mean top similarity of 0.845, against 0.861 for questions it could. Those two populations overlap almost completely, so a threshold tuned to reject unanswerable queries rejects answerable ones at close to the same rate. A similarity score reports closeness in embedding space, and it carries no information about whether an answer is present.
Test: plot the top-1 score distribution over your query set today and compare it with the distribution you tuned the floor against. If nobody recorded that distribution, you have the finding.
Fix: cut by rank where you can. Take the top k, rerank, then cut. Where an absolute floor is genuinely required, re-derive it per corpus and check what sits upstream of it, because a normalization stage rescales scores relative to the result set being normalized, which makes any threshold after it a threshold against a moving target.
Your filter is starving the candidate pool before the ranking runs
Filtered vector search has to reconcile two structures that know nothing about each other. A graph index walks edges between points near each other in embedding space, and a filter decides which points are allowed. When most points in a neighbourhood fail the filter, the traversal spends its budget on rejected candidates and returns a short list or an empty one, while thousands of matching documents sit in the index unvisited.
Growth makes this worse with the filter unchanged, because a filter naming one brand, one customer or one month becomes more selective in relative terms as everything else grows around it. This is the cause where "it worked last year" is most often literally true.
Test: run the query with no filter and count how many of the top 100 would have passed. Then run it filtered and compare the result count against the number of documents the filter matches in total. A filter matching 40,000 documents that yields 3 results is this.
Fix: engine-specific. Some implementations build additional edges so filtered subgraphs stay connected, some fall back to brute force below a selectivity threshold, and some partition by the filter field so the filter selects an index instead of rejecting candidates. The configuration to get rid of is post-filtering a fixed top k, which quietly returns three results and reports success.
Which order should I test these in?
1. The exhaustive control. Separates index problems from embedding problems, with nothing changed in production. 2. Distinct sources in the top 20. Finds duplicate crowding by reading, with no extra runs. 3. Four times the search budget. Finds a stale candidate budget and prices it. 4. The same queries with the filter removed. Finds selectivity starvation. 5. The top-1 score distribution against the one you tuned on. Finds a stale floor.
Four of the five change nothing in production. Keep the query set and the numbers from every run: the next time this happens, the baseline is the thing you will wish you had.
How does this work in Mixpeek?
At ingest, object dedup runs on a content hash, so the same bytes arriving twice resolve to one object rather than two documents competing for the same result slots.
skip_duplicates controls it per upload and content_hash is the key; the
deduplication docs cover the
strategies.At query time the behaviour lives in retriever stages, each a step you add, reorder or remove in a pipeline:
| stage | what it addresses | the limitation to know |
[deduplicate](/retrievers/deduplicate) | repeats in the result set, by field or by content similarity | similarity dedup keeps whichever copy survives the pass, so the better copy can be the one dropped. Dedupe on a field you trust when the choice matters |
[mmr](/retrievers/mmr) | a top ten that covers the same ground repeatedly | it trades relevance for coverage by construction, so the single best result can move down |
[score_threshold](/retrievers/score_threshold) | weak results you would rather not return at all | absolute, so it needs re-deriving per corpus, and it cannot separate an unanswerable query from an answerable one |
[score_normalize](/retrievers/score_normalize) | comparing results produced on different scales | it rescales relative to the result set, so a threshold placed after it thresholds a moving target |
[rerank](/retrievers/rerank) | ordering a candidate set more carefully than the index can | it reorders what retrieval already found, so it cannot recover recall |
Frequently asked questions
Will a bigger embedding model fix this?
Only if the exhaustive control failed. A better model changes where documents sit in embedding space, which helps when the right answer was never close to the query to begin with. Four of the five causes above happen after the embedding is already correct, and a larger model makes two of them harder by adding dimensions to an index that is already struggling to cover the corpus.
Should I just use a flat index and search everything?
For corpora up to a few hundred thousand vectors with a relaxed latency budget, yes, and it removes three of the five causes outright. The cost grows linearly in corpus size and in dimensions, so the point where it stops being viable depends on your latency budget and your hardware. Measure your own crossover rather than adopting somebody else's number.
Does adding data ever actually improve results?
Yes, which is what makes this hard to spot. More coverage means more queries have a correct answer present at all, so aggregate satisfaction can rise while recall on the queries that already worked falls. Track the two populations separately: queries whose answer existed in the old corpus, and queries whose answer exists only in the new one.
How often should I rebuild the index?
When one of the tests above says the structure has drifted: posting lists out of balance, recall on the frozen query set dropping, or the score distribution moving. A rebuild on a fixed schedule costs the same whether anything changed or not, and it papers over the drift instead of reporting it.
Is this the same thing as model drift?
No. Model drift is the world changing underneath a fixed model, so the embeddings stop describing what your users mean. Everything on this page happens with the embeddings held constant. The exhaustive control separates the two, because drift shows up as exhaustive search missing the answer as well.
Further reading
ef_search and nprobe do