How do I know if my search results got better?
Run the old and the new search system on the same fixed set of queries whose relevant results you have already labeled, score both with the same ranking metrics, and compare them query by query. If the new system wins on most queries and a paired significance test says the average gain is larger than the noise between queries, the change helped. If you only eyeball a few searches, or compare two numbers computed on different query sets, you cannot tell a real improvement from luck.
That procedure is called offline evaluation, and it has been the standard way to compare search systems since the TREC evaluations began at NIST (TREC overview). This guide covers the search side: how well a retriever ranks results. If you are grading generated answers in a RAG system, read how to evaluate a RAG pipeline after this one.
What do I need before I can compare two search systems?
You need three things, and all three stay fixed while you compare:
The two systems differ in exactly one thing: the change you are testing. Same corpus, same index snapshot, same filters. If the corpus changed between runs, you are measuring the corpus.
Where do test queries come from, and what bias does each source add?
Real query logs. The best source, because they are what users actually type. Sample across head (frequent), torso and tail (rare) queries. Sampling uniformly from distinct queries overweights the tail; sampling from traffic overweights the head. Pick on purpose and write down which you chose.
Click and interaction data. Clicks tell you which results users found worth opening, so they can produce labels without a human judge. They are biased. An eye-tracking study by Joachims et al. (2005) found a "trust bias" (users click higher-ranked links partly because they are higher) and a "quality bias" (clicks depend on how good the other results are). Their conclusion: clicks are informative but biased, and relative preferences derived from clicks are more reliable than treating a click as an absolute relevance label. A dataset mined from clicks also only contains documents your current system already showed, which favors the current system in any comparison.
Synthetic queries. An LLM writes a question for each document, and that document becomes its relevant answer. This is cheap and covers content nobody has searched for yet. It was popularized for training retrievers (InPars, arXiv 2202.05144). For evaluation it has two weaknesses: generated questions tend to reuse the document's own words, which flatters keyword matching, and each query gets exactly one relevant document even when several would satisfy it.
A practical mix: real queries from logs as the core, a slice of synthetic queries for new content, and click-derived labels only as a cross-check.
Should relevance labels be graded or binary?
Binary labels (relevant or not) are faster to collect and are all that recall, precision and MRR use. Graded labels (for example 0 = not relevant, 1 = partly, 2 = relevant, 3 = exactly what I wanted) let a metric reward putting the best result first rather than any relevant result first. Järvelin and Kekäläinen (2002) introduced cumulated gain measures for this reason, using graded judgments on a four-point scale.
Graded labels cost more per judgment and judges disagree more on the middle grades. Write a one-paragraph rubric with an example for each grade before anyone labels anything.
Recall@k vs nDCG vs MRR: which metric should I use?
Each metric answers a different question about the same ranked list.
| Metric | Question it answers | Uses grades? | Use it when |
| Recall@k | Of all the relevant documents, how many made the top k? | No | Users or a downstream reranker need every match (compliance, a first-stage retriever feeding a reranker) |
| Precision@k | Of the top k, how many are relevant? | No | Users see a short, fixed list and junk is costly |
| MRR | How far down is the first relevant result? | No | One good answer is enough (navigational search, agents reading the top hit) |
| nDCG@k | How close is the ordering to the ideal ordering? | Yes | Users scan a page and some results are better than others (product, media, document search) |
The discounted form follows Järvelin and Kekäläinen (2002). Many libraries use an exponential gain, (2^rel_i - 1), in place of rel_i, which weights the top grade more heavily (Wikipedia: Discounted cumulative gain). Both are valid. Pick one and keep it fixed, because the two give different numbers for the same ranking. The Stanford IR book chapter on ranked evaluation covers precision, recall and MAP in more depth, and our precision and recall glossary entry has the short version.
Report recall@k together with a ranking metric. A reranker can raise nDCG@10 while a first-stage change quietly drops relevant documents below the cutoff, and only recall shows the second effect.
Worked example: scoring one query by hand
This is a worked example with made-up data, computed by hand and checked with the code below. It is not a measurement of any system.
Query: "waterproof hiking boots". The judged documents are A (grade 3), B (grade 2) and C (grade 1). Everything else is grade 0. You switch embedding models and get two top-5 lists:
| Metric | Old | New |
| Precision@5 | 2/5 = 0.400 | 2/5 = 0.400 |
| Recall@5 | 2/3 = 0.667 | 2/3 = 0.667 |
| Reciprocal rank | 1/2 = 0.500 | 1/1 = 1.000 |
| nDCG@5 | 0.488 | 0.895 |
Precision and recall did not move: both systems found two of the three relevant documents. The new system found the better two and put them at the top, which only MRR and nDCG register. If you had tracked recall@5 alone you would have called this change neutral.
Python: recall@k, MRR and nDCG@k
This runs as-is on Python 3 with no dependencies and reproduces the worked example above. For production use, trec_eval and ir_measures implement the standard definitions.
import math, random
def recall_at_k(ranked, qrels, k):
relevant = {d for d, g in qrels.items() if g > 0}
return len(relevant & set(ranked[:k])) / len(relevant)
def precision_at_k(ranked, qrels, k):
return sum(1 for d in ranked[:k] if qrels.get(d, 0) > 0) / k
def reciprocal_rank(ranked, qrels):
for i, d in enumerate(ranked, start=1):
if qrels.get(d, 0) > 0:
return 1 / i
return 0.0
def ndcg_at_k(ranked, qrels, k):
dcg = sum(qrels.get(d, 0) / math.log2(i + 1)
for i, d in enumerate(ranked[:k], start=1))
ideal = sorted(qrels.values(), reverse=True)[:k]
idcg = sum(g / math.log2(i + 1) for i, g in enumerate(ideal, start=1))
return dcg / idcg if idcg else 0.0
def paired_randomization_test(a, b, n=10_000, seed=0):
"""Two-sided p-value: randomly swap old/new labels per query."""
rng = random.Random(seed)
deltas = [y - x for x, y in zip(a, b)]
observed = abs(sum(deltas))
hits = 0
for _ in range(n):
flipped = sum(d if rng.random() < 0.5 else -d for d in deltas)
if abs(flipped) >= observed:
hits += 1
return sum(deltas) / len(deltas), hits / n
# Worked example: one query, graded labels 0-3
qrels = {"A": 3, "B": 2, "C": 1}
old = ["X", "A", "Y", "C", "Z"]
new = ["A", "B", "X", "Y", "Z"]
for name, run in [("old", old), ("new", new)]:
print(name,
f"P@5={precision_at_k(run, qrels, 5):.3f}",
f"R@5={recall_at_k(run, qrels, 5):.3f}",
f"RR={reciprocal_rank(run, qrels):.3f}",
f"nDCG@5={ndcg_at_k(run, qrels, 5):.3f}")
# Made-up per-query nDCG@10 for 12 queries, old vs new
old_q = [0.61, 0.72, 0.40, 0.88, 0.55, 0.69, 0.47, 0.90, 0.33, 0.76, 0.58, 0.64]
new_q = [0.66, 0.70, 0.52, 0.91, 0.60, 0.69, 0.45, 0.93, 0.41, 0.80, 0.57, 0.71]
mean_delta, p = paired_randomization_test(old_q, new_q)
wins = sum(n > o for o, n in zip(old_q, new_q))
losses = sum(n < o for o, n in zip(old_q, new_q))
print(f"mean delta={mean_delta:.3f} p={p:.3f} wins={wins} losses={losses}")Output:
old P@5=0.400 R@5=0.667 RR=0.500 nDCG@5=0.488 new P@5=0.400 R@5=0.667 RR=1.000 nDCG@5=0.895 mean delta=0.035 p=0.017 wins=8 losses=3
What happens when the new system returns results nobody judged?
This is the most common way an offline evaluation lies about a retrieval change. Labels were collected for documents the old system returned. A new embedding model surfaces different documents, those documents have no label, and the metric counts every unlabeled document as irrelevant. The new system gets penalized for being different.
TREC handles this with pooling: the top results from every participating system are merged into a pool, and assessors judge the pool (TREC relevance judgments). The BEIR authors measured what happens when a system was not in the pool. On TREC-COVID, 31.8% of TAS-B's and 14.4% of ANCE's top-10 results had never been judged, against 6.4% for BM25. After the missing documents were judged, ANCE's nDCG@10 rose from 0.654 to 0.735 (BEIR, arXiv 2104.08663, Table 4).
What to do:
1. Pool both systems. Before comparing, take the top k from old and new, and judge every document that does not have a label yet. 2. Report the hole rate. The share of top-k results with no judgment, per system. If one system's hole rate is much higher, its score is a lower bound. 3. Re-pool after every change. A judged set built for last quarter's model goes stale as soon as you swap the model.
How do I compare the old and new system fairly?
Compare them per query, on the same queries. Query difficulty varies far more than the difference between two reasonable systems, so two averages hide most of the information. Compute delta = new score minus old score for each query and analyze those deltas.
Three things to look at:
Is the difference real? A simple significance check
Use a paired test on the per-query deltas. Smucker, Allan and Carterette (2007) compared five tests on TREC runs and found little practical difference between the paired t-test, the bootstrap and the randomization (permutation) test. They found that the Wilcoxon signed rank test and the sign test disagreed with the other three, recommended the randomization test, and advised against the Wilcoxon and sign tests for IR evaluation.
The randomization test is short enough to write yourself (it is in the snippet above). If the old and new system were interchangeable, swapping their labels on any query would not matter. So you flip the sign of each delta at random many times and count how often the shuffled total is at least as large as the real one. That share is the p-value.
On the made-up 12-query data in the snippet, the new system wins 8 queries, loses 3 and ties 1, with a mean nDCG@10 gain of 0.035. The randomization test gives p = 0.017. A sign test on the same data uses only win/loss counts and gives p = 0.23 (two-sided binomial, 8 of 11 non-tied queries), because it ignores how large each win or loss was. That gap is the reason the sign test is a weak choice here. Both numbers come from a made-up example and illustrate the method only.
How many queries do I need?
It depends on how noisy the per-query deltas are and how small a gain you care about. A common approximation for a paired test at 5% significance and 80% power is:
n ≈ ((1.96 + 0.84) × sd / delta)^2
where sd is the standard deviation of the per-query deltas and delta is the smallest gain you want to detect. Worked example, using the made-up data above: the deltas have sd ≈ 0.043. Detecting a 0.02 nDCG@10 gain needs about 36 queries. Detecting 0.01 needs about 145, four times as many, because n grows with the square of 1/delta.
Estimate sd from a small pilot run of a few dozen queries, then size the full set. Small, real improvements need hundreds of queries. A 30-query set is enough to catch a regression that breaks a whole class of queries and too small to settle a 1-point disagreement.
Can an LLM judge relevance instead of people?
Often yes, after you check it against people. Thomas et al., reporting on large-scale LLM relevance labelling at Bing, found LLM labels as accurate as human labellers and better than third-party workers when measured against first-party "gold" labels from real searchers (arXiv 2309.10621). The same paper found that systematic prompt changes and simple paraphrases of the prompt both changed accuracy.
The known failure modes:
A calibration step before you trust it:
1. Have two people label a random sample of query-document pairs using your written rubric. 2. Run the LLM judge with a fixed prompt on the same pairs. 3. Measure agreement between the two people, and between each person and the LLM (Cohen's kappa for binary labels, weighted kappa for graded ones). 4. Use the LLM only if it agrees with people about as well as people agree with each other. Freeze the prompt and model version, and re-run the calibration when either changes. 5. Check that the LLM-labeled set ranks your known-better and known-worse systems the same way the human-labeled sample does.
How do I confirm the change helped after I ship it?
Offline evaluation tells you the new ranking matches your labels better. Online evaluation tells you users behave differently. Run both.
A checklist for testing a retrieval change
1. Freeze the corpus snapshot, the query set and the metric definitions. 2. Run old and new, top k for each query. 3. Judge every unlabeled document in either top k. Report hole rates. 4. Score recall@k plus nDCG@k (or MRR if one good result is enough). 5. Compute per-query deltas, wins and losses, and a paired randomization or bootstrap test. 6. Read the worst 10 regressions by hand. 7. Ship behind an A/B test or interleaving experiment, and keep the query set for the next change.
If the change you are testing is a reranker, the same procedure applies; the best rerankers list covers the options. For multimodal corpora (video, images, audio), evaluating multimodal retrieval covers what changes in the labels.
How to run this in Mixpeek
Mixpeek is a multimodal data platform that indexes video, images, audio and documents from object storage and serves search through configurable retrievers. Its retriever evaluations implement the offline half of this guide.
dataset_name and a list of queries. Each query has a query_id, a query_input that uses the keys your retriever's input schema declares, and relevant_documents. Optional relevance_scores take a 0 to 5 grade per document, which nDCG uses for graded relevance; without them all metrics use binary relevance (Evaluations docs; Create dataset API).min_interactions (default 5), lookback_days (default 30) and which interaction types count as positive (default: click, purchase, positive_feedback, bookmark, share) (Generate from interactions API; Interactions docs). The click biases described above apply to these labels, so pool and spot-check them before you compare two retrievers.task_id while it proceeds (Run evaluation API).k_values (default 1, 5, 10 and 20). Results include overall_metrics and metrics_by_k, plus total_queries, evaluated_queries, skipped_queries and per-query query_results, so you can see which queries failed input validation (Evaluations docs; Get results API).metrics_by_k side by side. The paired per-query test from this guide can be run on query_results (Evaluations docs).