What is Mean Reciprocal Rank (MRR)

How It Works

For each query, find the position of the first relevant result and take its reciprocal. Average those values across all queries. Three queries whose first relevant results sit at ranks 1, 3 and 2 give (1 + 1/3 + 1/2) / 3 = 0.611. With a cutoff K, a first relevant result below rank K counts as 0, which is written MRR@K.

Technical Details

• Formula: MRR = (1 / |Q|) × Σ 1 / rank_q, where rank_q is the position of the first relevant result for query q
• Range 0 to 1; higher is better
• Binary relevance: a result is relevant or it is not, and graded scores are ignored
• Only the first relevant result counts, so two systems that both put a correct answer at rank 1 score the same even if one of them returns nothing useful after it

When to Use MRR

• Known-item search: the user wants one particular video moment, image or file
• Question answering, where the first good passage is what gets read
• Reverse image search and duplicate lookup, where the best match should come first
• Use NDCG or mean average precision instead when the user reads many results or when relevance comes in grades

Measuring MRR in Mixpeek

Mixpeek retriever evaluations report MRR next to precision, recall, F1, MAP and NDCG. You upload a ground truth dataset of queries and their relevant document IDs, then run `POST /v1/retrievers/{retriever_id}/evaluations` with `"metrics": ["mrr", "ndcg"]`. Running the same dataset against two retrievers shows whether a change such as adding a reranker moved the first correct result up. See retriever evaluations.

Common Pitfalls

• Reading MRR as overall ranking quality when it only describes the first relevant result
• Comparing MRR values computed with different cutoffs
• Too few queries: with 20 queries, one query moving from rank 1 to rank 2 shifts MRR by 0.025
• Ground truth that marks only one of several correct results as relevant, which understates the score