> ## Documentation Index > Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Evaluations > Measure and compare retriever quality with ground truth datasets and standard IR metrics Evaluations run your retriever against a curated set of queries with known-relevant documents, then compute standard information retrieval metrics at multiple cutoff points. Use them to quantify retriever quality, compare configurations, and catch regressions before they reach production. ## Quickstart Each query pairs an input with the document IDs that should be returned. ```bash cURL theme={null} curl -X POST "$MP_API_URL/v1/retrievers/evaluations/datasets" \ -H "Authorization: Bearer $MP_API_KEY" \ -H "X-Namespace: $MP_NAMESPACE" \ -H "Content-Type: application/json" \ -d '{ "dataset_name": "product-search-golden", "queries": [ { "query_id": "q1", "query_input": {"query": "wireless earbuds"}, "relevant_documents": ["doc_a1", "doc_a2", "doc_a3"], "relevance_scores": {"doc_a1": 5, "doc_a2": 3, "doc_a3": 2} }, { "query_id": "q2", "query_input": {"query": "noise canceling headphones"}, "relevant_documents": ["doc_b1", "doc_b4", "doc_b7"] } ] }' ``` ```python Python theme={null} dataset = client.retrievers.evaluations.create_dataset( dataset_name="product-search-golden", queries=[ { "query_id": "q1", "query_input": {"query": "wireless earbuds"}, "relevant_documents": ["doc_a1", "doc_a2", "doc_a3"], "relevance_scores": {"doc_a1": 5, "doc_a2": 3, "doc_a3": 2}, }, { "query_id": "q2", "query_input": {"query": "noise canceling headphones"}, "relevant_documents": ["doc_b1", "doc_b4", "doc_b7"], }, ], ) ``` ```javascript JavaScript theme={null} const dataset = await client.retrievers.evaluations.createDataset({ datasetName: "product-search-golden", queries: [ { queryId: "q1", queryInput: { query: "wireless earbuds" }, relevantDocuments: ["doc_a1", "doc_a2", "doc_a3"], relevanceScores: { doc_a1: 5, doc_a2: 3, doc_a3: 2 }, }, { queryId: "q2", queryInput: { query: "noise canceling headphones" }, relevantDocuments: ["doc_b1", "doc_b4", "doc_b7"], }, ], }); ``` Execute your retriever against every query in the dataset. The evaluation runs asynchronously and returns a `task_id` for progress tracking. ```bash cURL theme={null} curl -X POST "$MP_API_URL/v1/retrievers/{retriever_id}/evaluations" \ -H "Authorization: Bearer $MP_API_KEY" \ -H "X-Namespace: $MP_NAMESPACE" \ -H "Content-Type: application/json" \ -d '{ "dataset_name": "product-search-golden", "evaluation_config": { "k_values": [1, 5, 10, 20], "metrics": ["precision", "recall", "f1", "f2", "ndcg", "map", "mrr"] } }' ``` ```python Python theme={null} evaluation = client.retrievers.evaluations.run( retriever_id="ret_abc123", dataset_name="product-search-golden", evaluation_config={ "k_values": [1, 5, 10, 20], "metrics": ["precision", "recall", "f1", "f2", "ndcg", "map", "mrr"], }, ) # evaluation.task_id, evaluation.evaluation_id ``` ```javascript JavaScript theme={null} const evaluation = await client.retrievers.evaluations.run({ retrieverId: "ret_abc123", datasetName: "product-search-golden", evaluationConfig: { kValues: [1, 5, 10, 20], metrics: ["precision", "recall", "f1", "f2", "ndcg", "map", "mrr"], }, }); // evaluation.taskId, evaluation.evaluationId ``` Poll the evaluation until `status` is `completed`. ```bash theme={null} curl "$MP_API_URL/v1/retrievers/{retriever_id}/evaluations/{evaluation_id}" \ -H "Authorization: Bearer $MP_API_KEY" \ -H "X-Namespace: $MP_NAMESPACE" ``` ```json Example response theme={null} { "evaluation_id": "eval_abc123", "status": "completed", "query_count": 50, "overall_metrics": { "precision_at_5": 0.85, "recall_at_5": 0.72, "f1_at_5": 0.78, "f2_at_5": 0.74, "ndcg_at_5": 0.81, "map": 0.71, "mrr": 0.93 }, "metrics_by_k": { "1": {"precision": 0.90, "recall": 0.18, "f1": 0.30, "f2": 0.21, "ndcg": 0.90}, "5": {"precision": 0.85, "recall": 0.72, "f1": 0.78, "f2": 0.74, "ndcg": 0.81}, "10": {"precision": 0.75, "recall": 0.88, "f1": 0.81, "f2": 0.85, "ndcg": 0.89}, "20": {"precision": 0.62, "recall": 0.95, "f1": 0.75, "f2": 0.86, "ndcg": 0.91} } } ``` ## Metrics Every metric is computed at each K value you specify. The defaults cover most use cases: | Metric | What It Measures | Formula | | ---------------- | ---------------------------------------- | -------------------------------------------------------------------------------- | | **Precision\@K** | Accuracy of top results | relevant in top K ÷ K | | **Recall\@K** | Coverage of relevant documents | relevant in top K ÷ total relevant | | **F1\@K** | Balanced precision/recall | harmonic mean of P and R | | **F2\@K** | Recall-weighted balance | weighted harmonic mean (β=2), penalizes missed docs 4× more than false positives | | **MAP** | Ranking quality across all queries | average of precision at each relevant doc's position | | **MRR** | How quickly users find a relevant result | 1 ÷ rank of first relevant document | | **NDCG\@K** | Ranking quality with graded relevance | normalized discounted cumulative gain | **F2 vs F1:** Use F2 when missing a relevant document is worse than showing an irrelevant one — the common case in search, recommendations, and discovery. F1 treats both errors equally. ### Reading Your Scores | Score | What It Tells You | | ----------------------- | ------------------------------------------------------------------------------------------ | | **NDCG\@10 = 0.89** | Your top-10 ranking captures 89% of the ideal ordering. Relevant docs appear near the top. | | **Precision\@5 = 0.85** | 4–5 of every 5 results are relevant. Users see high-quality results. | | **Recall\@20 = 0.95** | You surface 95% of all relevant documents within the top 20. Strong coverage. | | **F2\@10 = 0.85** | Recall-weighted balance is strong — few relevant documents are being missed. | | **MRR = 0.93** | The first relevant result typically appears at position 1 or 2. | | **MAP = 0.71** | Overall ranking quality is solid but there's room to improve ordering. | ### Graded Relevance When you provide `relevance_scores` in your dataset, NDCG uses graded relevance instead of binary. This distinguishes "exactly right" from "somewhat relevant": ```json theme={null} { "query_id": "q1", "query_input": {"query": "wireless earbuds"}, "relevant_documents": ["doc_a1", "doc_a2", "doc_a3"], "relevance_scores": { "doc_a1": 5, "doc_a2": 3, "doc_a3": 1 } } ``` | Score | Meaning | | ----- | ------------------- | | 5 | Perfect match | | 3–4 | Highly relevant | | 1–2 | Marginally relevant | | 0 | Not relevant | Without `relevance_scores`, all metrics use binary relevance (relevant or not). ## Comparing Retrievers Run the same dataset against different retriever configurations to find the best pipeline: ```bash theme={null} # Evaluate baseline (vector search only) curl -X POST "$MP_API_URL/v1/retrievers/ret_baseline/evaluations" \ -H "Authorization: Bearer $MP_API_KEY" \ -H "X-Namespace: $MP_NAMESPACE" \ -d '{"dataset_name": "product-search-golden"}' # Evaluate candidate (vector search + reranker) curl -X POST "$MP_API_URL/v1/retrievers/ret_reranked/evaluations" \ -H "Authorization: Bearer $MP_API_KEY" \ -H "X-Namespace: $MP_NAMESPACE" \ -d '{"dataset_name": "product-search-golden"}' ``` Compare the `metrics_by_k` side by side: | Metric | Baseline | + Reranker | Delta | | ------------ | -------- | ---------- | ----- | | NDCG\@10 | 0.78 | 0.89 | +14% | | Precision\@5 | 0.72 | 0.85 | +18% | | F2\@10 | 0.76 | 0.85 | +12% | | MRR | 0.81 | 0.93 | +15% | Run the same dataset after every pipeline change — adding stages, swapping models, adjusting fusion weights — to quantify the impact before deploying. ## Ground Truth Datasets ### Dataset Requirements * At least 1 query (aim for 50+ for statistically meaningful results) * Each query must have at least 1 relevant document * `query_input` must match your retriever's input schema * `relevance_scores`, if provided, must cover all `relevant_documents` ### Managing Datasets ```bash theme={null} # List all datasets curl "$MP_API_URL/v1/retrievers/evaluations/datasets" \ -H "Authorization: Bearer $MP_API_KEY" \ -H "X-Namespace: $MP_NAMESPACE" # Get a specific dataset curl "$MP_API_URL/v1/retrievers/evaluations/datasets/product-search-golden" \ -H "Authorization: Bearer $MP_API_KEY" \ -H "X-Namespace: $MP_NAMESPACE" ``` ### Building Good Datasets Cover head queries (popular), torso (moderate), and tail (rare/specific). Don't just test easy cases. Binary relevant/not-relevant misses nuance. Score documents 0–5 so NDCG can distinguish good rankings from great ones. Sample queries from production logs. Synthetic queries test what you think users ask, not what they actually ask. Keep datasets stable across evaluations so you can track metric trends over time. Create new versions for schema changes. ## Configuration Reference Name of the ground truth dataset to evaluate against. Cutoff positions for @K metrics. Include the K values that match your UI — if you show 10 results per page, include `10`. Metrics to compute. Available: `precision`, `recall`, `f1`, `f2`, `map`, `ndcg`, `mrr`. ## Related * [Retriever Benchmarks](/api-reference/retriever-benchmarks/create-benchmark) — replay live sessions against candidate retrievers * [Improve Relevance](/platform/improve-relevance) — interaction signals, fusion strategies, and the feedback loop * [API Reference: Run Evaluation](/api-reference/retriever-evaluations/run-evaluation) — full endpoint specification * [API Reference: Create Dataset](/api-reference/retriever-evaluations/create-evaluation-dataset) — dataset creation endpoint