What is NDCG (Normalized Discounted Cumulative Gain)

How It Works

Give each result a relevance grade, for example 3 for an exact match, 2 for a close one, 1 for related and 0 for irrelevant. DCG@K adds each grade divided by log2(position + 1), so rank 1 counts in full, rank 2 counts 0.63 and rank 4 counts 0.43. IDCG@K is the same sum for the best possible order of those results, and NDCG@K = DCG@K / IDCG@K.

Worked Example

• A query returns four results with grades 3, 2, 0, 1
• DCG@4 = 3/1 + 2/1.585 + 0/2 + 1/2.322 = 4.693
• The ideal order is 3, 2, 1, 0, so IDCG@4 = 3 + 1.262 + 0.5 + 0 = 4.762
• NDCG@4 = 4.693 / 4.762 = 0.985
• Putting the best result third instead (grades 0, 2, 3, 1) gives DCG@4 = 3.193 and NDCG@4 = 0.670

Technical Details

• Formula used here: DCG@K = Σ rel_i / log2(i + 1) for i = 1 to K. Some libraries use the exponential gain (2^rel_i - 1) instead, which rewards top grades more, so check which one a score was computed with before comparing
• Range 0 to 1, averaged over all queries in the evaluation set
• With binary relevance (every grade 0 or 1), NDCG still rewards putting relevant results higher
• A query with no relevant results has IDCG = 0 and is usually scored 0 or left out

Measuring NDCG in Mixpeek

Mixpeek retriever evaluations compute NDCG at each cutoff you set in `k_values` (for example 1, 5, 10 and 20). Add `relevance_scores` to a ground truth query, such as `{"doc_a1": 5, "doc_a2": 3}`, and NDCG uses those grades; without them it uses binary relevance. The example evaluation in the docs reports NDCG@5 of 0.81. See retriever evaluations.

Common Pitfalls

• Comparing NDCG across different K values or different gain formulas
• Grades that annotators apply inconsistently, which add noise larger than the change being measured
• Judging only the results the current system returned, so a new system's unjudged results count as irrelevant
• Reporting a single average that hides queries where the ranking got worse