1.  [](/)
2.  [Glossary](/glossary)
3.  RAG Evaluation (RAGAS)

[Back to Glossary](/glossary)

# What is RAG Evaluation (RAGAS)

## RAG Evaluation (RAGAS) - Measuring whether a RAG system retrieves the right context and answers from it

RAG evaluation measures the two halves of a retrieval augmented generation system separately: whether retrieval found the right context, and whether the generated answer is faithful to that context and answers the question. RAGAS is an open source framework for it, introduced in a 2023 paper, whose core metrics are faithfulness, response relevancy, context precision and context recall. Most of these metrics use an LLM as the judge, so they can run without a hand-written reference answer for every question.

### The Core RAGAS Metrics

*   Faithfulness: the share of claims in the answer that the retrieved context supports. Low faithfulness means the model is adding facts the context does not contain
*   Response relevancy: how directly the answer addresses the question asked
*   Context precision: whether the relevant chunks are ranked near the top of the retrieved context
*   Context recall: whether the retrieved context contains everything needed for the reference answer
*   Definitions and the full list, including noise sensitivity, are in the [RAGAS metrics documentation](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/)

### How to Run an Evaluation

*   Collect 50 to 200 real questions, ideally from logs, with a reference answer or the documents that should be retrieved
*   Run each question through the system and save the question, the retrieved context and the answer
*   Score retrieval with information retrieval metrics against labeled documents, and generation with faithfulness and relevancy
*   Change one thing at a time (chunking, embedding model, reranker, prompt) and rerun on the same set
*   Spot-check the judge: read 20 scored answers by hand to confirm the LLM judge agrees with you

### Evaluating Multimodal RAG

When the context is video segments, images or scanned pages, check retrieval on the media itself: did the system return the right clip or page, at the right timestamp. Text-only judges see transcripts or captions, so a wrong frame with a matching transcript can still score as correct. Label ground truth by document or segment ID and score it with retrieval metrics before judging the generated answer.

### Measuring the Retrieval Half in Mixpeek

Mixpeek retriever evaluations score retrieval against a ground truth dataset of queries and relevant document IDs, with precision, recall, F1, [MAP](/glossary/mean-average-precision), [NDCG](/glossary/ndcg) and [MRR](/glossary/mean-reciprocal-rank) at each cutoff. Generation metrics such as faithfulness need a separate judge such as RAGAS run on your answers. See [retriever evaluations](https://mixpeek.com/docs/retrieval/evaluations) and [multimodal RAG](/glossary/multimodal-rag).

### Common Pitfalls

*   Scoring only the final answer, which hides whether a failure came from retrieval or from generation
*   Using the same LLM as generator and judge without checking its scores by hand
*   Evaluation questions written by the team instead of taken from real users
*   Changing several settings at once, so a score change has no single cause

[One-click templatesStart from a templateEach template deploys a working search app on your own files.Browse templates](/templates)

[Put it to work: search your own files](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=glossary_onramp&utm_campaign=mvs_onramp)

### Related Terms

[ACID](/glossary/acid)[API](/glossary/api)[Blob Storage](/glossary/blob-storage)[CLIP](/glossary/clip)[Embedding](/glossary/embedding)