What is RAG Evaluation (RAGAS)
RAG Evaluation (RAGAS) - Measuring whether a RAG system retrieves the right context and answers from it
RAG evaluation measures the two halves of a retrieval augmented generation system separately: whether retrieval found the right context, and whether the generated answer is faithful to that context and answers the question. RAGAS is an open source framework for it, introduced in a 2023 paper, whose core metrics are faithfulness, response relevancy, context precision and context recall. Most of these metrics use an LLM as the judge, so they can run without a hand-written reference answer for every question.
The Core RAGAS Metrics
- Faithfulness: the share of claims in the answer that the retrieved context supports. Low faithfulness means the model is adding facts the context does not contain
- Response relevancy: how directly the answer addresses the question asked
- Context precision: whether the relevant chunks are ranked near the top of the retrieved context
- Context recall: whether the retrieved context contains everything needed for the reference answer
- Definitions and the full list, including noise sensitivity, are in the RAGAS metrics documentation
How to Run an Evaluation
- Collect 50 to 200 real questions, ideally from logs, with a reference answer or the documents that should be retrieved
- Run each question through the system and save the question, the retrieved context and the answer
- Score retrieval with information retrieval metrics against labeled documents, and generation with faithfulness and relevancy
- Change one thing at a time (chunking, embedding model, reranker, prompt) and rerun on the same set
- Spot-check the judge: read 20 scored answers by hand to confirm the LLM judge agrees with you
Evaluating Multimodal RAG
When the context is video segments, images or scanned pages, check retrieval on the media itself: did the system return the right clip or page, at the right timestamp. Text-only judges see transcripts or captions, so a wrong frame with a matching transcript can still score as correct. Label ground truth by document or segment ID and score it with retrieval metrics before judging the generated answer.
Measuring the Retrieval Half in Mixpeek
Mixpeek retriever evaluations score retrieval against a ground truth dataset of queries and relevant document IDs, with precision, recall, F1, MAP, NDCG and MRR at each cutoff. Generation metrics such as faithfulness need a separate judge such as RAGAS run on your answers. See retriever evaluations and multimodal RAG.
Common Pitfalls
- Scoring only the final answer, which hides whether a failure came from retrieval or from generation
- Using the same LLM as generator and judge without checking its scores by hand
- Evaluation questions written by the team instead of taken from real users
- Changing several settings at once, so a score change has no single cause
Put multimodal search to work
Connect a bucket and Mixpeek runs the whole multimodal search pipeline for you: extraction, indexing, and search over your own objects. No models to wire up, nothing to host.
Start with ManagedAlready have vectors?
Keep your embeddings on your own cloud and run dense, sparse, and BM25 search directly on object storage. From $25/mo.
Start with MVS