The Short Answer
A RAG pipeline has two stages that fail for different reasons, so you evaluate them separately with two pairs of metrics. Retrieval quality: context precision (are the retrieved chunks actually relevant, and are the relevant ones ranked first?) and context recall (did retrieval fetch everything needed to answer?). Generation quality: faithfulness / groundedness (is every claim in the answer supported by the retrieved context, i.e. no hallucination?) and answer relevance (does the answer actually address the question asked?). If you only measure the final answer, you can't tell whether a bad answer came from bad retrieval or bad generation — and those need opposite fixes. The widely used open framework for these is RAGAS; the definitions below are framework-agnostic.
The Two Failure Modes (and why one end-to-end score hides them)
Grade only the final answer and every failure looks the same. Split it and the fix becomes obvious:
| Symptom | Retrieval metric | Generation metric | Likely fix |
| Answer is wrong, context didn't contain the fact | low context recall | — | fix ingest/chunking/embeddings; widen retrieval |
| Right facts retrieved, but buried below junk | low context precision | — | rerank; tighten filters; better query |
| Context had the answer, model ignored or contradicted it | high context recall | low faithfulness | prompt grounding; smaller context; better model |
| Answer is grounded but off-topic | — | low answer relevance | fix the query/prompt; the model answered the wrong question |
The Four Core Metrics
Context Precision — of the chunks retrieved, what fraction are actually relevant, weighted by rank (relevant chunks near the top score higher)? Low precision means retrieval is noisy; the generator has to find the signal in the junk.
Context Recall — of the facts needed to answer, what fraction appear in the retrieved context? This is the ceiling on your whole pipeline: if the answer isn't in the context, no amount of prompting recovers it. Measuring recall requires a ground-truth answer or the set of "must-retrieve" facts per question.
Faithfulness / Groundedness — of the claims the answer makes, what fraction are directly supported by the retrieved context? This is your hallucination metric: an unfaithful answer asserts things the context never said. Scored by decomposing the answer into atomic claims and checking each against the context.
Answer Relevance — does the answer address the actual question, without padding or drift? A response can be perfectly faithful to the context and still not answer what was asked. Often scored by asking a model to generate questions the answer *would* be a good response to, then measuring their similarity to the real question.
How LLM-as-Judge Actually Scores These (and where it breaks)
Faithfulness and answer relevance don't have a string-match ground truth, so they're scored by an LLM judge: the answer is broken into atomic claims, and the judge marks each as supported / unsupported by the context (faithfulness), or the judge rates how well the answer targets the question (relevance). It works, but it has failure modes you must control for:
Treat the judge as an instrument that itself needs validating — not an oracle.
You Can't Evaluate What You Can't Compare To: the Golden Set
Every metric above needs a fixed evaluation set: representative questions, and for recall/correctness a ground-truth answer or the must-retrieve facts per question. Build it from real user queries (not invented ones), cover the hard cases (multi-hop, ambiguous, out-of-scope questions the system *should* refuse), and freeze it so scores are comparable across runs. This is the same discipline as building ground truth for retrieval evaluation — reuse that set and layer the generation metrics on top. Regenerate it on a cadence as your corpus and traffic shift.
The Multimodal Wrinkle
Over video, image, audio, and documents, both halves get harder. Context recall/precision must judge relevance of a *retrieved frame, clip, or region*, not just a text passage — "relevant" means the right moment or the right object, so your ground truth needs timestamps/regions, not just document IDs. Faithfulness must check the answer against what the media actually contains: an answer that describes an object not in the retrieved frame is a multimodal hallucination, and a text-only judge can't catch it — you need the judge to see the same modality the claim is about. And the retrieval plan itself is multimodal (transcript vs. on-screen text vs. visual similarity), so per-stage metrics should attribute which modality's retrieval helped or hurt.
How to Run It on Mixpeek
The per-stage split the metrics require is exactly what a multi-stage retriever already exposes:
The takeaway: instrument retrieval and generation separately, judge faithfulness against the real retrieved context, validate your judge before you trust its numbers, and hold the eval set fixed so a score change means a system change.
FAQ
What metrics should I use to evaluate RAG? Four, in two pairs. Retrieval: context precision (are retrieved chunks relevant and well-ranked) and context recall (did you retrieve everything needed). Generation: faithfulness/groundedness (are the answer's claims supported by the context — your hallucination metric) and answer relevance (does the answer address the question). Compute all four on the same run so you can tell whether a bad answer is a retrieval problem or a generation problem — they need opposite fixes.
How do I measure hallucination in RAG? Measure faithfulness: decompose the generated answer into atomic claims and check what fraction are directly supported by the retrieved context. An unfaithful claim — one the context never states — is a hallucination. It's typically scored with an LLM judge, so validate the judge against a sample of human labels before trusting the number, and note that high faithfulness with low answer relevance means the model is grounded but answering the wrong question.
Is RAGAS the standard for RAG evaluation? RAGAS is the most widely used open-source framework for these metrics and its definitions (context precision/recall, faithfulness, answer relevancy) have become a de-facto vocabulary, which is why they're used here. It isn't the only option — the metrics themselves are framework-agnostic, and you can compute them with any LLM-judge harness. What matters more than the tool is the discipline: a fixed golden set, per-stage metrics, and a calibrated judge.
Why do I need a golden/eval set — can't I just eyeball answers? Because eyeballing isn't comparable across runs, so you can't tell whether a change helped. A frozen set of representative questions (with ground-truth answers or must-retrieve facts for recall) turns evaluation into a repeatable score: change the system, re-run, see the delta. Build it from real queries including the hard and out-of-scope cases, and regenerate it on a cadence as your corpus drifts.