Should a RAG Pipeline Use a Classification Model or an LLM to Filter and Rerank Results?

Should a RAG pipeline use a classification model or an LLM to filter and rerank results?

Use a classification model for filtering and reranking, and keep the LLM for the steps that write text. A filter answers keep or drop, and a reranker answers how relevant each document is: both are choices from a set you can list, which a classifier answers faster, cheaper and with a score you can threshold. In practice that means a cross-encoder reranker or a decision model for those steps, and a generative LLM for summaries, answers and new labels.

Use a decision model for any pipeline step whose output is a choice from a set you can list: keep or discard a document, pick a category, rate relevance on a scale, choose which branch of a hierarchy to search. Summaries, fresh cluster names and answers are new text, and they stay with a generative LLM.

A decision model such as TypeSafe's Jev takes the text and a typed question and returns a probability for each option. That makes it cheaper and more predictable for the decision steps, and the probability is useful on its own: it tells you which answers to trust and which to send for review.

Spotting a decision step

Look at the output schema of each step. If each field is a boolean, an enum, a number from 0 to 1, or an integer with a small range, the step is a decision.

Where a cross-encoder reranker fits

A cross-encoder reads the query and one candidate document together and returns a relevance score. It is the standard second stage of a RAG pipeline: retrieve a few dozen candidates by vector or keyword search, score each pair, and pass the top few to the LLM. Open-weight options such as BGE Reranker v2-m3 run on your own GPU or CPU, and managed ones such as Cohere Rerank are priced per search (roughly $0.001 to $0.0025, per our reranker comparison).

A cross-encoder answers one question: how relevant is this document to this query. A decision model answers whatever typed question you write, such as "does this document state the refund policy for damaged items". Use the cross-encoder when relevance is the whole test, and a decision model when the filter has a condition a relevance score cannot express. An LLM judge earns its cost only when the step needs reasoning across several documents at once.

Use the probability

A decision model answers keep with a number such as 0.98 or 0.51. That number lets you set a threshold per use case, send the uncertain middle to a person or to a larger model, and notice when a batch of documents is more ambiguous than usual. In a reranker the probability is the score itself.

In our own tests, a filter for "product complaints" kept a complaint at 0.98 and dropped a recipe at 0.01. In production, a reranker asked for "food or a food gift" put four food products at the top of a 30-document candidate list, each at 0.98, in 377 ms, and a classifier over eight product categories got six of six right.

Phrasing the question

Decision models read the question word for word. Write the exact condition:

"The document states the refund policy for damaged items" works. "Relevant to

refunds" produces vague scores.

Keep arithmetic, counting and date comparison in code. Ask the model for the parts

and compute the answer yourself.

Send only the text the question needs. Unrelated text lowers accuracy.

Cost

TypeSafe prices Jev at $0.042 per million input tokens, with output free. A filter decision over a short document is a few hundred tokens, so a million decisions costs a few dollars. Jev reads text only, so for video, images and audio it judges the transcript, captions and metadata.

Which decision models can I run myself?

Several open-weight decision models now exist, so a decision step can run on your own hardware, down to an edge device. All of them answer with a probability per option and generate no text.



On the public Decision Index 0.2.1, Liquid reports d1-3B at 48.57, ahead of every other model under 10B on the leaderboard and of a 36B model at 47.11. d1-omni-600M scores 15.95 there, so it suits narrow checks such as moderation (95.8 on Civil Comments in Liquid's run) and voice-command routing better than open-ended judgement.

Pick by where the decision runs and what it reads. A step that judges images on a GPU you already have fits d1-3B. A step on a phone, a camera or a voice device fits d1-omni-600M. A step that only reads text and has no hosting constraint is simplest as an API call. Check the license before shipping: the d1 models need a commercial agreement with Liquid once your company passes $10M in annual revenue.

A probability per decision is also a record. Ethan Steininger, Mixpeek's founder, counts "no trace of why a result came back" among the three failures that sink unstructured-data projects. A filter that logs keep at 0.98 or drop at 0.01 leaves that trace, one number per document.

In Mixpeek

Mixpeek offers Jev at five decision points: llm_provider: "typesafe" in the text extractor, model_name: "jev-latest" in the LLM Filter and LLM Enrich stages, inference_name: "typesafe__jev" in the Rerank stage, fixed-vocabulary cluster labeling through candidate_labels, and the cluster_navigation strategy for walking a cluster hierarchy.

Frequently Asked Questions

Should I use a cross-encoder or an LLM to rerank RAG results?

A cross-encoder, for most pipelines. It scores each query and document pair in one forward pass and returns the same score every time, which an LLM reranker does not guarantee. Reach for an LLM only when relevance depends on reasoning across documents, and measure whether the gain is worth the latency.

Is a classification model cheaper than an LLM for filtering?

Yes, by a wide margin, because it generates no text. TypeSafe prices Jev at $0.042 per million input tokens with output free, so a million filter decisions over short documents costs a few dollars. A generative model bills for every token it writes back.

When should a RAG pipeline still use an LLM to filter?

When the condition cannot be written as a fixed set of answers, or when judging one document needs the others beside it, for example "keep the most recent policy and drop the ones it replaces". Everything with a fixed answer set (keep or drop, a label, a 1 to 5 rating) fits a classifier.

How do I pick a threshold for a filter model?

Run the filter over 50 to 100 documents you have labelled yourself and look at where the probabilities of the true keeps and true drops overlap. Set the threshold above most of the drops, and send the uncertain middle to a person or to a larger model.

What is a decision model?

A model that reads text (and for some models images or speech) plus a typed question, and returns a probability for each allowed answer without generating text. Jev, Liquid AI d1-3B and autotrust JEV-27B-VL are examples.

Related

What are the best rerankers for RAG?

How do you run agentic search over a hierarchy of clusters?

Jev in Mixpeek