Retrieval

10 min read

Updated 2026-11-17

# Why Does My AI Agent's Search Ignore the Filter It Chose?

The filter usually never reaches the vector search: the tool schema has no filters argument, the adapter drops it, or the model names a field the index lacks, so the agent returns other brands' or customers' results with no error. A runnable script shows 8 of 10 results leaking, and the schema, database filter and merge that fix it.

![Ethan Steininger](/people/ethan.webp)

By [Ethan Steininger](/blog/authors/ethan), Founder & CEO, Mixpeek

AI Agents

Metadata Filtering

Tool Calling

Vector Search

RAG

[Agents that search with the filters they choose. Try it as you read.Agent search docs →](https://mixpeek.com/docs/retrieval/stages/agent-search)

[One-click templatesStart from a templateEach template deploys a working search app on your own files.Browse templates](/templates)

## Why does my AI agent's search ignore the filter it chose?

  
  
Usually the filter never reaches the vector search. The model writes the constraint into its tool call, for example brand equals acme, but the search tool's schema has no filters argument, or the code that turns the call into a database query drops it. The search then runs over every document and returns other brands' or other customers' results with no error. The second common cause is an agent that searches several times in one turn, where each result list replaces the one before. The fix: put filters in the tool's schema with the allowed fields listed, pass them to the database's own metadata filter, refuse a field the index does not have so the model can try again, enforce access filters in code the model cannot change, and merge results across searches.  
  

## Where does the agent's filter get lost?

  
  
There are four places between the model and the index, and none of them raises an error:  
  
1\. **The tool schema has no filters argument.** The model can only send what the schema describes. If the tool takes a query string alone, the model either puts "acme" into the query text, where the embedding treats it as one more word, or leaves it out. 2. **The adapter drops or renames it.** The model sends a filter and the glue code builds the search without it, or the model writes a field called client while the index stores brand. A filter on a field the index does not have matches nothing or is ignored, depending on the database. 3. **The database filters after ranking.** A post-filter takes the top k by similarity and then removes what fails the filter, so a selective filter leaves few or zero results. See [why vector search returns fewer results when you add a filter](/guides/why-vector-search-returns-fewer-results-with-filter). 4. **A later search replaces an earlier one.** An agent that runs one search per sub-question and keeps only the last list loses everything the earlier searches found, while its summary still says every sub-question was answered.  
  

## Reproduce it in plain Python

  
  
This script builds a small index of 400 ad descriptions from four brands, then runs the same tool call ("summer sale" with brand equals acme) through a tool that drops filters and one that applies them. It needs no packages and no model: the tool call is fixed, because the bug lives in the tool layer.

"""Where an agent's filter gets lost between the model and the vector search.

Stdlib only. The "model" is a fixed tool call, so the run is deterministic: the
point is what the tool layer does with the arguments the model sends.
"""
import json
import math
import random

random.seed(7)
BRANDS = \["acme", "globex", "initech", "umbrella"\]
TOPICS = \["summer sale", "holiday gift", "back to school", "new product launch"\]


def unit(v):
    n = math.sqrt(sum(x \* x for x in v))
    return \[x / n for x in v\]


TOPIC\_VECS = {t: unit(\[random.gauss(0, 1) for \_ in range(32)\]) for t in TOPICS}


def embed(topic):
    base = TOPIC\_VECS\[topic\]
    return unit(\[x + random.gauss(0, 0.08) for x in base\])


DOCS = \[
    {"id": f"{b}-{t.replace(' ', '-')}-{i}", "brand": b, "topic": t, "vec": embed(t)}
    for b in BRANDS for t in TOPICS for i in range(25)
\]


def cos(a, b):
    return sum(x \* y for x, y in zip(a, b))


def vector\_search(query\_vec, k=10, filters=None):
    """filters: list of {field, operator, value}; only 'eq' here."""
    pool = DOCS
    for f in filters or \[\]:
        if f\["field"\] not in ("brand", "topic") or f\["operator"\] != "eq":
            raise ValueError(f"unsupported filter {f}")
        pool = \[d for d in pool if d\[f\["field"\]\] == f\["value"\]\]
    return sorted(pool, key=lambda d: -cos(query\_vec, d\["vec"\]))\[:k\]


# The model read "acme's summer sale ads" and chose this tool call.
TOOL\_CALL = {"query": "summer sale", "filters": \[{"field": "brand", "operator": "eq", "value": "acme"}\]}


def tool\_without\_filters(args):
    # Schema had no filters argument: the adapter passes the query and drops the rest.
    return vector\_search(embed(args\["query"\]))


def tool\_with\_filters(args):
    return vector\_search(embed(args\["query"\]), filters=args.get("filters"))


def off\_brand(results, brand="acme"):
    return sum(d\["brand"\] != brand for d in results)


a = tool\_without\_filters(TOOL\_CALL)
b = tool\_with\_filters(TOOL\_CALL)
print(f"filters dropped:  {off\_brand(a)} of {len(a)} results are another brand's")
print(f"filters applied:  {off\_brand(b)} of {len(b)} results are another brand's")

# Second bug: two searches in one turn, the second overwrites the first.
calls = \[
    {"query": "summer sale", "filters": \[{"field": "brand", "operator": "eq", "value": "acme"}\]},
    {"query": "holiday gift", "filters": \[{"field": "brand", "operator": "eq", "value": "acme"}\]},
\]
kept = \[\]
for c in calls:
    kept = tool\_with\_filters(c)          # overwrite
overwrite\_topics = sorted({d\["topic"\] for d in kept})

runs = \[tool\_with\_filters(c) for c in calls\]
merged, seen = \[\], set()
for rank in range(max(len(r) for r in runs)):   # rank 1 of each run, then rank 2, ...
    for r in runs:
        if rank < len(r) and r\[rank\]\["id"\] not in seen:
            seen.add(r\[rank\]\["id"\])
            merged.append(r\[rank\])
print(f"overwrite:        topics kept {overwrite\_topics}")
print(f"merge by rank:    topics kept {sorted({d\['topic'\] for d in merged})}, {len(merged)} results")

# Third: a filter the model invents is refused, so the model can correct it.
try:
    tool\_with\_filters({"query": "summer sale", "filters": \[{"field": "client", "operator": "eq", "value": "acme"}\]})
except ValueError as e:
    print(f"bad field:        refused -> {json.dumps(str(e))}")

Output, from running it with Python 3:

filters dropped:  8 of 10 results are another brand's
filters applied:  0 of 10 results are another brand's
overwrite:        topics kept \['holiday gift'\]
merge by rank:    topics kept \['holiday gift', 'summer sale'\], 20 results
bad field:        refused -> "unsupported filter {'field': 'client', 'operator': 'eq', 'value': 'acme'}"

With the filter dropped, 8 of the 10 results belong to other brands, close to the 7.5 a random draw from four brands would give. With it applied, none do. Overwriting keeps only the last search's topic, and merging by rank keeps both.  
  

## How do I give the search tool a filters argument?

  
  
Describe the filters in the tool's JSON schema and list the fields the index can filter on as an enum, so the model cannot invent one. With OpenAI function calling, set strict to true: the model's arguments then follow the schema, which requires additionalProperties false on every object and every property listed as required (a null type makes one optional).

{
  "type": "function",
  "name": "search\_ads",
  "description": "Search ad descriptions. Use filters for exact constraints such as brand or year.",
  "strict": true,
  "parameters": {
    "type": "object",
    "properties": {
      "query": {"type": "string", "description": "What the ad is about, without the constraints"},
      "filters": {
        "type": \["array", "null"\],
        "items": {
          "type": "object",
          "properties": {
            "field": {"type": "string", "enum": \["brand", "topic", "year"\]},
            "operator": {"type": "string", "enum": \["eq", "gte", "lte"\]},
            "value": {"type": "string"}
          },
          "required": \["field", "operator", "value"\],
          "additionalProperties": false
        }
      }
    },
    "required": \["query", "filters"\],
    "additionalProperties": false
  }
}

The query description matters as much as the filter. Tell the model to keep constraints out of the query text, so "acme summer sale ads" becomes the query "summer sale" plus the filter brand equals acme. LlamaIndex's auto-retriever works the same way: it puts a schema of the metadata fields into the prompt, and the model returns a query string plus a list of filters, each with a key, an operator and a value.  
  

## How do I turn the tool call into a database filter?

  
  
Translate each item into the database's native filter and pass it to the search call, so the index applies it during the search. Two common translations of brand equals acme:

\# Pinecone: metadata filter on the query
index.query(vector=qvec, top\_k=10, filter={"brand": {"$eq": "acme"}})

# Qdrant: a must condition on the payload
from qdrant\_client import models
client.query\_points(
    collection\_name="ads",
    query=qvec,
    limit=10,
    query\_filter=models.Filter(must=\[
        models.FieldCondition(key="brand", match=models.MatchValue(value="acme")),
    \]),
)

Check each item before translating it. A field outside the enum, an operator the field does not support, or a value of the wrong type should fail the tool call with a message the model reads, such as "unknown field client, use one of brand, topic, year". Models usually correct a named error on the next call.  
  

## Should the model choose the tenant or permission filter?

  
  
No. Treat the model's filters as relevance hints and access rules as code. Read the user's tenant, team or permissions from the session, add that filter in the tool itself, and combine it with whatever the model sends using AND. The model never sees it and cannot remove or widen it, so a prompt that talks the model out of its filter cannot reach another tenant's data.  
  

## What should happen when the agent searches more than once?

  
  
Keep every search's results and merge them. Two simple merges:  
  
*   **By rank, round robin:** rank 1 of each search in call order, then rank 2, and so on, each
document once at its first position. This is what the script above does.*   **Reciprocal rank fusion:** score each document by the sum of 1 / (60 + rank) over the
searches that returned it, then sort. A document found by several searches rises.  
  
If a later step should narrow the set, make that an explicit filter step over the merged list, so results narrow only when the agent asks for it.  
  

## How does Mixpeek handle this?

  
  
Mixpeek is a multimodal data warehouse: it indexes video, images, audio and documents where they sit in object storage and searches them with retrievers built from stages. In a retriever's agent\_search stage, the model's feature\_search tool takes filters as a list of field, operator and value, and they are applied inside the vector search. A filter the index cannot apply fails that tool call, and the model sees the error. Filters set on the agent\_search stage itself apply to every search the agent runs, under the model's own filters. Results from every search are merged by rank with each document once, and an attribute\_filter step narrows the merged list when the agent needs to. This shipped on November 16, 2026; see the [changelog entry](/changelog#agent-search-applies-model-filters-and-keeps-every-search) and the [agent search stage docs](https://mixpeek.com/docs/retrieval/stages/agent-search).  
  

## Frequently Asked Questions

  
  

### Why does my RAG agent return documents from the wrong customer?

  
  
Because the customer constraint never reached the vector search. Either the search tool has no filters argument, the adapter drops it, or the model wrote a field name the index does not store. Access filters such as the customer or tenant should never depend on the model: add them in the tool code from the session. In the script on this page, dropping one brand filter returned 8 results of 10 from other brands.  
  

### What is a self-querying retriever?

  
  
A retriever that asks a language model to split a natural-language request into a search query and structured metadata filters, then runs the query with those filters. It needs a description of the metadata fields (name, type, meaning) in the prompt. LlamaIndex calls its version the auto-retriever; it returns a query string plus a list of filters, each a key, operator and value.  
  

### Can I put the filter in the query text instead?

  
  
It does not enforce anything. An embedding of "acme summer sale" leans toward documents that mention acme, and documents from other brands that are similar enough still rank above acme's. A hard constraint needs a metadata filter that the index applies.  
  

### How do I stop the model from inventing filter fields?

  
  
List the allowed fields as an enum in the tool's JSON schema, turn on strict schema adherence where your provider has it, and refuse any other field in the tool code with an error that names the allowed ones. The model reads the error and calls the tool again with a valid field.  
  

### Why does my agent get fewer results when it adds a filter?

  
  
The database is probably filtering after ranking: it takes the top k nearest vectors and then drops the ones that fail the filter. With a filter that matches 1% of documents, a top 10 can come back empty. Use the database's filtered search, or see [why vector search returns fewer results when you add a filter](/guides/why-vector-search-returns-fewer-results-with-filter).  
  

### Should each agent search replace the previous results?

  
  
No. Keep each search's results and merge them by rank or with reciprocal rank fusion. Replacing them keeps only the last search, and an agent that answered three sub-questions returns the evidence for one.  
  

## Key Takeaways

  
  
*   An agent's filter is usually lost between the model and the index: a tool schema without a
filters argument, an adapter that drops it, a renamed field, or a post-filter.*   In the script on this page, dropping one brand filter returned 8 of 10 results from other
brands, and applying it returned 0.*   Give the tool a filters argument with the allowed fields as an enum, and refuse unknown fields
with an error the model can read.*   Access filters such as the tenant come from the session and are added in code with AND, out of
the model's reach.*   When the agent searches more than once, merge the result lists; replacing them keeps only the
last search.  
  

## Sources

  
  
*   [OpenAI function calling, strict mode](https://developers.openai.com/api/docs/guides/function-calling)
*   [LlamaIndex auto-retriever with metadata filters](https://developers.llamaindex.ai/python/framework/integrations/vector_stores/pinecone_auto_retriever/)
*   [Pinecone: filter by metadata](https://docs.pinecone.io/guides/search/filter-by-metadata)
*   [Qdrant: filtering](https://qdrant.tech/documentation/search/filtering/)
*   [Mixpeek agent search stage](https://mixpeek.com/docs/retrieval/stages/agent-search)
  
  

## Where to go next

  
  
[Filtered vector search strategies](/guides/filtered-vector-search-pre-post-in-place) · [agentic retrieval](/guides/agentic-retrieval-how-agents-search-differently) · [multi-stage retrieval](/guides/multi-stage-retrieval-retrieve-filter-rerank) · [best vector databases](/curated-lists/best-vector-databases) · [MVS, vector search on object storage](/mvs) · [pricing](/pricing)

## Agents that search with the filters they choose

Mixpeek's agent\_search stage passes the model's filters into the vector search, applies the retriever's own filters to every search, and merges every search's results.

[Agent search docs](https://mixpeek.com/docs/retrieval/stages/agent-search)[See pricing](/pricing)

[Explore MVS](/mvs)[See pricing](/pricing)

## Related guides

[

Retrieval

### What Is Hybrid Search? BM25, Vector Retrieval, and How to Fuse Their Rankings

Why keyword and vector retrieval fail on opposite queries, what BM25 actually computes, and how Reciprocal Rank Fusion combines two rankings whose scores are not on the same scale. Covers the term-frequency saturation and length normalization inside BM25, why raw score addition breaks, RRF versus min-max and convex combination, how to pick the weighting, when hybrid is worse than either half, and what changes when one side of the index is video or images.

Read guide →](/guides/hybrid-search-bm25-vector-fusion)[

Retrieval

### Should a RAG Pipeline Use a Classification Model or an LLM to Filter and Rerank Results?

Use a classification model for any filter or rerank step whose answer is a choice from a list: keep or drop, a label, a relevance score. Keep a generative LLM for steps that write new text. This compares cross-encoder rerankers, decision models such as Jev and Liquid AI d1, and LLM judges on cost, speed and how far you can trust each score.

Read guide →](/guides/decision-models-in-retrieval-pipelines)[

Retrieval

### Why Does My Docs Chatbot Return Half a Code Example?

Your docs assistant answers with a code sample that stops mid-function, or cannot say which API it belongs to. The page was chunked by length and the cut landed inside the code block. We measured how often common splitters do this on 1,539 code blocks from Mixpeek's own documentation, and how to keep every block whole.

Read guide →](/guides/why-does-my-docs-chatbot-return-half-a-code-example)