NEWVectors or files. Pick a path.Start →
    Retrieval
    10 min read
    Updated 2026-11-17

    Why Does My AI Agent's Search Ignore the Filter It Chose?

    The filter usually never reaches the vector search: the tool schema has no filters argument, the adapter drops it, or the model names a field the index lacks, so the agent returns other brands' or customers' results with no error. A runnable script shows 8 of 10 results leaking, and the schema, database filter and merge that fix it.

    Ethan Steininger
    By , Founder & CEO, Mixpeek
    AI Agents
    Metadata Filtering
    Tool Calling
    Vector Search
    RAG

    Why does my AI agent's search ignore the filter it chose?



    Usually the filter never reaches the vector search. The model writes the constraint into its tool call, for example brand equals acme, but the search tool's schema has no filters argument, or the code that turns the call into a database query drops it. The search then runs over every document and returns other brands' or other customers' results with no error. The second common cause is an agent that searches several times in one turn, where each result list replaces the one before. The fix: put filters in the tool's schema with the allowed fields listed, pass them to the database's own metadata filter, refuse a field the index does not have so the model can try again, enforce access filters in code the model cannot change, and merge results across searches.

    Where does the agent's filter get lost?



    There are four places between the model and the index, and none of them raises an error:

    1. The tool schema has no filters argument. The model can only send what the schema describes. If the tool takes a query string alone, the model either puts "acme" into the query text, where the embedding treats it as one more word, or leaves it out. 2. The adapter drops or renames it. The model sends a filter and the glue code builds the search without it, or the model writes a field called client while the index stores brand. A filter on a field the index does not have matches nothing or is ignored, depending on the database. 3. The database filters after ranking. A post-filter takes the top k by similarity and then removes what fails the filter, so a selective filter leaves few or zero results. See why vector search returns fewer results when you add a filter. 4. A later search replaces an earlier one. An agent that runs one search per sub-question and keeps only the last list loses everything the earlier searches found, while its summary still says every sub-question was answered.

    Reproduce it in plain Python



    This script builds a small index of 400 ad descriptions from four brands, then runs the same tool call ("summer sale" with brand equals acme) through a tool that drops filters and one that applies them. It needs no packages and no model: the tool call is fixed, because the bug lives in the tool layer.
    """Where an agent's filter gets lost between the model and the vector search.
    
    Stdlib only. The "model" is a fixed tool call, so the run is deterministic: the
    point is what the tool layer does with the arguments the model sends.
    """
    import json
    import math
    import random
    
    random.seed(7)
    BRANDS = ["acme", "globex", "initech", "umbrella"]
    TOPICS = ["summer sale", "holiday gift", "back to school", "new product launch"]
    
    
    def unit(v):
        n = math.sqrt(sum(x * x for x in v))
        return [x / n for x in v]
    
    
    TOPIC_VECS = {t: unit([random.gauss(0, 1) for _ in range(32)]) for t in TOPICS}
    
    
    def embed(topic):
        base = TOPIC_VECS[topic]
        return unit([x + random.gauss(0, 0.08) for x in base])
    
    
    DOCS = [
        {"id": f"{b}-{t.replace(' ', '-')}-{i}", "brand": b, "topic": t, "vec": embed(t)}
        for b in BRANDS for t in TOPICS for i in range(25)
    ]
    
    
    def cos(a, b):
        return sum(x * y for x, y in zip(a, b))
    
    
    def vector_search(query_vec, k=10, filters=None):
        """filters: list of {field, operator, value}; only 'eq' here."""
        pool = DOCS
        for f in filters or []:
            if f["field"] not in ("brand", "topic") or f["operator"] != "eq":
                raise ValueError(f"unsupported filter {f}")
            pool = [d for d in pool if d[f["field"]] == f["value"]]
        return sorted(pool, key=lambda d: -cos(query_vec, d["vec"]))[:k]
    
    
    # The model read "acme's summer sale ads" and chose this tool call.
    TOOL_CALL = {"query": "summer sale", "filters": [{"field": "brand", "operator": "eq", "value": "acme"}]}
    
    
    def tool_without_filters(args):
        # Schema had no filters argument: the adapter passes the query and drops the rest.
        return vector_search(embed(args["query"]))
    
    
    def tool_with_filters(args):
        return vector_search(embed(args["query"]), filters=args.get("filters"))
    
    
    def off_brand(results, brand="acme"):
        return sum(d["brand"] != brand for d in results)
    
    
    a = tool_without_filters(TOOL_CALL)
    b = tool_with_filters(TOOL_CALL)
    print(f"filters dropped:  {off_brand(a)} of {len(a)} results are another brand's")
    print(f"filters applied:  {off_brand(b)} of {len(b)} results are another brand's")
    
    # Second bug: two searches in one turn, the second overwrites the first.
    calls = [
        {"query": "summer sale", "filters": [{"field": "brand", "operator": "eq", "value": "acme"}]},
        {"query": "holiday gift", "filters": [{"field": "brand", "operator": "eq", "value": "acme"}]},
    ]
    kept = []
    for c in calls:
        kept = tool_with_filters(c)          # overwrite
    overwrite_topics = sorted({d["topic"] for d in kept})
    
    runs = [tool_with_filters(c) for c in calls]
    merged, seen = [], set()
    for rank in range(max(len(r) for r in runs)):   # rank 1 of each run, then rank 2, ...
        for r in runs:
            if rank < len(r) and r[rank]["id"] not in seen:
                seen.add(r[rank]["id"])
                merged.append(r[rank])
    print(f"overwrite:        topics kept {overwrite_topics}")
    print(f"merge by rank:    topics kept {sorted({d['topic'] for d in merged})}, {len(merged)} results")
    
    # Third: a filter the model invents is refused, so the model can correct it.
    try:
        tool_with_filters({"query": "summer sale", "filters": [{"field": "client", "operator": "eq", "value": "acme"}]})
    except ValueError as e:
        print(f"bad field:        refused -> {json.dumps(str(e))}")
    Output, from running it with Python 3:
    filters dropped:  8 of 10 results are another brand's
    filters applied:  0 of 10 results are another brand's
    overwrite:        topics kept ['holiday gift']
    merge by rank:    topics kept ['holiday gift', 'summer sale'], 20 results
    bad field:        refused -> "unsupported filter {'field': 'client', 'operator': 'eq', 'value': 'acme'}"
    With the filter dropped, 8 of the 10 results belong to other brands, close to the 7.5 a random draw from four brands would give. With it applied, none do. Overwriting keeps only the last search's topic, and merging by rank keeps both.

    How do I give the search tool a filters argument?



    Describe the filters in the tool's JSON schema and list the fields the index can filter on as an enum, so the model cannot invent one. With OpenAI function calling, set strict to true: the model's arguments then follow the schema, which requires additionalProperties false on every object and every property listed as required (a null type makes one optional).
    {
      "type": "function",
      "name": "search_ads",
      "description": "Search ad descriptions. Use filters for exact constraints such as brand or year.",
      "strict": true,
      "parameters": {
        "type": "object",
        "properties": {
          "query": {"type": "string", "description": "What the ad is about, without the constraints"},
          "filters": {
            "type": ["array", "null"],
            "items": {
              "type": "object",
              "properties": {
                "field": {"type": "string", "enum": ["brand", "topic", "year"]},
                "operator": {"type": "string", "enum": ["eq", "gte", "lte"]},
                "value": {"type": "string"}
              },
              "required": ["field", "operator", "value"],
              "additionalProperties": false
            }
          }
        },
        "required": ["query", "filters"],
        "additionalProperties": false
      }
    }
    The query description matters as much as the filter. Tell the model to keep constraints out of the query text, so "acme summer sale ads" becomes the query "summer sale" plus the filter brand equals acme. LlamaIndex's auto-retriever works the same way: it puts a schema of the metadata fields into the prompt, and the model returns a query string plus a list of filters, each with a key, an operator and a value.

    How do I turn the tool call into a database filter?



    Translate each item into the database's native filter and pass it to the search call, so the index applies it during the search. Two common translations of brand equals acme:
    # Pinecone: metadata filter on the query
    index.query(vector=qvec, top_k=10, filter={"brand": {"$eq": "acme"}})
    
    # Qdrant: a must condition on the payload
    from qdrant_client import models
    client.query_points(
        collection_name="ads",
        query=qvec,
        limit=10,
        query_filter=models.Filter(must=[
            models.FieldCondition(key="brand", match=models.MatchValue(value="acme")),
        ]),
    )
    Check each item before translating it. A field outside the enum, an operator the field does not support, or a value of the wrong type should fail the tool call with a message the model reads, such as "unknown field client, use one of brand, topic, year". Models usually correct a named error on the next call.

    Should the model choose the tenant or permission filter?



    No. Treat the model's filters as relevance hints and access rules as code. Read the user's tenant, team or permissions from the session, add that filter in the tool itself, and combine it with whatever the model sends using AND. The model never sees it and cannot remove or widen it, so a prompt that talks the model out of its filter cannot reach another tenant's data.

    What should happen when the agent searches more than once?



    Keep every search's results and merge them. Two simple merges:

  1. By rank, round robin: rank 1 of each search in call order, then rank 2, and so on, each
  2. document once at its first position. This is what the script above does.
  3. Reciprocal rank fusion: score each document by the sum of 1 / (60 + rank) over the
  4. searches that returned it, then sort. A document found by several searches rises.

    If a later step should narrow the set, make that an explicit filter step over the merged list, so results narrow only when the agent asks for it.

    How does Mixpeek handle this?



    Mixpeek is a multimodal data warehouse: it indexes video, images, audio and documents where they sit in object storage and searches them with retrievers built from stages. In a retriever's agent_search stage, the model's feature_search tool takes filters as a list of field, operator and value, and they are applied inside the vector search. A filter the index cannot apply fails that tool call, and the model sees the error. Filters set on the agent_search stage itself apply to every search the agent runs, under the model's own filters. Results from every search are merged by rank with each document once, and an attribute_filter step narrows the merged list when the agent needs to. This shipped on November 16, 2026; see the changelog entry and the agent search stage docs.

    Frequently Asked Questions



    Why does my RAG agent return documents from the wrong customer?



    Because the customer constraint never reached the vector search. Either the search tool has no filters argument, the adapter drops it, or the model wrote a field name the index does not store. Access filters such as the customer or tenant should never depend on the model: add them in the tool code from the session. In the script on this page, dropping one brand filter returned 8 results of 10 from other brands.

    What is a self-querying retriever?



    A retriever that asks a language model to split a natural-language request into a search query and structured metadata filters, then runs the query with those filters. It needs a description of the metadata fields (name, type, meaning) in the prompt. LlamaIndex calls its version the auto-retriever; it returns a query string plus a list of filters, each a key, operator and value.

    Can I put the filter in the query text instead?



    It does not enforce anything. An embedding of "acme summer sale" leans toward documents that mention acme, and documents from other brands that are similar enough still rank above acme's. A hard constraint needs a metadata filter that the index applies.

    How do I stop the model from inventing filter fields?



    List the allowed fields as an enum in the tool's JSON schema, turn on strict schema adherence where your provider has it, and refuse any other field in the tool code with an error that names the allowed ones. The model reads the error and calls the tool again with a valid field.

    Why does my agent get fewer results when it adds a filter?



    The database is probably filtering after ranking: it takes the top k nearest vectors and then drops the ones that fail the filter. With a filter that matches 1% of documents, a top 10 can come back empty. Use the database's filtered search, or see why vector search returns fewer results when you add a filter.

    Should each agent search replace the previous results?



    No. Keep each search's results and merge them by rank or with reciprocal rank fusion. Replacing them keeps only the last search, and an agent that answered three sub-questions returns the evidence for one.

    Key Takeaways



  5. An agent's filter is usually lost between the model and the index: a tool schema without a
  6. filters argument, an adapter that drops it, a renamed field, or a post-filter.
  7. In the script on this page, dropping one brand filter returned 8 of 10 results from other
  8. brands, and applying it returned 0.
  9. Give the tool a filters argument with the allowed fields as an enum, and refuse unknown fields
  10. with an error the model can read.
  11. Access filters such as the tenant come from the session and are added in code with AND, out of
  12. the model's reach.
  13. When the agent searches more than once, merge the result lists; replacing them keeps only the
  14. last search.

    Sources



  15. OpenAI function calling, strict mode
  16. LlamaIndex auto-retriever with metadata filters
  17. Pinecone: filter by metadata
  18. Qdrant: filtering
  19. Mixpeek agent search stage


  20. Where to go next



    Filtered vector search strategies · agentic retrieval · multi-stage retrieval · best vector databases · MVS, vector search on object storage · pricing
    Managed Mixpeek

    Put multimodal search to work

    Connect a bucket and Mixpeek runs the whole multimodal search pipeline for you: extraction, indexing, and search over your own objects. No models to wire up, nothing to host.

    Start with Managed
    MVS · bring your own

    Already have vectors?

    Keep your embeddings on your own cloud and run dense, sparse, and BM25 search directly on object storage. From $25/mo.

    Start with MVS

    Agents that search with the filters they choose

    Mixpeek's agent_search stage passes the model's filters into the vector search, applies the retriever's own filters to every search, and merges every search's results.

    Agent search docsSee pricing