The Query Gap: Why Agent Searches Fail
Traditional information retrieval assumes a clean, human-typed query: "best restaurants in Brooklyn" or "force majeure clause in vendor contracts." The entire field — from BM25 to dense embeddings to learned sparse retrieval — optimizes for this assumption.
AI agents don't search like that.
When an agent reasons through a task, its "query" to a retrieval system is often a messy reasoning trace: partial conclusions, function call results, hypotheses being tested, and context accumulated across multiple steps. A typical agent query looks like:
The user asked about API rate limits. I checked the pricing page but it only
mentions request quotas per plan tier. The technical docs should have the
actual rate limiting implementation — I need the section about request
throttling and retry behavior for the enterprise tier.
Feed that to a standard embedding model, and the retrieved documents will be about "pricing pages" and "plan tiers" — not about rate limiting. The signal is buried in noise. The agent's intent (find rate limit documentation) is clear to a human reading the trace, but the embedding model encodes the entire trace as a single vector, diluting the search intent with irrelevant context.
This is the query gap: the mismatch between what agents produce as search queries and what retrieval systems expect as input.
Why Dense Embeddings Struggle with Agent Queries
Dense embedding models like BGE-M3, Qwen3-Embedding, or nomic-embed compress an entire text into a single fixed-dimensional vector. This works when the text is a focused query — the vector captures the dominant semantic theme.
But agent reasoning traces are multi-thematic by nature. A single trace might reference:
A single vector can't faithfully represent all four themes. The embedding becomes an average of everything, matching nothing well. This is analogous to the "polysemous query" problem in classical IR, but amplified — agent traces aren't just ambiguous, they're actively multi-intent.
Empirical Evidence
The AgentIR benchmark (2026) measures retrieval quality when queries come from agent reasoning traces instead of clean human queries. Results are striking:
| Model | Clean Query nDCG@10 | Agent Trace nDCG@10 | Drop |
| BM25 | 42.1 | 31.4 | -25% |
| Dense (BGE-M3) | 56.3 | 38.7 | -31% |
| Dense (Qwen3-Embed-8B) | 59.1 | 41.2 | -30% |
| Late Interaction (ColBERT) | 58.7 | 48.3 | -18% |
| Agent-Aware (Agent-ModernColBERT) | 55.2 | 52.8 | -4% |
Late Interaction: A Natural Fit for Noisy Queries
Late interaction models like ColBERT, ColPali, and GTE-ModernColBERT produce per-token embeddings instead of a single document vector. Scoring uses MaxSim — for each query token, find the maximum cosine similarity to any document token, then sum across all query tokens.
This architecture is inherently more robust to noisy queries because:
1. Selective attention: If 40% of the query tokens are noise (prior context, tool outputs), they'll have low MaxSim scores against relevant documents. The signal tokens — the ones that match — still contribute their high scores.
2. Partial matching: An agent trace that mentions "rate limiting" in passing will still match a document about rate limiting, even if the trace also discusses pricing, API keys, and error handling.
3. No information loss: Unlike dense embeddings where multi-theme content gets averaged into mush, late interaction preserves each token's individual representation.
The trade-off is storage and compute: a 512-token query produces 512 × 128-dim vectors instead of a single 768-dim vector. But for agent workloads where retrieval quality directly determines task success, this trade-off is almost always worth it.
Agent-Aware Retrieval Models
The newest development is training retrieval models specifically on agent reasoning traces. Agent-ModernColBERT from LightOn is the first production-ready model in this category.
How It Works
Agent-ModernColBERT starts from a ModernBERT backbone and is fine-tuned on the AgentIR dataset — a collection of (reasoning_trace, relevant_document) pairs generated by diverse AI agents solving real tasks. The training teaches the model to:
1. Identify search intent within noisy reasoning traces 2. Downweight irrelevant context (prior tool outputs, status messages) 3. Recognize agent-specific query patterns (hypothesis testing, iterative refinement, negation of prior results)
At 150M parameters, it's remarkably small — yet it achieves 72.53% accuracy on BrowseComp-Plus, outperforming setups that use GPT-5 for query reformulation combined with Qwen3-8B for retrieval. The model itself learns to extract the right query, eliminating the need for a separate LLM reformulation step.
When to Use Agent-Aware vs. General Models
| Scenario | Recommended Model | Why |
| Human-typed queries | BGE-M3 or Qwen3-Embedding | Dense embeddings are fast and effective for focused queries |
| Agent tool-use (MCP) | Agent-ModernColBERT | Trained specifically on reasoning traces |
| Mixed (human + agent) | pplx-embed-v1-late | Late interaction handles both gracefully |
| Multi-modal agent queries | BidirLM-Omni-2.5B | Shared text/image/audio space for cross-modal agent search |
Multi-Hop Retrieval Patterns
Real agent tasks rarely resolve in a single retrieval step. An agent researching a topic might need to:
1. Find an overview document (broad retrieval) 2. Extract specific entities from that document 3. Search for detailed information about those entities (narrow retrieval) 4. Synthesize findings and identify gaps 5. Search for gap-filling information (targeted retrieval)
Each step produces a different kind of query, and the context from prior steps accumulates in the reasoning trace.
Pattern 1: Iterative Refinement
The agent starts with a broad query, examines results, then narrows:
Step 1: "machine learning model deployment"
→ Retrieves overview of deployment strategies
Step 2: "containerized ML inference with GPU scheduling,
not the serverless approach mentioned in the
deployment overview"
→ Narrower, references prior results, uses negation
Step 3: "Kubernetes GPU operator configuration for
multi-model serving with dynamic batching,
specifically the Triton Inference Server setup
from the containerization guide"
→ Highly specific, chains context from steps 1-2
By step 3, the query contains intent from all three steps. A dense embedding of this full trace will be dominated by the accumulated context, not the current search need. Late interaction models handle this better because the most recent, specific tokens ("Triton Inference Server setup", "dynamic batching") will have high MaxSim scores against relevant documents.
Pattern 2: Query Decomposition
For complex questions, an agent decomposes the query into sub-queries:
Original: "Compare the cost and latency of running CLIP vs DINOv2
for visual search at 10M images"
Sub-query 1: "CLIP inference cost per image GPU pricing"
Sub-query 2: "DINOv2 inference latency benchmark batch processing"
Sub-query 3: "vector index scaling 10 million images memory requirements"
Each sub-query is cleaner than the original, so standard dense retrieval works well. The challenge is in the decomposition — deciding how to split and when to merge results.
Pattern 3: Verification Search
After generating an answer, agents search for contradicting evidence:
"I concluded that FAISS IVF-PQ is the best index for 10M vectors,
but I should verify — are there benchmarks showing ScaNN or
HNSW outperforming IVF-PQ at this scale?"
This is adversarial self-retrieval: the agent searches for evidence against its own conclusion. The query contains the conclusion (which should NOT match) and the search intent (which should). Dense embeddings encode both, returning documents that confirm the conclusion rather than challenge it. Agent-aware models learn to focus on the verification intent.
Query Expansion for Agent Contexts
When the query is a reasoning trace, expanding it naively (adding synonyms, related terms) makes the noise problem worse. Agent-specific query expansion works differently:
Intent Extraction
Instead of expanding the full trace, extract the search intent first:
# Bad: expand the entire reasoning trace
expanded = expand(reasoning_trace) # adds more noise
# Good: extract intent, then expand just the intent
intent = extract_intent(reasoning_trace)
# "Kubernetes GPU operator configuration for Triton Inference Server"
expanded = expand(intent)
# "Kubernetes GPU operator Triton Inference Server NVIDIA device plugin
# multi-model serving configuration yaml"
The intent extraction step can use a lightweight LLM (even a 1B model) or the agent's own structured output. Many agent frameworks now expose a "current search intent" field alongside the full reasoning trace.
Hypothetical Document Embedding (HyDE) for Agents
HyDE generates a hypothetical answer document and embeds that instead of the query. For agent contexts, this is particularly effective because the agent often knows approximately what the answer should look like:
# Agent reasoning trace
trace = """I need the Kubernetes manifest for deploying Triton
with GPU scheduling. It should have resource limits for
nvidia.com/gpu and a readiness probe on the health endpoint."""
# Generate hypothetical document
hypothetical = llm.generate(
f"Write a short documentation snippet that would answer: {trace}"
)
# "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: triton-server
# ...\n resources:\n limits:\n nvidia.com/gpu: 1"
# Embed the hypothetical document, not the trace
embedding = embed(hypothetical)
results = search(embedding, top_k=10)
The hypothetical document is closer in embedding space to the real document than the reasoning trace would be.
Architecture: Building an Agent-Ready Retrieval Pipeline
A production agentic retrieval pipeline has three layers:
Layer 1: Intent Router
Classify incoming queries as human-typed or agent-generated. This determines which retrieval path to use:
Query → Intent Classifier → Human path (dense) OR Agent path (late interaction)
The classifier can be as simple as checking for agent metadata (MCP tool calls include a "source" field) or as sophisticated as a small model trained to distinguish reasoning traces from clean queries.
Layer 2: Multi-Strategy Retrieval
For agent queries, run multiple retrieval strategies in parallel:
1. Full-trace late interaction — encode the entire reasoning trace with Agent-ModernColBERT and search 2. Extracted-intent dense search — extract the core search intent, embed with BGE-M3, and search 3. Keyword extraction — pull out specific entities, code identifiers, or error messages and run exact-match filters
Combine results using Reciprocal Rank Fusion (RRF):
def rrf_combine(result_lists, k=60):
scores = {}
for result_list in result_lists:
for rank, doc_id in enumerate(result_list):
scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
return sorted(scores, key=scores.get, reverse=True)
Layer 3: Agent-Aware Reranking
After retrieval, rerank with a model that understands agent context. A cross-encoder reranker like Ettin-1B scores each (trace, document) pair. Cross-encoders see both texts jointly, so they can learn to focus on the relevant parts of the trace when scoring.
Mixpeek Implementation
Mixpeek's retriever pipeline supports agentic retrieval natively through multi-stage retrieval with mixed feature types:
from mixpeek import Mixpeek
mx = Mixpeek(api_key="YOUR_KEY")
# Ingest with both dense and late-interaction embeddings
mx.ingest.documents(
source="s3://knowledge-base/",
collection="agent_kb",
feature_extractors=[
{
"name": "text_embeddings",
"model": "BAAI/bge-m3",
"params": {"dim": 1024}
},
{
"name": "text_embeddings_late",
"model": "lightonai/Agent-ModernColBERT",
"params": {"interaction": "late", "dim": 128}
}
]
)
# Agent retrieval with reasoning trace
results = mx.retrievers.search(
collection="agent_kb",
query=agent_reasoning_trace,
stages=[
# Stage 1: Late interaction over full trace (broad recall)
{
"type": "feature_search",
"feature": "text_embeddings_late",
"top_k": 100
},
# Stage 2: Dense rerank with extracted intent
{
"type": "rerank",
"model": "cross-encoder/ettin-reranker-1b-v1",
"top_k": 10
}
]
)
For MCP-based agent integrations, Mixpeek exposes retrieval as a tool that agents call directly:
{
"name": "search_knowledge_base",
"description": "Search the knowledge base using your current reasoning context",
"input_schema": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Your search query or current reasoning trace"
},
"intent": {
"type": "string",
"description": "Optional: extracted search intent for better precision"
}
}
}
}
When both \`query\` and \`intent\` are provided, Mixpeek automatically runs the multi-strategy pipeline: late interaction on the full query, dense search on the intent, and RRF fusion of both result sets.
Key Takeaways
1. Agent queries ≠ human queries. Reasoning traces are multi-thematic, noisy, and accumulate context across steps. Traditional retrieval models lose 25-31% of their quality on agent traces.
2. Late interaction is the minimum viable architecture for agentic retrieval. Per-token matching (MaxSim) naturally filters noise by letting signal tokens dominate the score.
3. Purpose-built models exist now. Agent-ModernColBERT at 150M parameters outperforms systems using GPT-5 for query reformulation. The model itself learns to extract intent from reasoning traces.
4. Multi-strategy fusion beats any single approach. Combine late interaction (broad recall from traces), dense search (precision from extracted intent), and keyword filters (exact entity matching).
5. The query gap will widen. As agents become more capable, their reasoning traces become longer and more complex. Retrieval systems that assume clean queries will fall further behind. Building agent-aware retrieval now is an investment in the agent-native future.