The Short Answer
Capability-aware retrieval means the agent picks from a small set of named, described, costed retrieval contracts rather than assembling a query out of raw index primitives. The agent chooses *what it wants to find*. The layer decides which collection answers, how many candidates each stage requests, which filters apply, and what gets reranked.
The difference shows up on the second call. An agent holding raw index access makes five independent choices every time it searches, and it makes them differently each time, for reasons nothing in the trace explains.
What a capability is
A capability is a semantic contract with a description written for a model to read. Three properties make it usable by an agent.
A name makes the selection a discrete decision, so it shows up in a trace and can be compared across runs. The description says what the capability finds rather than how it works, which gives the model goals it can evaluate against the question in front of it. And a declared cost lets an agent working inside a budget commit with the number in hand.
What it deliberately hides: the collection, the embedding space, the candidate counts, the stage order, and every threshold. Those are the parts that change when the platform team improves something, and they are the parts an agent has no basis for choosing well.
What the declaration has to carry
Model-facing tool descriptions get written like documentation for humans and then get selected against by a model that only sees the string. The fields below are the ones that change selection behavior.
| Field | Why the agent needs it |
| What it searches over | "Video segments from marketing campaigns since 2024" tells the model when the capability is irrelevant, which is most of the time |
| Unit returned | An agent that expects whole documents and receives 4-second segments will summarize the wrong span |
| Required and optional inputs | An agent that has to guess an argument name burns a turn on a validation error |
| Typical latency and cost | Selection under a budget is impossible without it, and the model will otherwise assume everything is free |
| Freshness | An agent asked about something uploaded a minute ago needs to know whether this capability can see it yet, which depends on what the layer materializes |
| What it does NOT cover | The single highest-value line, because it is what stops the model reaching for the nearest-looking tool |
| Failure behavior | Whether an empty result means "nothing matched" or "a stage timed out and returned early" |
Why raw index access degrades
Give an agent a
vector_search(collection, query, k, filter) tool and it will use it. The failure is gradual.The agent picks
k based on nothing. It has no corpus statistics, so 10 and 500 are equally arbitrary, and it will choose differently on semantically identical queries. Filter syntax gets guessed, and a filter on a path that does not exist returns an empty set with a success status, which the agent reads as "no such content." Collection selection becomes a naming exercise, so an agent facing video_segments_v2 and video_segments_v3 picks by string similarity to the question.None of these produce errors. They produce plausible answers derived from the wrong candidate set, and the trace shows a successful tool call.
There is a second cost that shows up later. Access control has nothing to attach to. "This agent may search customer support transcripts but not HR recordings" is expressible as a capability boundary and inexpressible as a constraint on an index scan, so teams end up enforcing it in a wrapper that the next integration forgets to use.
The selection problem
Once capabilities exist, the failure mode moves. It stops being "the agent built a bad query" and becomes "the agent chose the wrong capability," which is a better problem because it is visible.
Overlap is the main one. Two capabilities that both plausibly answer a question will be selected roughly at random, and the one that wins is usually the one whose description shares more surface words with the prompt. The fix is in the descriptions rather than in the model: state the discriminator explicitly in both, including the negative half.
Cost blindness is the second. A model with a cheap keyword capability and an expensive multi-vector one will reach for whichever sounds more thorough. Putting the cost in the description helps. Putting a budget in the calling context, and letting the agent see what it has spent, helps more. Budget-aware multi-vector retrieval covers what the expensive path actually costs.
Too many capabilities is the third. Selection accuracy degrades as the roster grows, and it degrades fastest when the additions are near-duplicates. Ten well-separated capabilities beat forty finely-sliced ones. MCP tool design for multimodal search works through the granularity question.
Make the wrong choice observable
The reason to care about all of this is that a bad retrieval choice is invisible in an agent's output. The answer reads fine. It is based on a candidate set that was missing the relevant document, and nothing in the response says so.
Three things make it visible, and they are cheap to add at the layer rather than at every agent.
Log the capability name, the resolved plan and the candidate counts per stage, so a bad answer can be traced to the retrieval that produced it. Return provenance with results, so the agent can cite and a reviewer can check. Emit a distinguishable signal when a stage truncates or times out, so partial results never look like complete ones.
Retrieval control planes for AI agents covers the streaming and cancellation half of the same surface.
Antipatterns
One search tool with a
mode argument. The model now chooses a mode string, which is index selection wearing a different hat, and the modes are undocumented in the place the model is reading.Descriptions written for the docs site. "Powerful semantic search across your media library" gives a model nothing to discriminate on. Say what is in it, what unit comes back, and what it does not cover.
Exposing the candidate count. A
k parameter looks helpful and moves a decision the layer can make from measurement to a decision the model makes from nothing.Capabilities that mirror your collections. Collections are a storage decision. A capability that changes every time the platform team reorganizes storage is not a contract, and the agents bound to it inherit every migration.
The layer that holds these is retrievers, and the architecture they sit in is described in full in the semantic layer for unstructured data.
Frequently Asked Questions
What is capability-aware retrieval?
Capability-aware retrieval is a design where an AI agent selects from a small set of named retrieval contracts rather than composing queries from raw index operations. Each capability declares what it searches over, what unit it returns, what it costs, how fresh it is, and what it does not cover. The layer resolves the selected capability into a plan: collection, filters, embedding spaces, candidate budgets, reranking. The agent never chooses a candidate count or a filter path.
Why shouldn't an AI agent query a vector database directly?
Because every call requires choices the agent has no information to make. Candidate counts get picked arbitrarily and vary across semantically identical queries. Filter paths get guessed, and a filter on a nonexistent path returns an empty set with a success status that reads as "no such content." Collection selection degrades to string matching on names. None of these produce errors, so the agent returns a confident answer built on the wrong candidate set.
What should a retrieval tool description contain for an agent?
What the capability searches over, the unit it returns, its required and optional inputs, typical latency and cost, how quickly newly ingested content becomes visible, what it explicitly does not cover, and what an empty result means. The coverage boundary is the highest-value line, because it is what stops a model reaching for the nearest-looking tool. Descriptions written as marketing copy give a model nothing to discriminate on.
How many retrieval capabilities should an agent have?
Fewer and better separated beats many and finely sliced. Selection accuracy falls as the roster grows, and it falls fastest when new entries overlap existing ones, since a model presented with two plausible options tends to pick the one whose wording shares more surface words with the prompt. When two capabilities overlap, state the discriminator in both descriptions, including what each one does not cover.
How do you tell whether an agent chose the wrong retrieval capability?
Instrument the layer rather than the agent. Log the capability name, the resolved plan and the per-stage candidate counts for every call, return provenance alongside results, and emit a distinct signal when a stage truncates or times out. Without those, a bad retrieval choice produces a fluent answer built on a candidate set that never contained the relevant document, and nothing in the trace indicates it.