Retrieval

18 min read

Updated 2026-06-11

# Adaptive Indexing for Agentic Search: Query Logs, Payload Indexes, and Retrieval Routing

Learn how retrieval systems decide which indexes to build as agents search unstructured content. Covers query logs, slow-query diagnosis, payload indexes, hybrid routing, and Mixpeek MVS examples.

![Ethan Steininger](/people/ethan.webp)

By [Ethan Steininger](/blog/authors/ethan), Founder & CEO, Mixpeek

Agent Retrieval

Adaptive Indexing

Payload Indexes

MVS

Hybrid Search

Query Profiling

[Run this on your own data. Try it as you read.Search your own archive →](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)

[One-click templatesStart from a templateEach template deploys a working search app on your own files.Browse templates](/templates)

## Why Agent Search Needs Adaptive Indexing

  
  
Human search traffic is repetitive. Users type short queries, click results, reformulate, and leave behind stable query patterns.  
  
Agent search traffic is less predictable. An agent may start with a broad semantic query, add filters after seeing partial results, issue a lexical query for an exact phrase, inspect citations, expand a time window, and then search again with a narrower budget.  
  
That pattern matters for unstructured content:  
  
*   A video agent searches transcripts, captions, OCR, objects, faces, timestamps, and source metadata.
*   A document agent searches text, layout blocks, tables, page images, form fields, and access policies.
*   An audio agent searches transcript spans, speaker turns, acoustic events, languages, and timestamps.
*   A visual agent searches image embeddings, masks, detections, scene labels, and provenance.
  
  
One index rarely serves every query shape well. Dense vector search is useful for semantic recall, BM25 is useful for exact terms, payload indexes are useful for filters, and rerankers are useful for precision. Adaptive indexing is the process of watching real retrieval traffic, identifying the fields and query shapes that matter, and building the right indexes over time.  
  
The goal is not to index everything. The goal is to make the hot paths fast, citeable, and cheap without making the storage layer impossible to operate.  
  

## The Three Index Families

  
  
Most retrieval systems for agent memory use three index families.  
  

Index family

What it answers

Typical fields

Vector index

What is semantically similar?

embeddings for text, images, audio, video scenes

Lexical index

What contains this exact phrase or token pattern?

transcript text, OCR text, titles, captions

Payload index

Which records match structured constraints?

tenant\_id, object\_type, created\_at, speaker, camera\_id, policy\_label

  
  
Each family optimizes a different part of the retrieval problem.  
  
Vector indexes are good when the query is conceptual: "a customer sounds angry about a delayed shipment" or "a frame where someone opens a laptop."  
  
Lexical indexes are good when the query contains exact evidence: a part number, an error string, a SKU, a quoted sentence, or a person's name.  
  
Payload indexes are good when the agent must constrain search: only this customer, only last week, only videos, only English transcripts, only scenes with policy label "needs\_review."  
  
An agentic retrieval system needs all three because agents do not only ask semantic questions. They ask bounded, cited, tool-like questions.  
  

## Query Logs Are Training Data for the Storage Layer

  
  
Adaptive indexing begins with query logs. These logs should not just store the raw query string. They should describe the shape of the work the storage layer performed.  
  
A useful retrieval log includes:  
  

Field

Why it matters

namespace

Reveals tenant and workload skew

query\_type

Dense, sparse, hybrid, filter-only, rerank

filters

Shows which metadata fields are actually used

projected\_fields

Shows what payloads agents ask for

top\_k and candidate\_k

Shows recall and rerank pressure

latency breakdown

Separates parse, filter, search, rerank, materialization

bytes returned

Shows projection and payload pressure

result count

Reveals over-selective filters and empty searches

index\_hit

Shows whether a useful index was used

fallback\_path

Shows when the engine had to scan or degrade

  
  
This is not just observability. It is the feedback loop that tells the storage layer what to optimize.  
  
Example query-shape record:

{
  "namespace": "media-archive",
  "query\_type": "hybrid",
  "filters": {
    "object\_type": "video",
    "created\_at": {"$gte": "2026-06-01"},
    "policy\_label": "approved"
  },
  "projected\_fields": \["source\_uri", "start\_ms", "end\_ms", "caption"\],
  "candidate\_k": 200,
  "top\_k": 20,
  "latency\_ms": {
    "filter": 42,
    "vector": 81,
    "bm25": 27,
    "rerank": 133,
    "materialize": 9
  },
  "bytes\_returned": 18432,
  "index\_hit": \["object\_type", "created\_at"\],
  "fallback\_path": null
}

Once logs look like this, index decisions can be grounded in evidence instead of guesswork.  
  

## Slow Query Diagnosis

  
  
Before building an index, identify which part of the query is slow.  
  
The common latency components are:  
  

Component

Common cause

Parse and planning

Complex filters or many branches

Candidate generation

Large vector search, cold shard, high top\_k

Filter evaluation

Unindexed fields, low-selectivity predicates, nested payload scans

Lexical search

Large posting lists, phrase queries, fuzzy matching

Reranking

Too many query-candidate pairs

Materialization

Loading large payloads after ranking

Projection

Returning too many fields or large nested payloads

  
  
Do not build a payload index if the bottleneck is reranking. Do not tune the vector index if the bottleneck is materializing large transcript windows. The index has to match the actual bottleneck.  
  
For agents, many slow queries come from combinations:  
  
*   A semantic query plus a high-cardinality filter.
*   A broad date range plus an exact phrase.
*   A high top\_k because the agent wants fallback evidence.
*   A reranker applied to too many candidates.
*   A projection that returns full payloads when compact citations would work.
  
  
The fix may be a new index, but it may also be better routing, lower candidate\_k, a projection preset, or a two-step agent tool.  
  

## When to Build a Payload Index

  
  
A payload index is worth building when a field is both common in filters and selective enough to reduce work.  
  
Good payload index candidates:  
  
*   tenant\_id
*   object\_type
*   created\_at
*   source\_uri prefix or bucket
*   speaker
*   language
*   camera\_id
*   policy\_label
*   media\_type
*   extractor\_version
  
  
Weak candidates:  
  
*   One-off request IDs that are rarely filtered.
*   Free-form captions that should be lexical or vector indexed.
*   Large nested blobs that agents should not filter directly.
*   Fields with almost one value for every record unless exact lookup is the dominant path.
*   Fields that duplicate authorization state without a clear access-control model.
  
  
A practical rule:  
  
Build a payload index when a field appears in a meaningful share of slow queries and the filtered subset is much smaller than the namespace.  
  
Example:  
  

Field

Query frequency

Selectivity

Decision

object\_type

high

medium

Index

created\_at

high

high for recent windows

Index

speaker

medium

medium

Index if transcript search is hot

random\_trace\_id

low

high

Do not index unless exact lookup is common

caption

high

low as a filter

Use lexical and vector search instead

  
  
Payload indexes are not free. They add write cost, storage cost, rebuild complexity, and operational state. The best index is the one that removes repeated work from hot queries.  
  

## Filter-First vs. Vector-First Routing

  
  
Once indexes exist, the engine still has to choose how to use them.  
  
There are two common plans.  
  
**Filter-first:** apply payload filters first, then vector search inside the filtered subset.  
  
Use this when:  
  
*   The filter is highly selective.
*   The agent asks for a narrow tenant, source, date range, or media type.
*   Authorization filters must be enforced before candidate generation.
*   The vector space is large and the filter removes most records.
  
  
**Vector-first:** retrieve semantic candidates first, then apply filters.  
  
Use this when:  
  
*   The filter is weak or matches most records.
*   The vector index is very fast and the payload filter is cheap.
*   The query is broad and recall matters more than early pruning.
*   The filter field has no useful index yet.
  
  
Hybrid plans combine both:  
  
1\. Use payload indexes to find allowed or likely partitions. 2. Run dense vector search in those partitions. 3. Run BM25 over exact text fields. 4. Merge candidates with reciprocal rank fusion or weighted scoring. 5. Rerank a bounded candidate set. 6. Project compact evidence fields.  
  
The routing decision should be observable. When a query is slow, engineers need to see whether the planner chose filter-first, vector-first, lexical-first, or a fallback scan.  
  

## The Adaptive Loop

  
  
Adaptive indexing should be a controlled loop, not an automatic index explosion.  
  
The loop:  
  
1\. Observe query logs and slow traces. 2. Group slow queries by shape, not only by text. 3. Estimate index benefit using frequency, selectivity, and latency savings. 4. Propose an index with a specific field, type, and namespace. 5. Build the index in the background. 6. Warm or validate it with representative queries. 7. Route a small share of traffic through it. 8. Compare latency, recall, empty-result rate, and cost. 9. Promote, keep warming, or retire it.  
  
The key is step 8. An index is not successful because it built. It is successful because real queries became faster or more reliable without hurting recall or cost.  
  

## Agent Tool Design

  
  
Agents should not be asked to know index internals. They should express intent through tool parameters.  
  
Example tool shape:

{
  "name": "search\_media\_evidence",
  "description": "Search video, audio, image, and document evidence with filters and compact citations.",
  "input\_schema": {
    "type": "object",
    "properties": {
      "query": {"type": "string"},
      "content\_type": {"type": "string", "enum": \["video", "audio", "image", "document", "any"\]},
      "time\_window": {"type": "string"},
      "policy\_label": {"type": "string"},
      "top\_k": {"type": "integer", "minimum": 1, "maximum": 50},
      "projection": {"type": "string", "enum": \["answer", "visual", "compliance", "debug"\]},
      "latency\_budget\_ms": {"type": "integer"}
    },
    "required": \["query"\]
  }
}

The planner can translate this into indexes and routing:  
  
*   content\_type maps to a payload filter.
*   time\_window maps to a date index.
*   policy\_label maps to a governance filter.
*   projection maps to selected payload fields.
*   latency\_budget\_ms controls candidate\_k and rerank depth.
  
  
The agent sees a stable tool. The retrieval system evolves underneath it.  
  

## Failure Modes

  
  
**Indexing every field.** This creates write amplification and operational noise. Index hot fields, not all fields.  
  
**Ignoring low-result queries.** Empty results may mean the filter is too selective, the wrong modality was searched, or the planner applied filters too early.  
  
**Treating projection as indexing.** Projection reduces returned payload size. It does not make filtering faster unless the filter field is indexed.  
  
**Applying rerankers too broadly.** Cross-encoder reranking is powerful, but reranking 2,000 candidates often hides a bad first-stage plan.  
  
**Missing tenant skew.** A field may be hot for one tenant and irrelevant for another. Shared averages hide expensive outliers.  
  
**No retirement policy.** Old indexes consume space and slow writes. If query logs show no benefit, retire them.  
  
**No authorization boundary.** Payload indexes can accelerate filtering, but access control must still be enforced before results are returned.  
  

## Evaluation

  
  
Evaluate adaptive indexing at the query-shape level.  
  

Metric

What it tells you

p50 and p95 latency by query shape

Whether hot agent paths are improving

index hit rate

Whether queries use intended indexes

filter selectivity

Whether payload indexes reduce candidate work

candidate recall

Whether routing keeps enough relevant evidence

empty-result rate

Whether filters or routing are too strict

rerank candidate count

Whether first-stage search is bounded

bytes returned

Whether projection is controlling payload size

citation success

Whether agents can verify answers

build cost and storage overhead

Whether the index is worth keeping

  
  
The right evaluation question is not "is the index fast?" It is "does this index make the agent's task faster and more correct?"  
  

## Mixpeek MVS Example

  
  
Suppose a support video agent searches call recordings. Each transcript span is stored as a vector with structured payload fields.

from mixpeek import Mixpeek

NAMESPACE\_ID = "YOUR\_STANDALONE\_NAMESPACE\_ID"
mx = Mixpeek(api\_key="YOUR\_API\_KEY", namespace=NAMESPACE\_ID)

mx.namespaces.documents.upsert(
    namespace\_id=NAMESPACE\_ID,
    documents=\[
        {
            "document\_id": "call\_481:822180:826920:bge\_m3",
            "vectors": {"bge\_m3": span\_embedding},
            "payload": {
                "source\_uri": "s3://support-video/2026/06/09/call\_481.mp4",
                "object\_type": "video",
                "speaker": "customer",
                "language": "en-US",
                "policy\_label": "approved",
                "start\_ms": 822180,
                "end\_ms": 826920,
                "text": "I want a refund because the outage affected our launch",
            },
        }
    \],
)

After query logs show repeated filters on object\_type, policy\_label, and time ranges, build payload indexes for those fields.

import requests

API = "https://api.mixpeek.com/v1"
HEADERS = {"Authorization": "Bearer YOUR\_API\_KEY"}

# Standalone namespaces already build payload indexes from filter usage
# (auto\_create\_indexes is on by default). Declaring them makes the choice
# explicit and reviewable.
requests.patch(
    API + "/namespaces/" + NAMESPACE\_ID,
    headers=HEADERS,
    json={
        "payload\_indexes": \[
            {"field\_name": "object\_type", "type": "keyword"},
            {"field\_name": "policy\_label", "type": "keyword"},
            {"field\_name": "start\_ms", "type": "integer"},
        \]
    },
)

# ensure-indexes confirms each declared index exists in the vector store and
# backfills the documents already written, with no re-ingest.
requests.post(API + "/namespaces/" + NAMESPACE\_ID + "/ensure-indexes", headers=HEADERS)

Then search with filters and compact projection.

evidence = mx.retrievers.create(
    retriever\_name="support-video-evidence",
    input\_schema={"qv": {"type": "array", "required": True}},
    stages=\[
        {
            "stage\_name": "knn",
            "stage\_id": "feature\_search",
            "parameters": {
                "searches": \[
                    {
                        "feature\_uri": "bge\_m3",
                        "query": {"input\_mode": "vector", "value": "{{INPUT.qv}}"},
                        "top\_k": 20,
                    }
                \],
                "final\_top\_k": 20,
            },
        },
        {
            "stage\_name": "project",
            "stage\_id": "json\_transform",
            "parameters": {
                "template": (
                    '{"source\_uri": {{ DOC.source\_uri | tojson }}, "speaker": {{ DOC.speaker | tojson }}, '
                    '"text": {{ DOC.text | tojson }}, "start\_ms": {{ DOC.start\_ms }}, '
                    '"end\_ms": {{ DOC.end\_ms }}}'
                ),
            },
        },
    \],
)

results = mx.retrievers.execute(
    evidence\["retriever\_id"\],
    inputs={"qv": query\_embedding},
    filters={"AND": \[
        {"field": "object\_type", "operator": "eq", "value": "video"},
        {"field": "policy\_label", "operator": "eq", "value": "approved"},
        {"field": "start\_ms", "operator": "gte", "value": 600000},
    \]},
)

The agent receives compact cited evidence. The storage layer uses payload indexes to avoid scanning unrelated records. If logs later show that speaker is a frequent slow filter, add it. If it is rarely used, leave it unindexed.  
  

## Managed Mixpeek Example

  
  
Managed Mixpeek is the right path when the system should extract the features before indexing.  
  
For video, that means the pipeline can produce:  
  
*   Transcript spans from speech.
*   OCR spans from screen text.
*   Scene captions from visual models.
*   Object and face metadata.
*   Timestamps, source handles, and extractor versions.
  
  
Adaptive indexing still applies, but the fields come from the extraction pipeline instead of an upstream application.

from mixpeek import Mixpeek

mx = Mixpeek(api\_key="YOUR\_API\_KEY", namespace="training-video")

bucket = mx.buckets.create(
    bucket\_name="raw-training",
    bucket\_schema={"properties": {"video": {"type": "video"}}},
)

# One collection whose extractor writes a transcript, a scene description and
# on-screen text for every segment. Those are the fields later filters and
# payload indexes target.
collection = mx.collections.create(
    collection\_name="field-training",
    source={"type": "bucket", "bucket\_ids": \[bucket\["bucket\_id"\]\]},
    feature\_extractor={
        "feature\_extractor\_name": "multimodal\_extractor",
        "version": "v1",
        "parameters": {
            "run\_transcription": True,
            "run\_video\_description": True,
            "run\_ocr": True,
        },
    },
)

mx.buckets.upload(
    bucket\["bucket\_id"\],
    blobs=\[{"property": "video", "type": "video", "data": "s3://raw-training/forklift-safety.mp4"}\],
)
mx.collections.trigger(collection\["collection\_id"\])

Use MVS when you already have embeddings and metadata. Use Managed when you want the extraction, indexing, and retrieval system together.  
  

## Design Checklist

  
  
*   Log query shape, filters, projected fields, candidate counts, and latency breakdowns.
*   Separate vector, lexical, and payload index decisions.
*   Build payload indexes for frequent, selective filters.
*   Choose filter-first routing for selective or authorization-critical predicates.
*   Keep reranker candidate sets bounded.
*   Use projection presets so agents receive compact evidence.
*   Evaluate index benefit by query shape, not global averages.
*   Track tenant skew and namespace-specific hot paths.
*   Retire indexes that no longer improve latency, recall, or cost.
  
  

## Key Takeaways

  
  
1\. Adaptive indexing is a retrieval feedback loop, not a one-time schema decision.  
  
2\. Query logs should capture the work the storage layer performed, not just the text the user typed.  
  
3\. Vector, lexical, and payload indexes answer different questions. Agent search needs all three.  
  
4\. Payload indexes are most valuable when filters are frequent, selective, and repeated.  
  
5\. Routing matters as much as indexing. Filter-first and vector-first plans serve different query shapes.  
  
6\. The best outcome is not the most indexes. It is faster, cheaper, citeable evidence for the agent.  
  

## Further Reading

  
  
*   [Payload Projection for Agentic Vector Search](/guides/payload-projection-for-agentic-vector-search)
*   [Retrieval Control Planes for AI Agents](/guides/retrieval-control-planes-for-ai-agents)
*   [Multi-Stage Retrieval: How AI Agents Search Unstructured Data at Scale](/guides/multi-stage-retrieval-how-agents-search-unstructured-data)
*   [Evaluating Multimodal Retrieval Systems for AI Agents](/guides/evaluating-multimodal-retrieval)
*   [MVS: Agent-native vector store on object storage](/mvs)

## Run this on your own data

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this guide describes. Build starts at $25/mo for up to 1M vectors.

[Search your own archive](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)[Read Docs](/docs)

[Explore MVS](/mvs)[See pricing](/pricing)

## Related guides

[

Retrieval

### Payload Projection for Agentic Vector Search: Field Selection, Evidence Handles, and Context Budgets

Learn how field projection keeps agent retrieval fast, citeable, and context-efficient. Covers payload design, late materialization, evidence envelopes, select\_fields, and Mixpeek MVS examples for unstructured media search.

Read guide →](/guides/payload-projection-for-agentic-vector-search)[

Retrieval

### What Is Hybrid Search? BM25, Vector Retrieval, and How to Fuse Their Rankings

Why keyword and vector retrieval fail on opposite queries, what BM25 actually computes, and how Reciprocal Rank Fusion combines two rankings whose scores are not on the same scale. Covers the term-frequency saturation and length normalization inside BM25, why raw score addition breaks, RRF versus min-max and convex combination, how to pick the weighting, when hybrid is worse than either half, and what changes when one side of the index is video or images.

Read guide →](/guides/hybrid-search-bm25-vector-fusion)[

Retrieval

### Cyberdefense as a search problem: narrow billions of audit logs to the few an AI investigator should read

How to build a two-stage threat detection loop on Mixpeek: keep every audit event in object storage, count activity per identity, flag rate changes and first-time actions, then let a read-only model investigate the few dozen actors that changed. Working retriever configs for each step.

Read guide →](/guides/security-log-search-ai-threat-investigation)