Agent Perception

17 min read

Updated 2026-06-04

# Audio-Visual Retrieval for AI Agents: How to Search What Happened, Not Just What Was Said

A technical guide to building retrieval over video with sound. Learn chunking, audio-video embeddings, transcript and object channels, fusion, reranking, evaluation, and agent tool design.

![Ethan Steininger](/people/ethan.webp)

By [Ethan Steininger](/blog/authors/ethan), Founder & CEO, Mixpeek

AI Agents

Audio

Video

Multimodal Retrieval

Agent Perception

[Run this on your own data. Try it as you read.Search your own archive →](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)

[One-click templatesStart from a templateEach template deploys a working search app on your own files.Browse templates](/templates)

## Why Audio-Visual Retrieval Is Different

  
  
Most video search systems start with transcripts. That works when the important evidence is spoken. It fails when the event is visible, audible, or only meaningful when sound and motion are interpreted together.  
  
Examples:  
  
*   A support agent needs the clip where a device clicks twice, flashes red, and then shuts down.
*   A safety agent needs near misses where a forklift horn sounds before a worker steps back.
*   A media agent needs the crowd reaction after a shot, not just the commentator saying it was a shot.
*   A robotics agent needs the moment a motor begins squealing before the arm stalls.
  
  
In all four cases, the evidence is not a document. It is an observation with time, pixels, sound, speech, objects, and source lineage. The agent does not need a vague summary. It needs a bounded retrieval tool that returns exact clips and explains which signals matched.  
  
That is the hard gate for this topic: audio-visual retrieval helps an AI agent see, hear, and search unstructured content.  
  

## The Core Mental Model

  
  
Think of video with audio as a stream of observations. Retrieval turns that stream into searchable evidence packets.

raw media
  -> temporal chunks
  -> feature channels
  -> per-channel indexes
  -> fusion and reranking
  -> evidence packet for the agent

The evidence packet is the unit the agent consumes. It should contain:  
  
*   source URI
*   start and end timestamps
*   transcript span when speech is present
*   visual frame samples
*   audio event evidence
*   object, OCR, face, or speaker metadata when available
*   feature provenance: model ID, extractor version, score, and stage
*   neighboring context the agent can request next
  
  
This structure is more important than any single model. Models change quickly. The retrieval architecture should keep source media, timestamps, features, and evaluation labels stable enough to survive model upgrades.  
  

## Step 1: Chunk Time Before You Embed

  
  
Continuous video is not directly searchable. You need temporal units.  
  
Common [chunking strategies](/guides/multimodal-chunking-strategies):  
  
*   **Fixed windows:** 2, 5, 10, or 30 second windows with overlap.
*   **Shot boundaries:** cuts detected from color histograms, embeddings, optical flow, or scene-change models.
*   **Speaker turns:** transcript segments bounded by diarization or ASR timestamps.
*   **Audio events:** sound activity windows from energy, spectral change, or audio-event models.
*   **Object tracks:** windows derived from when an object appears, moves, disappears, or interacts.
  
  
Fixed windows are predictable and easy to batch. Shot boundaries better preserve visual meaning. Speaker turns are useful for conversation search. Audio-event windows catch sounds that ASR ignores. Object tracks give agents spatial continuity.  
  
In practice, use more than one segmentation layer:  
  
*   short overlapping windows for precise retrieval
*   longer scene windows for context
*   transcript or speaker turns for speech search
*   object-track windows for grounded visual events
  
  
Every feature should carry timestamps. Without timestamps, the agent can retrieve a file but cannot inspect the moment.  
  

## Step 2: Extract Separate Feature Channels

  
  
Audio-visual retrieval is usually multi-index retrieval, not one magic vector. Each channel preserves a different kind of evidence.  
  

### Transcript Channel

  
  
ASR converts speech into text spans. It is strong for named entities, exact phrases, instructions, decisions, and conversational search.  
  
Weaknesses:  
  
*   misses non-speech sound
*   loses tone, timing, and visual context
*   fails when speech is noisy, multilingual, overlapping, or off-camera
*   cannot prove that a visible event actually happened
  
  
Use transcript search as a high-recall text channel, not as the whole system.  
  

### Audio Embedding Channel

  
  
Audio embeddings represent sound events, music, environmental noise, alarms, mechanical patterns, and speech acoustics. CLAP-style models align audio and text. Newer audio-video models such as [PE-AV](https://huggingface.co/facebook/pe-av-large) and [WAVE](https://huggingface.co/tsinghua-ee/WAVE-7B) also align audio with visual clips.  
  
This matters when the query names an event by sound:  
  
*   "glass breaking"
*   "siren before impact"
*   "machine squeal"
*   "applause gets louder"
*   "customer sounds frustrated"
  
  
An agent should be able to search these sounds even when no one says those words.  
  

### Video Embedding Channel

  
  
Video embeddings preserve motion and scene dynamics. Image embeddings over keyframes are useful, but they miss events that require motion. Video encoders such as [V-JEPA 2](https://huggingface.co/docs/transformers/model_doc/vjepa2), [VideoPrism](/models/videoprism-base-f16r288), and VideoLLaMA-style models help represent action, motion, and temporal state.  
  
Use video embeddings for:  
  
*   actions and gestures
*   sports plays
*   manufacturing process steps
*   camera motion
*   object movement
*   before-and-after visual changes
  
  
For fast events, short clips matter. For procedures, longer clips matter. Store both when the agent may need precise evidence and surrounding context.  
  

### Object, OCR, Face, and Scene Channels

  
  
Dense embeddings are good at fuzzy similarity. Structured channels are good at constraints.  
  
Examples:  
  
*   object: "forklift", "helmet", "red box"
*   OCR: "E113", "subtotal", "approved"
*   face: known person or cast member when permitted
*   scene: "warehouse dock", "checkout counter", "sports court"
*   metadata: camera ID, tenant, object URI, campaign, date, product SKU
  
  
Agents need these channels because tool calls usually include filters. A query like "find clips where a forklift moves near a worker in dock-3 after 8 PM" should not rely only on nearest-neighbor similarity.  
  

## Step 3: Search Channels Independently

  
  
A robust retrieval pipeline starts by searching several channels separately.

user task
  -> query planner
  -> transcript search
  -> audio embedding search
  -> video embedding search
  -> object/OCR/filter search
  -> candidate pool

Each stage should return candidates with:  
  
*   evidence ID
*   timestamp
*   score
*   stage name
*   modality
*   feature version
*   source URI
  
  
Do not throw away provenance. When the agent later sees a result, it should know whether the match came from a transcript phrase, an audio event, a visual motion pattern, or a structured filter.  
  

## Step 4: Fuse Results Without Pretending Scores Are Comparable

  
  
Scores from transcript search, vector search, object filters, and rerankers are not naturally comparable. A cosine score of 0.31 from an audio model is not the same thing as BM25 score 14 or a visual reranker score 0.72.  
  
Use rank-based fusion when scores come from different systems.  
  

### Reciprocal Rank Fusion

  
  
Reciprocal Rank Fusion is simple and strong:

RRF(candidate) = sum over result lists 1 / (k + rank\_in\_list)

The constant k, often 60, reduces the dominance of the top few positions. RRF works well because it only needs ranks, not calibrated scores.  
  
Use RRF when:  
  
*   multiple modalities return candidate lists
*   score scales are not calibrated
*   you need a stable first implementation
  
  

### Weighted Fusion

  
  
Weighted fusion lets the query planner emphasize channels:  
  
*   speech-heavy query: transcript weight high
*   sound-event query: audio weight high
*   action query: video weight high
*   compliance query: filters and OCR weight high
  
  
Weighted fusion is powerful but risky. If weights are hand-tuned globally, one modality can dominate. Track per-query-class metrics so improvements in one class do not hide regressions in another.  
  

### Diversification

  
  
Agents often need evidence variety, not twenty near-duplicates. Maximal Marginal Relevance helps balance relevance and diversity:

select next = relevance\_to\_query - lambda \* similarity\_to\_selected\_results

Use diversification when returning clips from long videos. It helps the agent inspect different moments before deciding whether to expand context.  
  

## Step 5: Rerank With the Full Question and Evidence

  
  
First-stage retrieval should optimize recall. Reranking should optimize precision.  
  
Rerankers can inspect richer inputs:  
  
*   user query
*   transcript span
*   keyframe captions
*   object names
*   OCR text
*   audio event labels
*   neighboring context
*   source metadata
  
  
For multimodal search, reranking can be:  
  
*   text cross-encoder over query plus transcript and generated captions
*   vision-language reranker over query plus keyframes
*   late-interaction document or visual retriever for pages and screenshots
*   LLM or VLM verifier for small candidate sets
  
  
Keep reranking bounded. An agent retrieval tool should return fast enough for iterative use. Rerank top 50, not top 5,000. Cache features and do not send raw video to a VLM unless the candidate set is already small.  
  

## Step 6: Expand Time After Retrieval

  
  
The top result is rarely the exact context an agent needs. A five-second audio event may require the preceding thirty seconds to explain what caused it.  
  
Use temporal expansion after ranking:  
  
*   include the matching window
*   include one or two neighboring windows
*   include parent scene boundaries
*   include speaker turn before and after
*   include object tracks that overlap the time range
  
  
Return the match and the context separately. The agent should know what matched and what is surrounding evidence.

{
  "match": {
    "source\_uri": "s3://ops/video/cam-4.mp4",
    "start\_sec": 184.0,
    "end\_sec": 191.0,
    "matched\_modalities": \["audio", "video"\],
    "matched\_stages": \["pe\_av\_embedding", "object\_filter"\]
  },
  "context": {
    "before\_sec": 30,
    "after\_sec": 20,
    "parent\_scene\_id": "scene\_00042"
  }
}

This prevents a common failure: the retriever finds the right moment, but the agent answers from too narrow a clip.  
  

## Design the Agent Tool Surface

  
  
An agent should not call a vague "search everything" function. It should call a bounded tool with explicit arguments.

{
  "tool": "search\_audio\_visual\_evidence",
  "input\_schema": {
    "query": "string",
    "collections": \["string"\],
    "time\_range": {"from": "string", "to": "string"},
    "modalities": \["transcript", "audio", "video", "object", "ocr"\],
    "filters": {},
    "top\_k": 20,
    "budget\_ms": 3000,
    "include\_context": true
  }
}

The output should be evidence, not a final answer:

{
  "results": \[
    {
      "source\_uri": "s3://support/call-42.mp4",
      "start\_sec": 512.0,
      "end\_sec": 526.0,
      "summary": "Customer says the unit clicks, then a red light flashes twice.",
      "matched\_modalities": \["transcript", "audio", "video"\],
      "scores": {
        "rrf": 0.041,
        "rerank": 0.83
      },
      "provenance": \[
        {"stage": "asr\_bm25", "rank": 2},
        {"stage": "audio\_video\_embedding", "rank": 1},
        {"stage": "vl\_reranker", "rank": 1}
      \]
    }
  \],
  "next\_actions": \["expand\_context", "retrieve\_frames", "request\_human\_review"\]
}

This matches the direction of modern agent systems. MCP exposes tools through schemas. LangChain and LlamaIndex agents call tools with structured arguments. OpenAI trace grading and agent evals inspect tool trajectories, not just final answers. Audio-visual retrieval should be built with the same discipline.  
  

## Evaluation: What to Measure

  
  
Do not evaluate audio-visual retrieval with one aggregate relevance score. Break the eval by query class.  
  

### Query Classes

  
  
*   transcript-only: "where does the customer mention a refund"
*   audio-only: "glass breaking"
*   visual-only: "operator opens the red panel"
*   audio-video: "horn sounds as forklift enters aisle"
*   OCR plus video: "screen shows E113 before shutdown"
*   object plus speech: "speaker says approved while signing the form"
*   negative: "there should be no clip where the logo appears"
  
  

### Metrics

  
  
*   Recall@k by modality
*   nDCG@k by query class
*   MRR for exact evidence
*   temporal IoU between retrieved and labeled time spans
*   modality attribution accuracy
*   false positive rate for negative queries
*   average cost per successful evidence retrieval
*   p95 latency by stage
*   stale work cancelled when the agent changes direction
  
  

### Ablations

  
  
Run ablations before making architecture claims:  
  
*   transcript only
*   audio embedding only
*   video embedding only
*   transcript plus video
*   audio plus video
*   all channels plus reranker
  
  
A good system should show where each modality helps. If audio never improves recall on audio-event queries, the audio channel is not pulling its weight. If video improves recall but hurts latency too much, use it only for query classes that need motion.  
  

## Common Failure Modes

  
  
**Transcript tunnel vision.** The system retrieves what was said, not what happened. Add audio and video query classes to the eval set.  
  
**Clip boundary loss.** The event crosses a chunk boundary. Use overlapping windows and parent scene expansion.  
  
**Score scale confusion.** Audio, video, and text scores are fused as if they are comparable. Use rank fusion or calibrated per-channel scores.  
  
**Duplicate evidence.** Top results all come from adjacent windows around the same moment. Add diversification and collapse overlapping windows.  
  
**Missing provenance.** The agent cannot tell why a clip matched. Store stage, modality, rank, score, model, and feature version.  
  
**Over-broad tool calls.** The agent searches every collection with every modality. Add required filters, top-k, budget, and cancellation.  
  
**Wrong context width.** A retrieved clip is correct but too short for reasoning. Return matched evidence separately from expanded context.  
  

## Mixpeek Implementation Pattern

  
  
In Mixpeek, model the pipeline as feature extraction plus a retriever that searches multiple feature channels. The exact model choices depend on your corpus, but the structure is stable.

from mixpeek import Mixpeek

mx = Mixpeek(api\_key="YOUR\_API\_KEY", namespace="my-namespace")

# Visual channel: a multimodal embedding for every segment.
mx.collections.create(
    collection\_name="cell\_video",
    source={"type": "bucket", "bucket\_ids": \["bkt\_your\_bucket"\]},
    feature\_extractor={"feature\_extractor\_name": "multimodal\_extractor", "version": "v1"},
)

# Sound channel: CLAP embeddings over the same videos' audio track, which carry
# motor noise and alarms that a transcript leaves out.
mx.collections.create(
    collection\_name="cell\_audio",
    source={"type": "bucket", "bucket\_ids": \["bkt\_your\_bucket"\]},
    feature\_extractor={"feature\_extractor\_name": "audio\_fingerprint\_extractor", "version": "v1"},
)

Then expose a retriever as an agent tool:

results = mx.retrievers.execute(
    "your-retriever-id",
    inputs={"query": "high pitched motor squeal followed by abrupt arm movement"},
)

The agent receives timestamped evidence with provenance. It can answer only when the evidence supports the claim, expand context when needed, or ask a human to review uncertain clips.  
  

## Design Checklist

  
  
*   Preserve raw media and source URIs.
*   Segment into short windows and parent scenes.
*   Store timestamps on every feature.
*   Use transcript, audio, video, object, OCR, and metadata channels where relevant.
*   Search channels independently before fusion.
*   Use RRF or calibrated fusion across score systems.
*   Rerank a bounded candidate set.
*   Collapse duplicates and diversify results.
*   Return matched evidence separately from expanded context.
*   Log modality, stage, score, model ID, and extractor version.
*   Evaluate by query class, not only aggregate quality.
*   Expose the retriever as a bounded agent tool with filters, limits, budgets, and cancellation.
  
  

## Key Takeaways

  
  
1\. Audio-visual retrieval is about timestamped evidence, not just video summaries.  
  
2\. Transcripts are necessary but incomplete. Agents need audio, video, object, OCR, and metadata channels.  
  
3\. Multi-index retrieval works best when each modality searches independently, then fusion and reranking combine the evidence.  
  
4\. Temporal expansion after retrieval is what lets the agent reason from context instead of isolated clips.  
  
5\. The safest agent interface returns evidence, provenance, budgets, and next actions. It does not hide uncertainty behind a final answer.  
  

## Further Reading

  
  
*   [Video Scene Segmentation](/guides/video-scene-segmentation)
*   [Video Temporal Grounding](/guides/video-temporal-grounding)
*   [Multi-Index Search Architecture](/guides/multi-index-search-architecture)
*   [Multi-Stage Retrieval](/guides/multi-stage-retrieval-how-agents-search-unstructured-data)
*   [Why does my podcast search find the right episode but not the moment?](/guides/why-does-my-podcast-search-find-the-right-episode-but-not-the-moment) for the pain-shaped entry point into this topic
*   [Agent Perception Evals](/guides/agent-perception-evals)
*   [PE-AV on Hugging Face](https://huggingface.co/facebook/pe-av-large)
*   [WAVE-7B on Hugging Face](https://huggingface.co/tsinghua-ee/WAVE-7B)
*   [MCP tool specification](https://modelcontextprotocol.io/specification/2025-11-25/server/tools)

## Run this on your own data

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this guide describes. Build starts at $25/mo for up to 1M vectors.

[Search your own archive](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)[Read Docs](/docs)

[Explore MVS](/mvs)[See pricing](/pricing)

## Related guides

[

Agent Perception

### Multimodal Chunking Strategies: How to Decompose Video, Audio, Images, and Documents for Search

A deep technical guide to chunking strategies across every content modality. Covers why text splitting fails for rich media, the algorithms behind video scene segmentation, audio diarization boundaries, document layout parsing, image tiling, and how to align chunks across modalities into unified retrieval units.

Read guide →](/guides/multimodal-chunking-strategies)[

Agent Perception

### Audio Fingerprinting: How Agents Recognize a Specific Recording in Noise

A first-principles guide to audio fingerprinting -- the algorithm behind Shazam-style recognition that identifies an exact recording even when it is noisy, pitch-shifted, or buried in other sound. Covers spectrogram peak picking, the constellation map, combinatorial landmark hashing, inverted-index voting with time-offset alignment, and how identity-level audio search differs from semantic similarity for AI agents.

Read guide →](/guides/audio-fingerprinting-constellation-landmark-hashing)[

Agent Perception

### Mask-Aware Retrieval for AI Agents: Segment First, Search Crops, Then Reason

Learn how segmentation masks, foreground crops, region embeddings, and spatial metadata make visual search more precise for agents that need to inspect images, video frames, screenshots, and product media.

Read guide →](/guides/mask-aware-retrieval-for-ai-agents)