The Blind Agent Problem
Most AI agents today operate in a text-only world. They can read documents, call APIs, and write code, but they cannot look at an image, watch a video, or listen to an audio recording. When a user asks "find every frame in this surveillance video where a person enters through the back door," the agent has no way to even begin.
This is not a hypothetical limitation. Consider what happens when you connect an LLM to a file system containing thousands of media files:
The gap between what agents can reason about and what organizations actually store is enormous. Estimates from IDC suggest that over 80% of enterprise data is unstructured -- images, videos, audio recordings, scanned documents, CAD files. An agent that cannot perceive this content is operating with a fraction of the available information.
What Perception Means for an Agent
Human perception works in parallel across multiple senses. You can glance at a photograph and simultaneously recognize faces, read text, estimate the setting, and notice unusual objects. Your brain decomposes the scene into multiple features without conscious effort.
Agent perception works the same way, but the decomposition must be explicit. A perception pipeline breaks unstructured content into structured, searchable features:
Raw Media File
|
v
+-------------------+
Feature Extraction
<-- Multiple specialized models run in parallel
+-------------------+
v v v v
Objects Faces Caption Transcript
[bbox] [embed] [text] [text+ts]
v v v v
+-------------------+
Feature Index
<-- Vectors + metadata stored together
+-------------------+
|
v
+-------------------+
Retrieval API
<-- Agent calls this as a tool
+-------------------+
Each extractor produces a different type of feature. Object detection outputs bounding boxes and labels. Face detection outputs identity embeddings. Scene captioning outputs natural language descriptions. Transcription outputs timestamped text. The key insight is that no single model captures everything -- perception requires an ensemble.
The Perception Pipeline
Stage 1: Ingestion and Segmentation
Before any model runs, the raw media must be segmented into processable units:
Video is split into segments at scene boundaries. A scene change detector (typically based on frame-to-frame histogram differences or learned features) identifies transition points. Each segment becomes an independent unit for downstream extraction. Common segment lengths range from 5 to 30 seconds.
Audio is segmented by voice activity detection (VAD). Silence regions are trimmed, and the remaining audio is split at natural pause points. Speaker diarization can further segment by who is speaking.
Documents are split by structural elements -- pages, sections, paragraphs. For PDFs, layout analysis identifies text blocks, tables, figures, and headers. Each structural element becomes a separate unit.
Images are typically processed as single units, though very large images (satellite imagery, microscopy) may be tiled.
# Pseudocode: video segmentation
def segment_video(video_path, method="scene_detect"):
if method == "scene_detect":
scenes = detect_scene_boundaries(video_path)
segments = []
for start, end in scenes:
if (end - start) > MIN_SEGMENT_DURATION:
segments.append(extract_clip(video_path, start, end))
return segments
elif method == "fixed_interval":
duration = get_duration(video_path)
return [
extract_clip(video_path, t, t + INTERVAL)
for t in range(0, duration, INTERVAL)
]
Stage 2: Feature Extraction
Each segment passes through multiple extraction models in parallel. The choice of extractors depends on the content type and the queries you expect:
Visual Embeddings (CLIP, SigLIP, DINOv2) produce dense vector representations of visual content. These enable semantic similarity search -- "find frames that look like this reference image" or "find scenes matching this text description." CLIP-family models are particularly useful because they map both images and text into the same embedding space, enabling cross-modal search.
Object Detection (DETR, YOLO, Grounding DINO) identifies and localizes objects within frames. The output is a set of bounding boxes, each with a class label and confidence score. Open-vocabulary detectors like Grounding DINO can detect objects from arbitrary text descriptions, not just a fixed class list.
Face Detection and Recognition (RetinaFace, ArcFace) detects faces, extracts identity embeddings, and optionally matches against a known gallery. The identity embedding allows searching across a video corpus for "all appearances of this person" without knowing their name.
Scene Captioning (BLIP-2, Florence-2, PaliGemma) generates natural language descriptions of visual content. These descriptions become searchable text features. A caption like "a person in a hard hat inspecting a concrete foundation on a construction site" enables queries that no single object detector could handle.
OCR (TrOCR, PaddleOCR) extracts text visible in images and video frames -- signs, labels, documents, screens, whiteboards. For document-heavy workflows, OCR is often the highest-value extractor.
Transcription (Whisper) converts spoken audio to timestamped text. Combined with speaker diarization, this produces a structured transcript where every utterance is attributed to a speaker and anchored to a time range.
Audio Embeddings (CLAP) produce vector representations of audio content, enabling similarity search for sounds, music, and ambient audio independent of speech.
Stage 3: Indexing
Extracted features are indexed for retrieval. The index must support multiple query types:
Vector search for embeddings -- nearest-neighbor lookup in high-dimensional space. Used for semantic similarity ("find visually similar frames") and cross-modal search ("find frames matching this text query").
Structured filters for metadata -- exact match and range queries on object labels, face IDs, timestamps, confidence scores. Used for precise lookups ("all frames containing a stop sign with confidence > 0.9").
Full-text search for captions and transcripts -- keyword and phrase matching with BM25 or similar. Used for lexical queries where exact wording matters.
A production index combines all three. A single query might filter by time range, search captions for a keyword, and rank results by visual embedding similarity -- a multi-stage retrieval pipeline.
Stage 4: Retrieval as a Tool
The agent interacts with the perception pipeline through a retrieval API exposed as a tool. In the Model Context Protocol (MCP), this looks like:
{
"name": "search_media",
"description": "Search indexed media files by visual similarity, text, objects, faces, or metadata filters",
"parameters": {
"query": "text description of what to find",
"filters": {
"file_type": ["video", "image"],
"date_range": {"start": "2026-01-01", "end": "2026-05-01"},
"objects": ["hard hat", "safety vest"],
"min_confidence": 0.8
},
"top_k": 10
}
}
The agent does not need to know which models produced the features or how the index is structured. It formulates a query in natural language with optional structured filters, and the retrieval system handles the rest. This separation of concerns is critical -- it means the perception pipeline can be upgraded (better models, different extractors) without changing the agent's code.
Choosing Extractors: The Coverage vs. Cost Tradeoff
Running every extractor on every file is expensive. A 10-minute video processed through visual embeddings, object detection, face recognition, scene captioning, OCR, and transcription might cost $0.50-2.00 in compute and take 3-5 minutes. At scale (millions of files), you need a strategy.
The Feature Matrix
Map your expected query types to the extractors that serve them:
| Query Type | Required Extractors | Example |
| "Find similar scenes" | Visual embeddings | Product catalog dedup |
| "Find all mentions of X" | Transcription + text embeddings | Meeting search |
| "Find person Y" | Face detection + recognition | Security, media |
| "Find objects of type Z" | Object detection | Inventory, safety |
| "What's happening in this scene?" | Scene captioning | Video summarization |
| "Read text in images" | OCR | Document processing |
| "Find similar sounds" | Audio embeddings | Audio cataloging |
Tiered Extraction
Not all content needs the same depth of processing:
Tier 1 (always run): Visual embeddings + transcription. These two extractors cover the broadest range of queries at the lowest cost. Every file gets at least a vector representation and a text transcript.
Tier 2 (conditional): Object detection + scene captioning. Run these when the content type warrants it (e.g., surveillance footage benefits from object detection, training videos benefit from captioning).
Tier 3 (on-demand): Face recognition, OCR, audio embeddings. Run these when a specific use case requires them, or when a Tier 1/2 search returns ambiguous results and the agent needs more detail.
# Pseudocode: tiered extraction
def choose_extractors(file_type, use_case, budget):
tier1 = ["visual_embeddings", "transcription"]
tier2_map = {
"surveillance": ["object_detection", "face_identity"],
"training_video": ["scene_caption", "ocr"],
"podcast": ["speaker_diarization"],
"product_catalog": ["object_detection", "ocr"],
}
extractors = tier1 + tier2_map.get(use_case, [])
if budget == "high":
extractors += ["face_identity", "ocr", "audio_embeddings"]
return deduplicate(extractors)
Embedding-Based Memory
Perception without memory is useless. An agent that can analyze a single image but cannot recall what it saw yesterday has no persistent understanding of its environment.
Embeddings solve this by converting perceptions into storable, searchable vectors. When the agent perceives new content, the resulting embeddings are added to an index that grows over time. This index becomes the agent's long-term multimodal memory.
How Memory Queries Work
When the agent needs to recall something, it formulates a retrieval query:
1. "Have I seen this before?" -- The agent embeds the current input and searches for nearest neighbors in its memory index. High similarity scores indicate prior exposure.
2. "What do I know about X?" -- The agent converts the concept X into a text embedding and searches across all modalities. Visual embeddings of red cars, captions mentioning "red sedan," and transcript segments discussing "the red vehicle" all surface in a single query.
3. "What changed since last time?" -- The agent compares current feature extractions against stored features for the same source. New objects, missing faces, or shifted scene descriptions indicate change.
The Modality Gap
A well-known challenge in cross-modal retrieval is the modality gap -- the systematic offset between text embeddings and image embeddings in joint spaces like CLIP. Even when a text query and an image are semantically identical, their embeddings cluster in different regions of the vector space.
Text embeddings
cluster here
* *
* * *
* *
<-- modality gap
* *
* * *
* *
Image embeddings
cluster here
This gap reduces cross-modal recall. Mitigation strategies:
Agent Perception Patterns
Pattern 1: Perception-First Agent
The agent always perceives before reasoning. Every incoming media file is processed through the extraction pipeline before the LLM sees it.
User: "Analyze this surveillance video for safety violations"
|
v
[1. Extract features from video]
- Objects detected: person (no hard hat), forklift, ladder
- Scene caption: "warehouse floor, person climbing ladder near forklift"
- Transcript: (no speech detected)
|
v
[2. LLM reasons over structured features]
"I detected a person climbing a ladder without a hard hat
near an active forklift. This violates OSHA regulations
29 CFR 1926.100 (head protection) and 1926.602
(material handling equipment clearance)."
When to use: The agent's task requires understanding the content of specific files. The files are known in advance.
Pattern 2: Retrieval-Augmented Perception
The agent searches a pre-indexed corpus to find relevant content, then reasons over the results.
User: "Find all instances of our logo being displayed incorrectly"
|
v
[1. Agent searches memory: visual similarity to reference logo]
- 47 results with similarity > 0.7
|
v
[2. Agent filters: OCR + object detection for logo region]
- 12 results where logo text or proportions differ from reference
|
v
[3. Agent reports findings with timestamps and thumbnails]
When to use: The corpus is too large to process on-demand. Features are pre-extracted and indexed.
Pattern 3: Iterative Perception
The agent refines its perception over multiple rounds. Initial broad extraction reveals areas that need deeper analysis.
Round 1: Visual embeddings identify 200 candidate frames
Round 2: Object detection on candidates narrows to 45 frames with relevant objects
Round 3: Scene captioning on finalists produces detailed descriptions
Round 4: LLM reasons over descriptions to answer the original question
When to use: Compute budget is limited. The agent needs to be selective about which extractors to run.
Evaluation: Measuring Perception Quality
Perception quality is measured at two levels:
Extractor-Level Metrics
Each extractor has its own evaluation metrics:
System-Level Metrics
The full perception pipeline is evaluated end-to-end:
Perception recall: Given a set of ground-truth facts about a media file (objects present, text visible, words spoken), what fraction does the pipeline extract? Low recall means the agent is missing information.
Perception precision: Of the features the pipeline extracts, what fraction is correct? Low precision means the agent is reasoning over wrong information -- a more dangerous failure mode than low recall.
Query-answer accuracy: Given a question about a media file and the perception pipeline's output, can an LLM answer correctly? This is the ultimate measure -- it captures whether the pipeline extracts the right features for the questions users actually ask.
# Pseudocode: perception evaluation
def evaluate_perception(test_set):
results = []
for item in test_set:
# Extract features
features = pipeline.extract(item.media_file)
# Check recall: did we find the known facts?
found = set()
for fact in item.ground_truth_facts:
if fact_matches_any_feature(fact, features):
found.add(fact)
recall = len(found) / len(item.ground_truth_facts)
# Check query-answer accuracy
for question in item.test_questions:
context = pipeline.retrieve(question.text, item.media_file)
answer = llm.generate(question.text, context)
correct = judge_answer(answer, question.expected)
results.append({
"recall": recall,
"question": question.text,
"correct": correct,
})
return aggregate(results)
Common Pitfalls
Running one model and calling it perception. CLIP embeddings alone cannot tell you who is in a video, what text is visible, or what was said. Perception requires multiple extractors. The whole point of decomposition is that each model captures a different aspect of the content.
Ignoring temporal alignment in video. Object detection on isolated frames misses motion and context. A person "entering a room" is a temporal event spanning multiple frames. Frame-level features must be aggregated into segment-level understanding, typically by pooling or sequence modeling over consecutive frame features.
Treating all embeddings as interchangeable. A CLIP embedding and a DINOv2 embedding of the same image are not comparable -- they live in different vector spaces with different dimensionalities and different semantic structures. Never mix embeddings from different models in the same index without explicit alignment.
Skipping confidence thresholds. Object detectors and face recognizers produce predictions at all confidence levels. Without thresholds, the agent will reason over false detections. Set thresholds based on your precision requirements, not the model defaults.
Not versioning your extractors. When you upgrade a model (CLIP ViT-B → ViT-L, Whisper v2 → v3), existing embeddings become incompatible. You need a migration strategy: re-extract with the new model, or maintain a version tag on each embedding so the retrieval system uses the right index. See the embedding portability guide for details.