The Problem: Video Is Not a Single Frame
When a human watches a video and you ask "find the moment where the goalkeeper dives left," they scrub to the right timestamp in seconds. When an AI agent receives the same query, it faces a fundamentally harder problem: a 90-minute football match is 162,000 frames at 30fps. Treating each frame as an independent image and running similarity search returns individual frames with no temporal context — you get a still image of a goalkeeper mid-dive but no start time, no end time, and no understanding that the dive is part of a larger play.
Video temporal grounding solves this. It takes a natural language query and a video, and returns precise time intervals — start and end timestamps — where the described event occurs. This is the capability that transforms video from an opaque blob into a queryable timeline.
Why This Matters for Agents
An AI agent connected to a video library via MCP or LangChain needs to answer queries like:
Without temporal grounding, the agent can only do frame-level similarity search: embed the query, embed sampled frames, return the nearest neighbors. This gives you individual frames ranked by visual similarity, but no temporal boundaries. The agent cannot say "this event starts at 2:14 and ends at 2:31" — it can only say "frame 3,840 looks relevant."
Temporal grounding gives agents the ability to reason about video as a sequence of events with beginnings and endings, not a bag of frames.
Approach 1: Sliding Window with Frame Sampling
The simplest approach samples frames at fixed intervals (e.g., 1 frame per second), embeds each frame with a vision encoder like CLIP or SigLIP, then runs a sliding window over the embedding sequence to find contiguous regions that match the query.
How it works:
1. Sample N frames uniformly from the video 2. Encode each frame with a vision encoder: \`f_1, f_2, ..., f_N\` 3. Encode the text query with the same model's text encoder: \`q\` 4. Compute similarity \`sim(f_i, q)\` for each frame 5. Apply a sliding window of width W: for each window position, average the similarities 6. Threshold or rank windows by average similarity 7. Merge overlapping windows above threshold into temporal segments
Pseudocode:
\`\`\`python import numpy as np
def sliding_window_grounding(frame_embeddings, query_embedding, window_size=5, stride=1, threshold=0.3): # frame_embeddings: (N, D) array of frame features # query_embedding: (D,) query feature vector similarities = frame_embeddings @ query_embedding # (N,)
windows = [] for start in range(0, len(similarities) - window_size + 1, stride): end = start + window_size score = similarities[start:end].mean() if score > threshold: windows.append((start, end, float(score)))
# Merge overlapping windows merged = [] for start, end, score in sorted(windows): if merged and start <= merged[-1][1]: prev_start, prev_end, prev_score = merged[-1] merged[-1] = (prev_start, max(prev_end, end), max(prev_score, score)) else: merged.append((start, end, score))
return merged \`\`\`
Pros: Simple to implement, works with any frame-level encoder, no training required.
Cons: No learned temporal reasoning — the model has no understanding of motion, causality, or event boundaries. Window size is a hyperparameter that must be tuned per domain. Misses events that span variable durations.
Approach 2: Proposal-Based Methods (2D-TAN, Moment-DETR)
Proposal-based methods treat temporal grounding as a detection problem: generate candidate time intervals (proposals), score each proposal against the query, and return the highest-scoring ones.
2D Temporal Adjacent Networks (2D-TAN)
2D-TAN represents all possible temporal segments in a video as a 2D map where the x-axis is the start time and the y-axis is the end time. Each cell (i, j) represents the segment from time i to time j. The model:
1. Extracts frame features with a pretrained video encoder 2. Pools features within each candidate segment (i, j) to get a segment-level representation 3. Fuses the segment representation with the query text embedding 4. Predicts a score for each (start, end) pair 5. Applies non-maximum suppression to get the final predictions
The 2D map structure lets the model reason about segments of all lengths simultaneously, and adjacent segments share computation through the pooling operation.
Moment-DETR
Moment-DETR adapts the DETR (DEtection TRansformer) architecture from object detection to temporal grounding. Instead of detecting bounding boxes in images, it detects temporal segments in video:
1. A video encoder produces frame-level features 2. A text encoder produces the query embedding 3. Cross-attention layers fuse video and text features 4. A set of learned "moment queries" (analogous to DETR's object queries) attend to the fused features 5. Each moment query predicts a (center, width) pair defining a temporal segment, plus a confidence score 6. Hungarian matching during training assigns predictions to ground-truth segments
Key insight: Moment-DETR eliminates hand-crafted proposals entirely. The moment queries learn to specialize in different types of temporal patterns — some learn short events, others learn long sequences, and some learn to focus on specific modalities (visual vs. audio cues).
QD-DETR and Extensions
QD-DETR (Query-Dependent DETR) improves on Moment-DETR by making the video encoder query-aware: the text query modulates which video features get emphasized before the detection head runs. This is important because a query like "person opens a door" should weight spatial features (door location) differently than "crowd cheering" which should weight audio features.
Approach 3: Dense Video Captioning with Timestamps
Dense video captioning flips the temporal grounding problem: instead of starting with a query and finding timestamps, it starts with a video and generates timestamped descriptions for every notable event. This creates a searchable index of moments.
Architecture:
1. A video encoder processes the full video into frame features 2. A temporal proposal module identifies event boundaries 3. For each proposed segment, a captioning decoder generates a natural language description 4. The output is a list of (start_time, end_time, caption) triples
Example output:
\`\`\` [ {"start": 0.0, "end": 3.2, "caption": "A woman walks into a kitchen"}, {"start": 3.2, "end": 7.8, "caption": "She opens the refrigerator and takes out a bottle of milk"}, {"start": 7.8, "end": 12.1, "caption": "She pours milk into a glass on the counter"}, {"start": 12.1, "end": 15.4, "caption": "She drinks from the glass and puts it in the sink"} ] \`\`\`
Once you have dense captions, temporal grounding becomes text search: embed the query, embed each caption, return the segments whose captions are most similar. This is often more accurate than direct visual grounding because the captions encode high-level semantics that visual similarity misses.
Models to know:
Approach 4: Hierarchical Feature Decomposition
The most powerful approach decomposes video into multiple independent feature streams — visual embeddings, object detections, face identities, audio transcripts, scene classifications — and grounds queries against the appropriate feature stream.
Why this works: A query like "find the moment John speaks about the merger" requires:
1. Face identity to find frames where John appears 2. Speaker diarization to find segments where John is speaking 3. Transcript search to find where "merger" is mentioned 4. Temporal intersection to find the overlap of all three
No single model can do all of this. Hierarchical decomposition runs specialized extractors, stores each feature stream in its own collection with timestamps, then composes queries across collections using temporal joins.
Temporal join pseudocode:
\`\`\`python def temporal_join(segments_a, segments_b, max_gap=1.0): """Find overlapping or near-overlapping segments from two feature streams.""" joined = [] for a_start, a_end, a_data in segments_a: for b_start, b_end, b_data in segments_b: overlap_start = max(a_start, b_start) overlap_end = min(a_end, b_end) if overlap_end - overlap_start >= -max_gap: joined.append(( max(a_start, b_start), min(a_end, b_end), {a_data, b_data} )) return joined \`\`\`
This is where temporal grounding connects to multi-stage retrieval. Each feature stream is a collection. Each query stage searches or filters one collection. Temporal joins compose the results. The final output is a set of moments that satisfy all constraints simultaneously.
Practical Architecture: Building a Moment Retrieval Pipeline
Here is how these approaches compose into a production system:
Step 1: Decompose at Ingest
When a video is ingested, run multiple extractors in parallel:
Each extractor writes to its own collection. Every feature carries a timestamp linking it back to the source video timeline.
Step 2: Ground Queries with Multi-Stage Retrieval
A natural language query triggers a multi-stage retriever pipeline:
\`\`\` Stage 1: semantic_search on frame_embeddings → returns candidate frame ranges scored by visual similarity
Stage 2: filter on object_detections → keeps only segments containing detected objects matching the query
Stage 3: feature_search on transcripts → finds segments where spoken words match the query
Stage 4: moment_group → groups overlapping results into unified moment segments → each moment has a start time, end time, and composite score
Stage 5: rerank → cross-encoder re-scores the top moments using full context \`\`\`
The \`moment_group\` stage is the key temporal grounding primitive. It takes results from multiple upstream stages — each carrying timestamps — and merges overlapping or adjacent results into coherent moments. Results from different feature streams that overlap in time get fused into a single moment with features from all streams.
Step 3: Return Structured Moments
The pipeline returns moments, not individual frames:
\`\`\`json { "moments": [ { "start": 134.2, "end": 141.8, "score": 0.94, "features": { "visual_similarity": 0.87, "transcript_match": "...discusses the merger with Acme Corp...", "face_identity": "john_smith", "objects_detected": ["podium", "microphone", "presentation_slide"] } } ] } \`\`\`
An agent receiving this response can answer the original question precisely: "John discusses the merger from 2:14 to 2:21 in the earnings call recording."
Evaluation Metrics
Temporal grounding is typically evaluated with:
Current state of the art on the ActivityNet Captions benchmark: R@1 IoU=0.5 is above 60% for the best models. On Charades-STA (shorter videos, more precise queries): R@1 IoU=0.5 exceeds 70%.
When to Use Each Approach
| Approach | Best for | Latency | Accuracy |
| Sliding window | Quick prototypes, fixed-length events | < 50ms | Low |
| 2D-TAN / Moment-DETR | Single-query grounding on short videos | 100-500ms | High |
| Dense captioning | Building a searchable video index | Offline (minutes) | High |
| Hierarchical decomposition | Multi-constraint queries, production systems | < 100ms (after ingest) | Highest |