Temporal Grounding: Query In, Timestamp Out

Frame-level similarity search on video returns stills. You embed the query, embed sampled frames, take nearest neighbors, and the best answer available is that frame 3,840 looks relevant: no start, no end, no context. A 90 minute match is 162,000 frames at 30 fps, and a still from the middle of it is not an answer. Temporal grounding returns intervals instead, which is what turns a video from an opaque blob into a queryable timeline.

The core loop is per-segment scoring plus interval merging. Sample frames at 1 fps, embed the frames and the query with the same model (CLIP or SigLIP), score every frame against the query embedding, slide a window and average the scores, then merge the overlapping windows that clear a threshold into one segment with a start and an end, like start 134.2 to end 141.8 at score 0.94. Everything more sophisticated keeps that same contract. Proposal networks such as 2D-TAN and Moment-DETR learn the boundaries directly, dense captioning turns grounding into a text search over timestamped captions, and production systems decompose the video into timestamped feature streams (frames, objects, faces, transcripts) and intersect intervals with temporal joins when one model cannot answer alone.

This is the deliberate opposite of reverse video search, which is clip in and trajectory match out. Temporal grounding is text in and interval out. Score it on Recall at 1 with IoU 0.5, where the best models clear 60% on ActivityNet Captions and 70% on the shorter, more precise Charades-STA.