Scene Segmentation: How a Video Becomes Searchable Scenes

A raw video file has no scenes. It is a continuous stream of pixels and audio samples, 108,000 frames per hour at 30 fps, with no natural boundaries until something draws them. You cannot run search on that directly: embedding every frame is wasteful and redundant, because neighbors inside a shot are nearly identical.

So the pipeline draws boundaries in two levels. First, shot boundary detection compares consecutive frames and watches the difference signal. A hard cut produces one sharp spike that a simple histogram comparison catches at over 1000 fps on a CPU. A dissolve or fade spreads the change across many frames, so no single frame pair clears a fixed threshold; adaptive detectors keep a rolling window of recent scores and flag a boundary when the current score beats the local mean by 2 to 3 sigma. Second, shots group into scenes. A shot is one continuous camera take; a scene is a semantic unit that can span many shots, like a conversation that cuts between two camera angles. Cleanup rules keep the segments useful: anything under 2 seconds merges into its more similar neighbor, anything over 60 seconds splits at audio silence.

The scene is the retrieval unit. You get one embedding per scene instead of per frame, results that carry start and end timestamps, and storage down 10 to 100x. Every downstream capability, from frame sampling to temporal grounding, assumes this step already ran.