Video Intelligence: How to Make Raw Video Footage Searchable
How to make raw video footage searchable: transcripts, on-screen text, descriptions and embeddings per timestamped segment, with storage and cost per hour worked out.
How do you make raw video footage searchable? Cut each video into short segments, then run AI models on every segment: speech-to-text for what is said, OCR for text on screen, a vision-language model for a written description, and an embedding model for what the segment looks like. Store every result with its start and end time. A search then matches a question against the text and the embeddings and returns the exact moment with its timestamp.
| Signal | Finds | Misses |
|---|---|---|
| Transcript (speech-to-text) | What people say, names and quotes | Anything silent on screen |
| On-screen text (OCR) | Titles, slides, signs, prices, captions burned into the video | Speech, actions |
| Written description | Actions, objects, people and setting, as text you can search | Detail the model did not mention |
| Embedding | Scenes that look like your query or a reference image | Exact names, codes and numbers |
| Face identity | Every appearance of one person | People you have not enrolled |
Updated October 2026: the code sample below uses Mixpeek's current features API, and the storage figures are worked out from vector sizes.
Video is the fastest-growing data type in enterprise systems, yet it remains the hardest to search. Unlike text, which can be indexed by keywords, or images, which can be tagged with labels, video contains temporal information — events unfold over time across visual, audio, and textual channels simultaneously.
Video intelligence is the discipline of using AI to extract structured, searchable information from this temporal complexity. This post covers what video intelligence involves, how the processing pipeline works, and how to implement it in practice.
What does video intelligence pull out of raw footage?
A comprehensive video intelligence pipeline produces multiple layers of structured data from a single video file:
- Scene boundaries — Timestamps where the visual content changes significantly (cuts, transitions, topic shifts)
- Transcription — Speech-to-text with speaker diarization (who said what and when)
- Visual embeddings — Per-frame or per-scene vector representations for similarity search
- Object detection — Identified objects with bounding boxes and confidence scores per frame
- Face identity — Detected and recognized faces linked to identity embeddings
- Action recognition — Classified activities (walking, presenting, driving) with temporal spans
- OCR — Text visible in frames (slides, signs, documents shown on screen)
- Audio events — Non-speech sounds (music, applause, alarms)
How does raw footage become searchable, step by step?
Stage 1: Ingestion and Preprocessing
Raw video files are decoded and normalized. This includes extracting audio tracks, generating thumbnail frames, detecting the video resolution and codec, and validating file integrity. Large videos are chunked for parallel processing.
Stage 2: Scene Detection
Scene boundary detection splits the video into semantically meaningful segments. The most common approach computes visual similarity between consecutive frames using embedding distance — when the distance exceeds a threshold, a scene boundary is inserted. This avoids over-processing similar consecutive frames and creates natural retrieval units.
Stage 3: Multi-Model Extraction
Each scene is processed by multiple AI models in parallel:
import os
from mixpeek import Mixpeek
client = Mixpeek(
api_key=os.environ["MIXPEEK_API_KEY"],
namespace=os.environ["MIXPEEK_NAMESPACE"],
)
# Index every video in a bucket. video_search stores timestamped segments with
# an embedding, the speech transcript and a written description; onscreen_text
# adds OCR and faces adds face identity.
collection = client.collections.create(
collection_name="video-intelligence",
source={"type": "bucket", "bucket_ids": [os.environ["BUCKET_ID"]]},
features=["video_search", "onscreen_text", "faces"],
)Stage 4: Temporal Indexing
Every extracted feature is stored with its timestamp range. This enables frame-accurate retrieval — when a user searches for "person writing on whiteboard", the system returns not just the video but the exact timestamp where that action occurs. This is what separates video intelligence from basic video tagging.
Stage 5: Retrieval
With all features indexed, multiple search modalities become possible:
- Text-to-video — Describe what you are looking for in natural language
- Image-to-video — Upload a reference image to find similar visual content
- Face search — Upload a photo to find all appearances of that person
- Transcript search — Search spoken content with keyword or semantic matching
- Combined queries — "Find scenes where [person] talks about [topic]"
How long does processing take, and how much storage does the index add?
Processing Time
Processing time grows with the length of the footage and with each feature you turn on. On-screen text and written descriptions add the most, so enable them only where you need them. Run ingestion asynchronously and use a webhook to learn when a video is ready.
Deduplication
Without deduplication, a person visible for 30 seconds generates hundreds of near-identical face embeddings. Scene-level deduplication groups similar consecutive detections, keeping only the highest-quality sample per scene.
Storage
Each indexed segment stores one vector per embedding model plus its text fields. A 3,072-dimension float32 vector is 12 KB and a 1,408-dimension one is 5.6 KB, so an hour of footage cut into 30-second segments adds about 1.5 MB of vectors. Keyframe thumbnails, if you keep them, usually cost more than the vectors do.
The index is small next to the footage. YouTube recommends uploading 1080p at 8 Mbps (YouTube upload settings), which is 3.6 GB an hour. Worked out per hour of video:
| What is stored | Per hour | Share of a 3.6 GB file |
|---|---|---|
| 120 segments of 30 s, one 3,072-dimension vector each (12 KB) | 1.5 MB | 0.04% |
| 360 segments of 10 s, one 1,408-dimension vector each (5.6 KB) | 2.0 MB | 0.06% |
| Transcript of an hour of speech, about 9,000 words | about 55 KB | under 0.01% |
| One 150 KB keyframe per 30 s segment | 18 MB | 0.5% |
Bandwidth works the same way. Indexing reads each file once from your bucket, and a search returns timestamps and metadata. Playback still streams from the original file, starting at the matched second, so search adds no copy of the video.
Who makes their footage searchable, and why?
- Media libraries — Search across thousands of hours of video content by visual similarity or spoken content
- Security and surveillance — Find specific events, people, or objects across camera feeds
- Ad tech — Analyze creative performance by detecting scenes, talent, and messaging patterns
- Education — Make lecture recordings searchable by topic, slide content, and speaker
- Legal and compliance — Search deposition videos and surveillance footage for specific events
Video intelligence transforms video from a storage cost into a searchable asset. Explore our glossary entry on video intelligence for more on the core concepts, or see how to build a video search engine in our FAQ.
Frequently asked questions
How much storage does AI video search add compared to the video itself?
Well under 1% for the searchable index. An hour of 1080p video at YouTube's recommended 8 Mbps is 3.6 GB, and 120 thirty-second segments with one 3,072-dimension vector each come to about 1.5 MB, or 0.04%. Transcripts add kilobytes. Thumbnails are the largest extra if you keep them: 18 MB an hour at one 150 KB keyframe per segment.
How do I extract text that appears on screen in a video so it can be searched?
Run OCR on frames from each segment and store the text with the segment's timestamps. Slides, product names, prices and burned-in captions then match keyword queries even when nobody says them aloud. In Mixpeek this is the onscreen_text feature, priced at $0.10 per minute on top of video search. Our guide on why video search misses on-screen text covers the failure in detail.
What is the difference between video intelligence and video tagging?
Tagging attaches labels to a whole file, so a search returns the file and you scrub for the moment. Video intelligence stores what it extracts per segment with start and end times, so a search returns the moment. It also keeps the transcript, on-screen text and descriptions as text you can query in your own words.
How much does it cost to make an hour of video searchable?
On Mixpeek, video search is $0.05 per minute, $3 an hour, and covers timestamped segments with an embedding, the speech transcript and a written description. On-screen text adds $0.10 per minute and faces another $0.10. The live rates are on the pricing page and at GET /v1/collections/features. For model-level costs, see what transcribing an hour of audio costs and how many tokens an hour of video is.
How long should each searchable video segment be?
Short enough that a result lands near the moment someone wants. Mixpeek's video_search uses fixed 30-second segments, up to max_video_segments (default 10, at most 50) per file, and the covered_s field shows when a long file was cut short. For long recordings or tighter moments, the multimodal extractor splits by time, scene change or silence, with a 10-second default. See how to find one moment in hours of footage.
Do I have to move my video to a new platform to search it?
No. Mixpeek reads the files where they already live, in S3, GCS or any bucket, and stores the index separately. The originals stay in your bucket, and search results point back to them by file and timestamp.