How do I find specific moments across thousands of hours of robot or dashcam recordings?
Narrow by telemetry first, then search what is left by meaning. Use the sensor data that came with the footage (GPS, speed, braking, IMU, robot state) to keep only the time windows that could hold your event, and skip stretches where nothing moves. Cut the remaining video into fixed 10 to 30 second segments, embed each one with a video or vision-language model, and store its start and end time next to the vehicle or robot ID. A query such as "cyclist passing on the right in the rain" then ranks a few thousand candidate segments.
Why does scrubbing or tagging stop working at fleet scale?
Continuous capture is mostly nothing happening: a parked car, a robot waiting for its next task, an empty highway. The moments you want are rare, short and described in words nobody tagged ahead of time. Ten thousand hours of 1080p video at YouTube's recommended 8 Mbps (upload settings) is 36 TB. Nobody watches that, and a fixed list of event tags only finds the events someone predicted.
What are the ways to search a large footage archive?
| Approach | Finds | Misses | Cost driver |
|---|---|---|---|
| Telemetry and event triggers (hard braking, collision flags, robot faults) | Events the sensors can see, with exact timestamps | Anything without a sensor signature, such as "a dog on the road" | Almost free; the data already exists |
| Frame embeddings (CLIP or SigLIP at 1 frame a second) | Objects and scenes that look like the query | Motion and order: "merging", "dropping", "about to" | Vector count: 36 million for 10,000 hours |
| Segment embeddings (Gemini Embedding 2, Qwen3-VL-Embedding) | Short actions and scenes over several seconds | Fine detail inside a long segment | Billed frames or segments |
| Captions from a vision-language model, searched as text | Exact nouns, colors, signs, and text you can filter on | What the model left out of its caption | Model calls per segment |
| Hosted video platforms (Twelve Labs, Voxel51 FiftyOne, NomadicML) | Natural-language search without building the pipeline | Your control over models and where the data lives | Vendor pricing |
The approaches stack. Telemetry decides which hours to look at, embeddings rank what is left, and captions give you text to filter and to show a reviewer why a segment matched.
How do I build the pipeline?
1. Keep footage and sensor data on one clock. Robot logs in MCAP or ROS bag files already timestamp every topic. For dashcams, store each file's start time and the GPS and speed track beside it. Every later step joins on time. 2. Drop what cannot hold your event. Filter by route, date, vehicle, speed range or robot state. Then skip segments with no visual change. The open-source SentrySearch skips still chunks by default, and its README notes that parked Sentry Mode recordings benefit most. 3. Cut what is left into fixed segments. Dashcam and robot video has no editorial cuts, so shot-change detection finds few boundaries. Fixed 10 to 30 second windows, with a few seconds of overlap, keep an event from being split in half. Our scene segmentation guide compares the methods. 4. Embed each segment and write a caption. The embedding answers "what looks like this". The caption answers "where does a red truck appear" and gives you words to filter and display. 5. Store each segment with its metadata. Start time, end time, vehicle or robot ID, location and any event flags. Search then filters on these fields before ranking, which is what makes it fast on a large archive. 6. Check the top results for rare events. Similarity search always returns a top result, even when the event never happened. For something rare, such as a near miss, ask a vision-language model a yes-or-no question about each of the top 20 or 50 segments before you count them.
How big is the index for 10,000 hours of footage?
Small next to the footage, as long as you do not embed every frame. Worked out from vector sizes (float32, 4 bytes per dimension) against 36 TB of 1080p video:
| Sampling | Vectors | Index size | Share of footage |
|---|---|---|---|
| Every frame at 30 fps, 512 dimensions | 1.08 billion | 2.2 TB | 6.1% |
| 1 frame a second, 512 dimensions | 36 million | 74 GB | 0.2% |
| 10 second segments, 1,408 dimensions | 3.6 million | 20 GB | 0.06% |
| 30 second segments, 3,072 dimensions | 1.2 million | 15 GB | 0.04% |
We worked these out from published vector sizes and bitrates on 2026-10-26. Segments also cut the number of results to review: a 30 second hit is one row, where per-frame search returns the same moment 30 times over.
What does it cost to index 10,000 hours?
At list prices on 2026-10-26:
Where Mixpeek fits
Mixpeek reads your files where they already live, in S3, GCS or any bucket, and turns what's in them into data your software can search, classify and moderate. Point a collection at the bucket that holds your recordings and turn on
video_search:import os
from mixpeek import Mixpeek
client = Mixpeek(
api_key=os.environ["MIXPEEK_API_KEY"],
namespace=os.environ["MIXPEEK_NAMESPACE"],
)
collection = client.collections.create(
collection_name="fleet-footage",
source={"type": "bucket", "bucket_ids": [os.environ["BUCKET_ID"]]},
features=["video_search", "onscreen_text"],
)Each segment comes back as a document with
Who it fits: teams with footage already in object storage who want search, filtering and classification without running embedding jobs and a vector database themselves. Who it does not: one person searching their own dashcam folder on a laptop, where SentrySearch or a local model is the cheaper choice.
Filter the recordings by time, location and speed from the GPS track first, then run a natural-language search over embeddings of short segments, for example "truck running a red light at night". The search returns segments with start and end times, so you jump straight to the moment. Open-source SentrySearch does this on a laptop with Gemini Embedding 2 or a local Qwen3-VL model; a hosted index such as Mixpeek does it across a whole fleet's bucket.
No. Every frame at 30 fps over 10,000 hours is about 1.08 billion vectors and 2.2 TB of index. Thirty second segments are 1.2 million vectors and about 15 GB, and a segment embedding from a video model captures motion that single frames miss. Use 1 frame a second only when you need image-level matches, such as finding one license plate or one product.
About $30,000 to $34,000 at list prices on 2026-10-26 if you index every hour: $3.41 an hour with SentrySearch on Gemini Embedding 2 and $3 an hour on Mixpeek video search. Filtering by telemetry and skipping idle footage first is the biggest saving. Indexing a quarter of the hours costs a quarter as much.
Yes, once the camera streams are video files. Export each camera topic to video, keep the log's timestamps, and store the other topics (joint states, faults, task IDs) as metadata on the same time range. A search can then ask for "gripper dropping a box" only during pick tasks, or only on one robot.
Search ranks by similarity, so it returns a best match even when no near miss happened. Narrow with telemetry first (hard braking, sudden steering, a robot's safety stop), search the windows that remain, and then ask a vision-language model a yes-or-no question about each top result. Count only the segments it confirms, and spot-check a sample by eye.
SentrySearch is Apache-2.0 licensed with about 4,500 GitHub stars. It splits footage into overlapping chunks, embeds them with Gemini Embedding 2, Alibaba's Qwen3-VL embedding API or a local Qwen3-VL model, stores the vectors in ChromaDB, and trims the best match into a clip. It runs on one machine, which suits a personal archive more than a fleet.
How to find one specific moment in hours of footage
Why does my video search keep returning the wrong scene?
Why does AI miss things in a long video?
How to stop duplicate videos showing up in search results
How many tokens is an hour of video?
Best video search tools
start_time and end_time, and the metadata you
attach to each recording (vehicle ID, route, recording time, event flags) stays at the field path
you gave it, so a retriever can filter on it before
ranking. video_search uses fixed 30 second segments, up to 50 per file (25 minutes), and reports
covered_s when a file was longer. For long recordings or 10 second windows, use the
multimodal extractor, which splits by
time, scene or silence.Who it fits: teams with footage already in object storage who want search, filtering and classification without running embedding jobs and a vector database themselves. Who it does not: one person searching their own dashcam folder on a laptop, where SentrySearch or a local model is the cheaper choice.
Frequently Asked Questions
How do I search dashcam footage for a specific event?
Filter the recordings by time, location and speed from the GPS track first, then run a natural-language search over embeddings of short segments, for example "truck running a red light at night". The search returns segments with start and end times, so you jump straight to the moment. Open-source SentrySearch does this on a laptop with Gemini Embedding 2 or a local Qwen3-VL model; a hosted index such as Mixpeek does it across a whole fleet's bucket.
Should I embed every frame of my footage?
No. Every frame at 30 fps over 10,000 hours is about 1.08 billion vectors and 2.2 TB of index. Thirty second segments are 1.2 million vectors and about 15 GB, and a segment embedding from a video model captures motion that single frames miss. Use 1 frame a second only when you need image-level matches, such as finding one license plate or one product.
How much does it cost to index 10,000 hours of dashcam video?
About $30,000 to $34,000 at list prices on 2026-10-26 if you index every hour: $3.41 an hour with SentrySearch on Gemini Embedding 2 and $3 an hour on Mixpeek video search. Filtering by telemetry and skipping idle footage first is the biggest saving. Indexing a quarter of the hours costs a quarter as much.
Can I search robot recordings stored as ROS bag or MCAP files?
Yes, once the camera streams are video files. Export each camera topic to video, keep the log's timestamps, and store the other topics (joint states, faults, task IDs) as metadata on the same time range. A search can then ask for "gripper dropping a box" only during pick tasks, or only on one robot.
How do I find rare events like near misses?
Search ranks by similarity, so it returns a best match even when no near miss happened. Narrow with telemetry first (hard braking, sudden steering, a robot's safety stop), search the windows that remain, and then ask a vision-language model a yes-or-no question about each top result. Count only the segments it confirms, and spot-check a sample by eye.
Is there an open-source tool for semantic dashcam search?
SentrySearch is Apache-2.0 licensed with about 4,500 GitHub stars. It splits footage into overlapping chunks, embeds them with Gemini Embedding 2, Alibaba's Qwen3-VL embedding API or a local Qwen3-VL model, stores the vectors in ChromaDB, and trims the best match into a clip. It runs on one machine, which suits a personal archive more than a fleet.