How do I find edge cases in my video data?
Split your footage into scenes and embed each one, so every scene becomes a point you can search and group. Then look in three places: scenes that match conditions you already suspect (rain at night, a person partly hidden), scenes that look like clips your model already got wrong, and scenes that sit apart from every cluster. Remove near-duplicates, label what is left, and keep part of it as a separate edge-case test set so you can see whether the next model actually handles them.
An edge case is a situation your training data rarely shows: unusual weather or light, a rare object, an odd camera angle, heavy occlusion, motion blur, a sensor glitch. A model can score well on average and still fail on all of them, because the average is made of common scenes.
Why random sampling misses edge cases
If a condition appears in 0.1% of your scenes, a random sample of 1,000 scenes holds about one of them, and often none. Labeling more random frames mostly labels more of what you already have. Video makes this worse, because consecutive frames are nearly identical: a sample of frames is really a much smaller sample of distinct moments.
Finding rare scenes needs a way to go to them directly, which is what embeddings give you. Once every scene is a vector, "show me scenes like this" and "show me scenes unlike anything else" are both queries.
Five ways to find edge cases, compared
| Method | What it finds | What it needs | What it misses |
|---|---|---|---|
| Search by description | Conditions you can name: "pedestrian crossing at night in rain" | An image-text embedding model such as CLIP or SigLIP 2 | Anything you did not think to name |
| Search by example | More scenes like a clip the model failed on | One failure clip and the same embeddings | Failures that look nothing like the ones you have |
| Clustering and outliers | Scenes that fit no group, and very small groups | Embeddings for every scene and a density-based clusterer | Rare scenes that look like common ones |
| Model uncertainty and mistakes | Scenes the model is unsure about or gets wrong | Your model's predictions, and labels for some data | Scenes the model is confidently wrong about |
| Metadata filters | Known slices: camera, location, time of day, weather tag | Metadata recorded with the footage | Anything the metadata does not record |
The methods cover each other's gaps, so most teams run several and merge the results.
How to find edge cases, step by step
1. Split video into scenes. Cut at shot changes, or into windows of a few seconds for continuous footage such as dashcam or CCTV. Keep each scene's video id, start time and end time. 2. Embed every scene. Use an image-text model on a few sampled frames per scene, so the same vectors answer text queries and image queries. 3. Search for what you suspect. Write down the conditions that worry you and query for each one in plain words. Review the top results; many will be real examples you did not know you had. 4. Search from your failures. Take clips where the model was wrong in testing or in production and find their nearest neighbours. Failures tend to come in families. 5. Cluster to find what you did not think of. Run a density-based clusterer such as HDBSCAN over the scene embeddings. It leaves points that belong to no dense region unassigned and labels them as noise (how HDBSCAN works). Those unassigned scenes, and clusters with only a handful of members, are your candidate edge cases. Look at them; some will be junk footage, and that is worth knowing too. 6. Add model signals. Where you have predictions, sort by low confidence, by disagreement between two models, or by prediction-label mismatch. Tools such as cleanlab find likely label errors, which often sit on edge cases. 7. Remove near-duplicates. Keep one scene per tight cluster so ten seconds of the same intersection does not take ten labeling slots. 8. Label, split and measure. Add most of what you found to training data, and keep the rest as an edge-case test set that stays fixed. Report the model's score on that set separately from the overall score.
Common mistakes
Tools for finding edge cases in video
Doing it with Mixpeek
Mixpeek is a multimodal retrieval platform: it indexes video from your bucket into scene-level documents with embeddings and timestamps, and a retriever runs a chain of stages over them. The multimodal extractor splits and embeds each video. A retriever for edge-case review can then:
output_mode: "labeled", every scene comes back with a cluster_id, and scenes that fit no cluster get cluster_id: -1. Those are the outliers to review. output_mode: "representatives" returns one scene per cluster, which doubles as near-duplicate removal.A minimal stage chain looks like this:
[
{
"stage_name": "feature_search",
"stage_type": "filter",
"config": {
"stage_id": "feature_search",
"parameters": {
"searches": [
{
"feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": {"input_mode": "text", "value": "{{INPUT.query}}"},
"top_k": 500
}
],
"final_top_k": 500
}
}
},
{
"stage_name": "cluster",
"stage_type": "reduce",
"config": {
"stage_id": "cluster",
"parameters": {"algorithm": "hdbscan", "min_cluster_size": 5, "output_mode": "labeled"}
}
}
]Every result keeps its document id, source video and start and end time, so the scenes you select can go straight to a labeling tool or a training manifest. See retrievers for how to create and run one, and pricing for what indexing costs.
Split the footage into scenes, embed each scene with an image-text model, and look in three places: scenes that match conditions you suspect, scenes similar to clips your model got wrong, and scenes a density-based clusterer such as HDBSCAN leaves out of every cluster. Remove near-duplicates, label what is left, and keep part of it as a separate edge-case test set.
A situation the training data rarely shows, such as unusual weather or lighting, a rare object, an odd camera angle, heavy occlusion, motion blur or a sensor fault. Models tend to fail there because their accuracy is learned mostly from common scenes.
Because rare scenes are rare in the sample too. A condition in 0.1% of scenes shows up about once in a random sample of 1,000, and often not at all. In video, adjacent frames repeat each other, so a frame sample holds even fewer distinct moments than its size suggests.
Embed every scene and run a density-based clustering algorithm such as HDBSCAN over the vectors. Points that fall in no dense region are labeled as noise, and those, along with very small clusters, are the outliers. Review them by hand: some are real edge cases and some are corrupted or irrelevant footage.
There is no fixed number. Keep adding examples of a condition until the model's score on a held-out set of that condition stops improving, and keep that held-out set fixed so the scores are comparable across model versions.
Yes. Take a clip the model got wrong, embed it, and search for its nearest neighbours across the library. Failures tend to come in families, so one example usually leads to more of the same condition.
Frequently Asked Questions
How do I find edge cases in my video data?
Split the footage into scenes, embed each scene with an image-text model, and look in three places: scenes that match conditions you suspect, scenes similar to clips your model got wrong, and scenes a density-based clusterer such as HDBSCAN leaves out of every cluster. Remove near-duplicates, label what is left, and keep part of it as a separate edge-case test set.
What is an edge case in computer vision?
A situation the training data rarely shows, such as unusual weather or lighting, a rare object, an odd camera angle, heavy occlusion, motion blur or a sensor fault. Models tend to fail there because their accuracy is learned mostly from common scenes.
Why doesn't random sampling find edge cases?
Because rare scenes are rare in the sample too. A condition in 0.1% of scenes shows up about once in a random sample of 1,000, and often not at all. In video, adjacent frames repeat each other, so a frame sample holds even fewer distinct moments than its size suggests.
How do I find outliers in a video dataset?
Embed every scene and run a density-based clustering algorithm such as HDBSCAN over the vectors. Points that fall in no dense region are labeled as noise, and those, along with very small clusters, are the outliers. Review them by hand: some are real edge cases and some are corrupted or irrelevant footage.
How many edge-case examples do I need?
There is no fixed number. Keep adding examples of a condition until the model's score on a held-out set of that condition stops improving, and keep that held-out set fixed so the scores are comparable across model versions.
Can I search video for scenes my model fails on?
Yes. Take a clip the model got wrong, embed it, and search for its nearest neighbours across the library. Failures tend to come in families, so one example usually leads to more of the same condition.