How do I find the images my computer vision model gets wrong?
Store every prediction next to the image and the label you trust, then ask three questions of that table. Which images did the model get wrong with high confidence? What else in the dataset looks like each of those failures? Which of those, deduplicated and balanced across labels, should go into the next training or evaluation set? The first is a filter and a sort, the second is an embedding similarity search, and the third is a diversity step followed by a stratified sample.
A confident miss is the most useful single signal. Either the label is wrong, which is common (an average of at least 3.3% of test labels across ten popular benchmarks are errors, per Northcutt, Athalye and Mueller), or the model has a systematic gap, a slice of data where it fails the same way.
What data do I need before I can find failures?
One row per image with:
low_light or overcast)Correctness comes from your own evaluation code. A search system can store and filter it, and it should keep these fields at the paths you gave them so your evaluation tools can still read them.
How do I rank the worst mistakes first?
Filter to
correct = false and sort by confidence, highest first. The top of that list is where the model was most sure and most wrong, which is where label errors and real blind spots concentrate. Low-confidence misses near the decision boundary are a different problem, usually fixed by more data of the same kind.Review the top of the list by hand. Each confident miss is one of:
1. A label error. Fix the label. The model was right. 2. A systematic gap. The model fails on a condition (night, rain, an unusual angle). Find more images with that condition. 3. An ambiguous case. Write the labeling rule down so the next annotator decides it the same way.
Tools that score label quality automatically, such as cleanlab, apply the confident-learning method from the same research group to rank likely label errors.
How do I find more images like a failure?
Embed every image once with a model that places images and text in one space, then search. Two kinds of query work:
This is the idea behind automated slice discovery. Domino (Eyuboglu et al., ICLR 2022) uses cross-modal embeddings and an error-aware mixture model to find coherent groups where a model fails, and describes each group in words; it identified 36% of the 1,235 slices in the paper's evaluation framework. A search over the same embeddings is the manual version: you supply the description and get the slice back.
For choosing the embedding model, see the best multimodal embedding models.
How do I turn the failures into a training or evaluation set?
A raw search result over-represents whatever is most common and repeats near-identical frames. Two steps fix that:
1. Drop near-duplicates with maximal marginal relevance (MMR) on the image embedding, which trades similarity to the query against similarity to what is already picked. 2. Balance across labels with a stratified sample, a fixed count with a minimum per label, so an evaluation set does not end up all one class.
Record which images were picked and why, then export that selection to your labeling or training tools as a versioned release. Selection and labeling are separate steps, and keeping the release pinned lets you compare the next model on the same set.
What does this look like on a real dataset?
We ran the steps above on Mixpeek production on 2026-10-29 with 40 CC0 street photos (20 day, 20 night) and an illustrative day-or-night scene model that gets 16 of them wrong. The data and every step are in the AI data curation template.
| Step | What it does | Result |
|---|---|---|
| Index | Embed 40 images, keep label, prediction, confidence, correct and error category as fields | 40 documents after 247.5 s |
| Rank misses | correct = false, sorted by confidence | 16 misses, confidence 0.84 down to 0.52, in 0.7 s |
| Describe a failure | Text search: "a dark city street at night" | top 5 all night images, all low_light misses; 23.9 s on the first, cold call |
| Curate | Search, MMR, stratified sample by label | 12 images, 6 day and 6 night, in 1.4 s |
| Cost | Whole run | 880 credits (0.88 USD), nearly all of it the 40 image embeddings |
The most confident miss was a rainy daytime street the model called night at 0.84 confidence, tagged
overcast. The ranking put it first, and it points at a condition the training data lacks: rain in daylight.How does Mixpeek fit?
Mixpeek reads your files where they already live, in S3, GCS or any bucket, and turns what's in them into data your software can search, classify and moderate. For failure discovery that means one collection holds the images with your prediction fields kept at the paths you gave them, and four retrievers answer the questions above: failure-discovery from text, similar-to-failure from an image, hard-examples with an attribute filter and a sort on confidence, and curate with MMR and a stratified sample. An optional alert emails you when a newly indexed image is a confident miss.
Who it is for: teams with a model in production or evaluation and more images than anyone can review by hand. Who it is not for: teams that need labeling or training done for them. Mixpeek finds and selects; labeling, training and scoring stay in your own tools. Pricing is per credit, on the pricing page.
Frequently Asked Questions
How do I find mislabeled images in my dataset?
Sort the images your model got wrong by confidence, highest first, and review the top. A model that is very confident and disagrees with the label is often right. Across ten widely used test sets, at least 3.3% of labels are errors on average, and at least 6% of the ImageNet validation set (Northcutt et al.). cleanlab automates the ranking with confident learning.
What is a confident miss?
A prediction that is wrong and was made with high confidence, for example a day photo called night at 0.84. Confident misses are where label errors and systematic gaps concentrate, which makes them the first images to review.
What is slice discovery?
Finding coherent subsets of data where a model fails, such as night scenes or one camera angle, without listing them in advance. Methods like Domino cluster model errors in a cross-modal embedding space and describe each cluster in words.
Do I need labels for every image?
No. Ranking misses needs labels, but similarity search works on unlabeled images too. A text or image query over the whole dataset returns the unlabeled images that look like a known failure, which is how you choose what to label next.
How many images should go into a new evaluation set?
Enough per label that one image does not swing the metric, with the same minimum for every label. The curate step above uses a stratified sample with a fixed total and a per-label minimum (12 images, at least 4 per label, in the measured run).
How much does it cost to index images for failure search?
Embedding dominates. In the run above, indexing and querying 40 images cost 0.88 USD, nearly all of it the embeddings. Cost grows with the number of images indexed, and each query after that is small.