1.  [](/)
2.  [Templates](/templates)
3.  AI data curation

Model data

v0.1.0 · updated 2026-10-29

# AI data curation

Find the data your model is missing. Turn model failures into training and evaluation datasets with multimodal search and sampling.

[Deploy this namespace](https://studio.mixpeek.com/manifests/apply?url=https%3A%2F%2Fmixpeek.com%2Ftemplates%2Fai-data-curation.namespace.yaml&manifest_url=https%3A%2F%2Fmixpeek.com%2Ftemplates%2Fai-data-curation.namespace.yaml&src=template-ai-data-curation&template=ai-data-curation&template_name=AI+data+curation&sync=1&utm_source=homepage&utm_medium=template_deploy&utm_campaign=mvs_onramp)[Book a workshop](/pilot?src=template-ai-data-curation)View manifest

What deploys today10 ready1 need your input

Applies empty and reads Your own samples with your model's predictions. Connect your own data before querying.

![A London street in daylight seen through a rain-covered window, with a red bus and cars with their brake lights on](/templates/ai-data-curation/rain.webp)overcast sky, read as nightbrake lights in daylightreview

samples -> hard-examples · The model called this rainy London street night, at 0.84 confidence. It is daytime, and hard-examples ranks it first because it is the most confident miss in the set.Still: Wikimedia Commons (CC0) / Anjana Menon

samples indexed after apply

40 in 248 s

night scenes in failure-discovery's top 5

5 of 5

misses ranked by confidence, 0.84 down to 0.52

16

day and night in the 12-sample curated pick

6 / 6

click to first answer

68.1 s

How these were measured

*   40 in 248 s samples indexed after apply: Proof run on production, 2026-10-29: apply created the 7 resources and the seed batch in 27.5 s, and the samples collection reached 40 documents at 247.5 s.
*   5 of 5 night scenes in failure-discovery's top 5: "a dark city street at night" returned only night samples, all of them low\_light misses. 23.9 s on the first, cold call.
*   16 misses ranked by confidence, 0.84 down to 0.52: hard-examples returned every wrong prediction in the seed set, most confident first, in 0.7 s. The rainy street the model called night ranks first.
*   6 / 6 day and night in the 12-sample curated pick: curate with "a city street" returned 12 results balanced across both labels, in 1.4 s. The whole proof run cost 880 credits (0.88 USD).
*   68.1 s click to first answer: from clicking apply to the first search that answers, timed on production on 7 November 2026 by the harness that runs nightly

## What’s inside

Extractors

[multimodal](/extractors/multimodal)

Models

[Vertex AI multimodalembedding@001](/models/multimodalembedding)

Retrievers

failure-discovery·[feature\_search](/retrievers/feature_search)similar-to-failure·[feature\_search](/retrievers/feature_search)hard-examples·[verdict\_filter](/retrievers/verdict_filter) → [sort\_attribute](/retrievers/sort_attribute)curate·[feature\_search](/retrievers/feature_search) → [mmr](/retrievers/mmr) → [sample](/retrievers/sample)confident-miss-monitor·[verdict\_filter](/retrievers/verdict_filter)

## See it work

*   a dark city street at night
    
    failure-discovery
    
    on the seed set, the top 5 are all night scenes the model called day
    
*   every sample the model got wrong
    
    hard-examples
    
    filtered on correct = false, sorted by confidence, highest first
    
*   a city street
    
    curate
    
    12 kept, near-duplicates dropped, at least 4 per label
    

Simulated walkthrough · illustrative frames, scripted decisions

Pause

![A London street in daylight seen through a rain-covered window, with a red bus and cars with their brake lights on](/templates/ai-data-curation/rain.webp)overcast sky, read as nightbrake lights in daylightreview

samples -> hard-examplesStill: Wikimedia Commons (CC0) / Anjana Menonframe 1 of 1

Detections on this frame

*   overcast sky, read as night
*   brake lights in daylight

Decision path

1.  samples -> hard-examples
2.  overcast sky, read as night
3.  brake lights in daylight
4.  route: review

label day, predicted night at 0.84: the most confident miss

Your call on this frame

Clear itExclude it

Running tally

0

cleared

1

review

0

excluded

What a person's call does here

The selection itself, recorded on each document as curation\_versions with a reason, is what a release exports.

Try a search

“a dark city street at night”“every sample the model got wrong”“a city street”

Pick a search to see which retriever answers it and whether it works on a one-click deploy.

InsamplesOutsamplesa dataset release

## How it's wired

Applies 1 buckets, 1 collections and 5 retrievers today

SourceBucketCollectionRetrieverAlertNot applied yet

.wf-flow { stroke-dasharray: 6 6; animation: wf-dash 1.2s linear infinite; } @keyframes wf-dash { to { stroke-dashoffset: -12; } } @media (prefers-reduced-motion: reduce) { .wf-flow { animation: none; } } SOURCENeeds your inputsampless3 · continuousBUCKETsamplesimage: image · dedupCOLLECTIONDeploys readysamplesmultimodal\_extractor@v1RETRIEVERDeploys readyfailure-discoveryrrf · top 50RETRIEVERDeploys readysimilar-to-failurerrf · top 50RETRIEVERDeploys readyhard-examplesRETRIEVERDeploys readycuraterrf · top 200RETRIEVERDeploys readyconfident-miss-monitorALERTconfident-miss-indexed

Drag nodes to rearrange. Drag the background to pan, ctrl or cmd plus scroll to zoom. Click a node for its settings.

## What it costs

Files per monthSearches per month

Reset to defaults

Usage cost per month

$18.07

<$0.01 per file · rate card when this page was built

Processing is priced the way a batch is charged: each bucket's minutes, images, pages or tokens times its rate card lines. The batch dry run in Studio gives the binding quote.

## Models

*   [Vertex AI multimodalembedding@001](/models/multimodalembedding)Provider APIEmbedding

### Collections and their documents

samplesfrom bucket samples, by multimodal\_extractor@v1

Reads image from image

Field

Type

Required

document\_id

string

set by Mixpeek

collection\_id

string

set by Mixpeek

label

string

no

prediction

string

no

confidence

number

no

correct

boolean

no

error\_category

string

no

model\_version

string

no

*   `mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding` from [Vertex AI multimodalembedding@001](/models/multimodalembedding)

## Retrievers

*   failure-discovery
    
    feature\_search over samples; input query
    
*   similar-to-failure
    
    feature\_search over samples; input query
    
*   hard-examples
    
    verdict\_filter then sort\_attribute over samples; input limit
    
*   curate
    
    feature\_search then mmr then sample over samples; input query
    
*   confident-miss-monitor
    
    verdict\_filter over samples; input document\_id
    

## Spend limits

Daily spend budget in the manifest

2 USD a day; new batches pause once it is spent

Acceptance run ceiling

1.25 USD

Model cost

Unmeasured for 1 of 1 models

Monthly cost at your volume is in the [calculator](#pricing). Served models start at zero replicas, so an idle namespace runs no model.

## Reward signals

The loop runs through your tools: you send samples with predictions, Mixpeek selects, your labeling tool labels the selection, your training job uses it, and the next model's predictions come back as new samples.

How reviewer decisions move the thresholds

Docs[Export a collection](https://mixpeek.com/docs/api-reference/collections/export-collection)

Explicit signals

The selection itself, recorded on each document as curation\_versions with a reason, is what a release exports.

Implicit signals

None from the template. Mixpeek records retriever executions server-side, which is telemetry about queries rather than feedback about results.

Where they land

`samples``mxp_retriever_executions`

How the loop closes

Each model version's predictions arrive with model\_version on them, so the next round of failure-discovery runs against what the newest model still gets wrong.

OverviewManifestWhat deploysComponentsDeploymentEvaluationQuestions

Upload your samples with your model's predictions and the ground truth you have. Mixpeek finds more examples like the failures, ranks the confident misses, drops near-duplicates and balances the pick across labels, so the selection is ready for your labeling and training tools. Mixpeek finds and selects; labeling, training and scoring stay in your own tools.

*   failure-discovery: describe a failure in words and get every sample that looks like it, labeled or not.
*   similar-to-failure: pass the URL of an image the model got wrong and get the samples closest to it.
*   hard-examples: every sample the model got wrong, most confident first, so a reviewer sees the confident misses before the ambiguous ones.
*   curate: search wide, drop near-duplicates on the image vector, then sample evenly across labels: 12 kept, at least 4 per label.
*   Mixpeek finds and selects. Labeling, training and model evaluation stay in your own tools, and Mixpeek does not score predictions: correctness and error categories arrive with your data.

For: AI labs, robotics and autonomous-vehicle teams, and data and evaluation vendors who know where their model fails and need more examples like the failures, picked without near-duplicates.

What goes in

samples

One image per sample, with your own label, prediction, confidence, correct, error\_category and model\_version fields. Only the image is required; every field you send passes through at the path you gave it.

What comes out

samples

One document per sample with a multimodal embedding of the image and your prediction fields carried through, so the same index answers a sentence, a photo of a failure, and a filter on correct = false.

a dataset release

Documented, not deployed: record the selection on each document (curation\_versions), then export the collection filtered on that version as Parquet or WebDataset, with lineage and a release manifest giving every file's size and SHA-256, into your bucket.

Docs[Manifest schema](https://mixpeek.com/docs/operations/manifests)

One file spins up the namespace. Generated from the workflow above. Also served at `/templates/ai-data-curation.namespace.yaml`.

\# ai-data-curation: one manifest spins up the namespace.
# Platform manifest schema (GET /v1/discovery/schema). Validate with POST /v1/manifest/validate,
# apply with POST /v1/manifest/apply or the Deploy button. Wiring comes from the flow diagram:
# edges are bucket -> collection sources, collection -> retriever scope, retriever -> view.
# Applying a SECOND time, over a namespace this template already created: use
# POST /v1/manifest/apply?mode=create\_missing, which creates what is missing and leaves
# what exists alone. The default, create\_only, fails the WHOLE apply and rolls it back if
# any resource already exists, so an upgrade looks like a dead end without this. Use
# mode=upsert to also patch resources that exist but have drifted from this file.
version: '1.0'
metadata:
  name: ai-data-curation
  description: "Namespace template ai-data-curation. Generated from the flow diagram on mixpeek.com/templates/ai-data-curation."
namespaces:
  - name: ai-data-curation
    description: "Everything below lives in this namespace."
    feature\_extractors:
      - name: multimodal\_extractor
        version: v1
    # Daily spend budget, in dollars of what you are charged (1,000 credits a dollar).
    # New batches pause once the namespace has spent this much today; queries are not capped.
    # Raise it later on the namespace: PATCH /v1/namespaces/<id> {"spend\_budget": {"daily\_usd": N}}.
    budget:
      daily\_usd: 2

# Data sources. A storage connection carries credentials, so it is created in Studio
# (or POST /v1/organizations/storage-connections) and synced into the bucket named here.
#   samples: s3, continuous, your samples with your model's predictions, in your account -> bucket samples
buckets:
  - name: samples
    namespace: ai-data-curation
    description: "Meant to be fed by samples (s3, continuous). No source is connected yet: applying this manifest creates the bucket only. Connect the source to this bucket in Studio (Syncs) to start the feed."
    schema:
      properties:
        image:
          type: image
          required: true
        label:
          type: string
        prediction:
          type: string
        confidence:
          type: number
        correct:
          type: boolean
        error\_category:
          type: string
        model\_version:
          type: string

# Connect your own storage. Create the two secrets, uncomment, and apply again;
# apply live-tests the connection, so it must have real credentials to succeed.
# Until then the buckets above accept direct uploads.
# storage\_connections:
#   - name: samples-connection
#     provider: s3
#     description: "Read-only access to the s3 location holding your samples with your model's predictions, in your account."
#     config:
#       region: us-east-1
#       credentials:
#         type: access\_key
#         access\_key\_id: ${{ secrets.AWS\_ACCESS\_KEY\_ID }}
#         secret\_access\_key: ${{ secrets.AWS\_SECRET\_ACCESS\_KEY }}

# bucket\_syncs:
#   - name: samples-sync
#     bucket: samples
#     connection: samples-connection
#     source\_path: "samples/"
#     sync\_mode: continuous
#     polling\_interval\_seconds: 300
#     skip\_duplicates: true
#     file\_filters:
#       include\_patterns: \["\*.jpg", "\*.jpeg", "\*.png", "\*.webp"\]
#     schema\_mapping:
#       mappings:
#         image:
#           target\_type: blob
#           source: {type: file}
#           blob\_type: image

collections:
  - name: samples
    namespace: ai-data-curation
    description: "multimodal\_extractor@v1 over bucket samples. Feeds confident-miss-monitor, confident-miss-indexed, failure-discovery, similar-to-failure, hard-examples, curate."
    source:
      type: bucket
      bucket: samples
    feature\_extractor:
      name: multimodal\_extractor
      version: v1
      input\_mappings:
        image: image
      field\_passthrough:
        - source\_path: label
        - source\_path: prediction
        - source\_path: confidence
        - source\_path: correct
        - source\_path: error\_category
        - source\_path: model\_version
    alert\_applications:
      - alert: confident-miss-indexed
        input\_mappings:
          - input\_key: document\_id
            source: {source\_type: document\_field, path: \_internal.document\_id}
    enabled: true
retrievers:
  - name: failure-discovery
    namespace: ai-data-curation
    description: "Describe a failure in words and get back every sample that looks like it, labeled or not, so one known miss turns into the set of samples like it."
    collections:
      - samples
    input\_schema:
      query:
        type: text
        required: true
        description: "What to look for; searched across every index below"
    stages:
      - stage\_name: search
        stage\_id: feature\_search
        parameters:
          searches:
            - feature\_uri: "mixpeek://multimodal\_extractor@v1/vertex\_multimodal\_embedding"
              query:
                input\_mode: text
                value: "{{INPUT.query}}"
              top\_k: 50
          fusion: rrf
          final\_top\_k: 50
    tags:
      - template:ai-data-curation
  - name: similar-to-failure
    namespace: ai-data-curation
    description: "Pass the URL of an image the model got wrong and get back the samples closest to it."
    collections:
      - samples
    input\_schema:
      query:
        type: text
        required: true
        description: "URL of the image or frame to match; encoded by the index below and compared against it"
    stages:
      - stage\_name: search
        stage\_id: feature\_search
        parameters:
          searches:
            - feature\_uri: "mixpeek://multimodal\_extractor@v1/vertex\_multimodal\_embedding"
              query:
                input\_mode: content
                value: "{{INPUT.query}}"
              top\_k: 50
          fusion: rrf
          final\_top\_k: 50
    tags:
      - template:ai-data-curation
  - name: hard-examples
    namespace: ai-data-curation
    description: "Every sample the model got wrong, most confident first."
    collections:
      - samples
    input\_schema:
      limit:
        type: text
        required: false
        description: "Optional cap on documents grouped per run"
    stages:
      - stage\_name: verdict\_filter
        config:
          stage\_id: attribute\_filter
          parameters:
            conditions:
              AND:
                - field: correct
                  operator: eq
                  value: false
      - stage\_name: most\_confident\_first
        stage\_id: sort\_attribute
        parameters:
          field: "confidence"
          direction: "desc"
    tags:
      - template:ai-data-curation
  - name: curate
    namespace: ai-data-curation
    description: "Search for what a dataset should cover, then keep a diverse, balanced pick: near-duplicates dropped and every label represented."
    collections:
      - samples
    input\_schema:
      query:
        type: text
        required: true
        description: "What to look for; searched across every index below"
    stages:
      - stage\_name: search
        stage\_id: feature\_search
        parameters:
          searches:
            - feature\_uri: "mixpeek://multimodal\_extractor@v1/vertex\_multimodal\_embedding"
              query:
                input\_mode: text
                value: "{{INPUT.query}}"
              top\_k: 200
          fusion: rrf
          final\_top\_k: 200
      - stage\_name: drop\_near\_duplicates
        stage\_id: mmr
        parameters:
          lambda: 0.5
          top\_k: 100
          diversity\_feature\_uri: "mixpeek://multimodal\_extractor@v1/vertex\_multimodal\_embedding"
      - stage\_name: balance\_labels
        stage\_id: sample
        parameters:
          strategy: "stratified"
          stratify\_by: "label"
          count: 12
          min\_per\_stratum: 4
          seed: 7
    tags:
      - template:ai-data-curation
  - name: confident-miss-monitor
    namespace: ai-data-curation
    description: "The alert's check: is the sample just written one the model got wrong with confidence 0.8 or higher. The run is scoped to that one sample and reads its fields only, so it calls no model. Materialises into confident-miss-indexed."
    collections:
      - samples
    input\_schema:
      document\_id:
        type: text
        required: true
        description: "The written document this run is scoped to, supplied by the alert attachment"
    stages:
      - stage\_name: verdict\_filter
        config:
          stage\_id: attribute\_filter
          parameters:
            conditions:
              AND:
                - field: \_internal.document\_id
                  operator: eq
                  value: "{{INPUT.document\_id}}"
                - field: correct
                  operator: eq
                  value: false
                - field: confidence
                  operator: gte
                  value: 0.8
    tags:
      - template:ai-data-curation
alerts:
  - name: confident-miss-indexed
    namespace: ai-data-curation
    description: "Emails the org notifications address (Settings -> Notifications; unset means opted out) when a newly indexed sample is one the model got wrong with confidence 0.8 or higher."
    source: retriever
    retriever: confident-miss-monitor
    trigger\_on: results
    notification\_config:
      channels:
        - channel\_type: email
      include\_matches: true
      include\_scores: true
    enabled: false

# The curation sample set: 20 daytime and 20 night street photos (CC0, from
# Wikimedia Commons), each with the label it truly has and an illustrative
# prediction from a day-or-night scene model. The model gets 16 wrong, 14 of
# them at night, so the failures share a condition the retrievers can find.
# The blobs are the Wikimedia Commons copies of the same photos as
# samples/curation/, because the API serving mixpeek.com does not accept that
# sample set yet. Switch back to the sample keys once it does.
seed:
  - bucket: samples
    max\_objects: 40
    objects:
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/9/9d/Steam\_in\_new\_york\_street\_%28Unsplash%29.jpg/1920px-Steam\_in\_new\_york\_street\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.97, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/2/2b/Doorman\_on\_Leadenhall\_Street\_%28Unsplash%29.jpg/1920px-Doorman\_on\_Leadenhall\_Street\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.95, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/7/77/Muscle\_cars\_at\_Rundle\_Street\_%28Unsplash%29.jpg/1920px-Muscle\_cars\_at\_Rundle\_Street\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.93, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/d/dc/Driving\_outside\_Flinders\_Street\_Railway\_Station\_%28Unsplash%29.jpg/1920px-Driving\_outside\_Flinders\_Street\_Railway\_Station\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.91, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/2/2e/First\_Street\_%28Unsplash%29.jpg/1920px-First\_Street\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.89, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/d/d0/Neighborhood\_street\_%28Unsplash%29.jpg/1920px-Neighborhood\_street\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.97, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/d/d4/Street\_View\_urban\_street\_%28Unsplash%29.jpg/1920px-Street\_View\_urban\_street\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.95, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/9/9c/Street\_%28Unsplash%29.jpg/1920px-Street\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.93, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/3/30/Street\_Curve\_%28Unsplash%29.jpg/1920px-Street\_Curve\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.91, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/0/06/Bleak\_city\_street\_%28Unsplash%29.jpg/1920px-Bleak\_city\_street\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.89, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/9/93/Shadowed\_street\_London\_%28Unsplash%29.jpg/1920px-Shadowed\_street\_London\_%28Unsplash%29.jpg"}, label: day, prediction: night, confidence: 0.71, correct: false, error\_category: deep\_shadow, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/7/78/Street\_corner\_Japan\_%28Unsplash%29.jpg/1920px-Street\_corner\_Japan\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.95, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/1/17/Crossing\_The\_Street\_%28Unsplash%29.jpg/1920px-Crossing\_The\_Street\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.93, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/7/7f/People\_crossing\_street\_%28Unsplash%29.jpg/1920px-People\_crossing\_street\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.91, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/2/22/Steep\_city\_street\_%28Unsplash%29.jpg/1920px-Steep\_city\_street\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.89, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/7/79/London\_street\_in\_the\_rain\_%28Unsplash%29.jpg/1920px-London\_street\_in\_the\_rain\_%28Unsplash%29.jpg"}, label: day, prediction: night, confidence: 0.84, correct: false, error\_category: overcast, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/b/bc/Cobblestone\_street\_%28Unsplash%29.jpg/1920px-Cobblestone\_street\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.95, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/d/dd/Aerial\_view\_of\_Smolenskaya\_Street%2C\_Moscow\_%28Unsplash%29.jpg/1920px-Aerial\_view\_of\_Smolenskaya\_Street%2C\_Moscow\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.93, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/2/2f/New\_York\_Street\_Fashion\_%28Unsplash%29.jpg/1920px-New\_York\_Street\_Fashion\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.91, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/b/b0/Busy\_crossroads\_at\_daytime\_in\_Budapest\_%28Unsplash%29.jpg/1920px-Busy\_crossroads\_at\_daytime\_in\_Budapest\_%28Unsplash%29.jpg"}, label: day, prediction: day, confidence: 0.89, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/1/13/Toronto%2C\_Harbour\_Street\_at\_Night\_in\_2017\_-\_Sean\_Pollock\_2017\_%28Unsplash%29.jpg/1920px-Toronto%2C\_Harbour\_Street\_at\_Night\_in\_2017\_-\_Sean\_Pollock\_2017\_%28Unsplash%29.jpg"}, label: night, prediction: day, confidence: 0.52, correct: false, error\_category: low\_light, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/5/5e/Hong\_Kong\_night\_%28Unsplash%29.jpg/1920px-Hong\_Kong\_night\_%28Unsplash%29.jpg"}, label: night, prediction: night, confidence: 0.87, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/0/04/Cityscape\_and\_traffic\_light\_trails\_%28Unsplash%29.jpg/1920px-Cityscape\_and\_traffic\_light\_trails\_%28Unsplash%29.jpg"}, label: night, prediction: day, confidence: 0.56, correct: false, error\_category: low\_light, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/1/12/Flatiron\_at\_night\_%28Unsplash%29.jpg/1920px-Flatiron\_at\_night\_%28Unsplash%29.jpg"}, label: night, prediction: night, confidence: 0.81, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/a/ad/Speed\_by\_night\_%28Unsplash%29.jpg/1920px-Speed\_by\_night\_%28Unsplash%29.jpg"}, label: night, prediction: day, confidence: 0.6, correct: false, error\_category: low\_light, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/7/7f/Man\_on\_a\_bicycle\_at\_night\_%28Unsplash%29.jpg/1920px-Man\_on\_a\_bicycle\_at\_night\_%28Unsplash%29.jpg"}, label: night, prediction: day, confidence: 0.64, correct: false, error\_category: low\_light, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/2/2d/Inside\_a\_car\_at\_night-time\_in\_Williamsburg\_%28Unsplash%29.jpg/1920px-Inside\_a\_car\_at\_night-time\_in\_Williamsburg\_%28Unsplash%29.jpg"}, label: night, prediction: day, confidence: 0.68, correct: false, error\_category: low\_light, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/5/5e/Downtown\_Los\_Angeles\_at\_night\_%28Unsplash%29.jpg/1920px-Downtown\_Los\_Angeles\_at\_night\_%28Unsplash%29.jpg"}, label: night, prediction: night, confidence: 0.81, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/0/0c/Driving\_through\_a\_bridge\_at\_night\_%28Unsplash%29.jpg/1920px-Driving\_through\_a\_bridge\_at\_night\_%28Unsplash%29.jpg"}, label: night, prediction: day, confidence: 0.72, correct: false, error\_category: low\_light, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/5/55/Car\_light\_trails\_at\_night\_%28Unsplash%29.jpg/1920px-Car\_light\_trails\_at\_night\_%28Unsplash%29.jpg"}, label: night, prediction: day, confidence: 0.76, correct: false, error\_category: low\_light, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/0/0a/Dim\_urban\_street\_%28Unsplash%29.jpg/1920px-Dim\_urban\_street\_%28Unsplash%29.jpg"}, label: night, prediction: day, confidence: 0.52, correct: false, error\_category: low\_light, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/a/a2/Boston\_firetruck\_parked\_on\_a\_street\_at\_night\_%28Unsplash%29.jpg/1920px-Boston\_firetruck\_parked\_on\_a\_street\_at\_night\_%28Unsplash%29.jpg"}, label: night, prediction: day, confidence: 0.56, correct: false, error\_category: low\_light, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/2/29/Cobblestone\_street\_at\_night\_%28Unsplash%29.jpg/1920px-Cobblestone\_street\_at\_night\_%28Unsplash%29.jpg"}, label: night, prediction: day, confidence: 0.6, correct: false, error\_category: low\_light, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/9/97/Colorful\_urban\_traffic\_at\_night\_%28Unsplash%29.jpg/1920px-Colorful\_urban\_traffic\_at\_night\_%28Unsplash%29.jpg"}, label: night, prediction: night, confidence: 0.87, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/e/e8/New\_York\_city\_center\_street\_at\_night-time\_%28Unsplash%29.jpg/1920px-New\_York\_city\_center\_street\_at\_night-time\_%28Unsplash%29.jpg"}, label: night, prediction: night, confidence: 0.84, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/0/0d/On\_an\_empty\_road\_at\_night\_%28Unsplash%29.jpg/1920px-On\_an\_empty\_road\_at\_night\_%28Unsplash%29.jpg"}, label: night, prediction: day, confidence: 0.64, correct: false, error\_category: low\_light, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/9/9c/Highway\_intersection\_at\_night-time\_%28Unsplash%29.jpg/1920px-Highway\_intersection\_at\_night-time\_%28Unsplash%29.jpg"}, label: night, prediction: day, confidence: 0.68, correct: false, error\_category: low\_light, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/0/09/Jette\_highway\_traffic\_at\_night\_%28Unsplash%29.jpg/1920px-Jette\_highway\_traffic\_at\_night\_%28Unsplash%29.jpg"}, label: night, prediction: day, confidence: 0.72, correct: false, error\_category: low\_light, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/3/32/Neon\_streets\_%28Unsplash%29.jpg/1920px-Neon\_streets\_%28Unsplash%29.jpg"}, label: night, prediction: night, confidence: 0.84, correct: true, model\_version: scene-lighting-v1}
      - {blobs: {image: "https://thumb.wikimedia.org/wikipedia/commons/thumb/f/fe/Chicago\_nighttime\_%28Unsplash%29.jpg/1920px-Chicago\_nighttime\_%28Unsplash%29.jpg"}, label: night, prediction: day, confidence: 0.76, correct: false, error\_category: low\_light, model\_version: scene-lighting-v1}
# Measured on production 2026-10-29: the seed and these four queries cost
# 880 credits ($0.88), nearly all of it the 40 image embeddings.
acceptance:
  max\_cost\_usd: 1.25
  steps:
    - name: samples-indexed
      kind: documents
      collection: samples
      expect: {min\_documents: 40}
    - name: failure-discovery-finds-night
      kind: query
      retriever: failure-discovery
      inputs: {query: a dark city street at night}
      expect: {top\_field: label, top\_equals: night}
    - name: hard-examples-ranks-the-most-confident-miss
      kind: query
      retriever: hard-examples
      expect: {top\_field: error\_category, top\_equals: overcast}
    - name: curate-returns-a-stratified-pick
      kind: query
      retriever: curate
      inputs: {query: a city street}
      expect: {min\_results: 8}

Read from the manifest, so a part it ships commented out never shows as ready.

samplesCollectionDeploys ready

failure-discoveryRetrieverDeploys ready

similar-to-failureRetrieverDeploys ready

hard-examplesRetrieverDeploys ready

curateRetrieverDeploys ready

confident-miss-monitorRetrieverDeploys ready

attribute\_filterRetriever stageDeploys ready

sort\_attributeRetriever stageDeploys ready

mmrRetriever stageDeploys ready

sampleRetriever stageDeploys ready

samplesData sourceNeeds your input

What each part needs

samples

Deploys and processes documents as they arrive.

failure-discovery

Deploys and answers queries once its collections hold documents.

similar-to-failure

Deploys and answers queries once its collections hold documents.

hard-examples

Deploys and answers queries once its collections hold documents.

curate

Deploys and answers queries once its collections hold documents.

confident-miss-monitor

Deploys and answers queries once its collections hold documents.

attribute\_filter

The one-click manifest includes this stage.

sort\_attribute

The one-click manifest includes this stage.

mmr

The one-click manifest includes this stage.

sample

The one-click manifest includes this stage.

samples

Create the s3 credentials as manifest secrets, then uncomment the sync. It ships commented out because apply tests a connection as it creates it.

Docs[Multimodal extractor](https://mixpeek.com/docs/processing/extractors/multimodal)[MMR stage](https://mixpeek.com/docs/retrieval/stages/mmr)[Sample stage](https://mixpeek.com/docs/retrieval/stages/sample)

Component

Role

Source

Today

[multimodal\_extractor@v1](/extractors/multimodal)

Embeds each image into the space a text query and an image query shareRuns on CPU through the Vertex API, so the template does not wait on GPU capacity.

Built in

Deploys ready

[attribute\_filter](/retrievers/attribute_filter)

Keeps the samples your data marks as wrong (correct = false)

Built in

Deploys ready

[sort\_attribute](/retrievers/sort_attribute)

Ranks them by your model's confidence, highest first

Built in

Deploys ready

[mmr](/retrievers/mmr)

Drops near-duplicates by trading relevance for distance on the image vector

Built in

Deploys ready

[sample](/retrievers/sample)

Takes a stratified pick across labels so one condition cannot fill the set

Built in

Deploys ready

Docs[Customer-hosted overview](https://mixpeek.com/docs/customer-hosted/overview)

Mixpeek cloud

Apply the manifest and the namespace comes up with one collection and four retrievers, with a daily budget of 2 USD.

Your own cloud

The same manifest applies against a customer-hosted deployment when the samples cannot leave your environment.

Permissions

A key with namespace create and collection create. The storage connection is a separate grant and the manifest ships it commented out, because apply live-tests a connection and would roll the whole namespace back on a bad credential.

Footprint

One collection, one document per sample, with one multimodal vector each. No GPU is needed.

Docs[Retriever execution traces](https://mixpeek.com/docs/retrieval/retrievers)

Reference fixtures

The acceptance checks from the proof run: 40 samples indexed; failure-discovery ranks a night sample first for "a dark city street at night"; hard-examples ranks the overcast miss first; curate returns at least 8 for "a city street"; cost under 1.25 USD.

Category

Rule

Failure discovery

A described failure returns samples with that failure at the top. On the seed set the top 5 for night scenes were all night.

Hard examples

Every row is a wrong prediction and the order is by confidence, descending. Checkable without ground truth beyond what you sent.

Curated pick

Every label meets its minimum (4) and the pick holds no near-duplicates of each other.

Throughput

On the proof run, 40 images indexed in about 4 minutes after apply; queries answered in 0.7 to 1.4 s warm and 23.9 s on the first, cold call.

Does Mixpeek label the data or train the model?+

No. Mixpeek finds and selects. Labeling, training and model evaluation stay in your own tools, and the selection reaches them as an export they can pin.

Does it tell me which predictions are wrong?+

No. You send correct and error\_category with each sample, from your own evaluation. hard-examples filters and ranks on those fields.

Can it group failures into clusters?+

Not in this version. A cluster over the samples, writing a cluster label onto every document, is the next phase, and it needs only cluster and retriever configuration.

Does it work on video?+

This version is images only. The extractor already takes video and text as separate inputs, so video is a schema change rather than a new pipeline.

How is this different from the video moderation template?+

Video moderation screens footage against a policy and records why each segment was flagged. This template picks the examples a model needs to learn from.