Video Understanding

8 min read

Updated 2026-11-21

# How Do I Find the Robot Episodes Worth Labeling?

Search robot and egocentric episodes where they sit in S3 or GCS before labeling any of them: index short windows with the episode fields, search for failed grasps, handoffs and off-task stretches, confirm each match with a vision model, and drop near-duplicate demonstrations by comparing episode embeddings in DuckDB. The output is a list of episodes and time ranges for your labeling tool.

![Ethan Steininger](/people/ethan.webp)

By [Ethan Steininger](/blog/authors/ethan), Founder & CEO, Mixpeek

Robotics

Physical AI

Training Data

Data Curation

Egocentric Video

Video Search

DuckDB

[Search your episodes before you label them. Try it as you read.Index your episodes →](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide&utm_campaign=guide_cta&utm_content=robot_labeling)

[One-click templateAI data curationFind the data your model is missing. Turn model failures into training and evaluation datasets with multimodal search and sampling.Deploy it](/templates/ai-data-curation)

## How do I find the robot episodes worth labeling?

  
  
Search the episodes before you label any of them. Index each episode's camera video in short windows where it already sits in S3 or GCS, with the episode ID, task and robot on every window. Then run one search per kind of episode you want labeled (failed grasps, handoffs, off-task stretches), confirm each match by sending the clip to a vision model with a yes-or-no question, and drop near-duplicate demonstrations by comparing episode embeddings. What comes out is a short list of episodes and time ranges, and that list is what goes to your labeling tool.  
  

## Why select episodes before labeling them?

  
  
Most episodes in a demonstration archive are clean successes of the same task, recorded again and again. Labeling all of them pays the same per-episode cost for the hundredth clean pick as for the first failed one. The episodes that teach a policy something new are the rare ones: a grasp that slips, an object passed to a person, a stretch where the arm stops while the operator resets the scene.  
  
Those events are short. A failed grasp lasts about a second, and episodes are often under a minute: DROID, a public robot manipulation dataset, holds 76,000 trajectories and 350 hours of interaction ([DROID](https://droid-dataset.github.io/)), about 17 seconds per trajectory. Finding them by watching means watching everything, so the search has to work on what happens inside the video.  
  

## What should I search for?

  
  

Kind of episode

Search query

Confirm question for the vision model

Failed grasp

gripper closes but the object slips out

Keep only clips where the gripper closes on the object and the object falls or slips out

Handoff

a person's hand takes the object from the gripper

Keep only clips where the object passes from the robot to a person's hand

Off-task

the arm is still and a person rearranges the table

Keep only clips where the robot is not working on the task: the arm is idle, the object is out of view, or a person is resetting the scene or adjusting the camera

Near-duplicate demos

no query: compare episode embeddings

none: pick a similarity cutoff by eye

  
  
The first three are searches. The fourth compares episodes with each other, so it runs in SQL over an export of the embeddings.  
  

## How do I index the episodes?

  
  
Upload one video file per camera per episode as an object in a bucket, with the episode fields as metadata. Then create a collection that splits video by time. The [multimodal extractor](https://mixpeek.com/docs/processing/extractors/multimodal) writes `start_time` and `end_time` on every window, and `field_passthrough` copies your episode fields onto each one:

{
  "collection\_name": "robot-episodes",
  "source": { "type": "bucket", "bucket\_ids": \["bkt\_episodes"\] },
  "feature\_extractor": {
    "feature\_extractor\_name": "multimodal\_extractor",
    "version": "v1",
    "input\_mappings": { "video": "video\_url" },
    "field\_passthrough": \[
      { "source\_path": "metadata.episode\_id" },
      { "source\_path": "metadata.task" },
      { "source\_path": "metadata.robot\_id" },
      { "source\_path": "metadata.camera" }
    \],
    "parameters": {
      "split\_method": "time",
      "time\_split\_interval": 3,
      "run\_multimodal\_embedding": true,
      "run\_video\_description": true,
      "enable\_thumbnails": true
    }
  }
}

Three seconds keeps a one-second event in its own window. Index the camera that sees the gripper first; for many manipulation setups that is the wrist camera, and it is the cheapest way to start. [How do I find failed grasps and handoffs across thousands of robot episodes?](https://mixpeek.com/guides/how-to-find-failed-grasps-and-handoffs-in-robot-and-egocentric-video) covers window length, multiple cameras and egocentric video in more depth.  
  

## What retriever finds the episodes?

  
  
One [retriever](https://mixpeek.com/docs/retrieval/retrievers) serves every row of the table above. The query and the confirm question are inputs, so each call can ask for a different kind of episode:

{
  "retriever\_name": "episodes-to-label",
  "collection\_identifiers": \["robot-episodes"\],
  "input\_schema": {
    "query": { "type": "text", "required": true },
    "criteria": { "type": "text", "required": true },
    "task": { "type": "text", "required": true }
  },
  "stages": \[
    { "stage\_name": "search", "stage\_type": "filter",
      "config": { "stage\_id": "feature\_search", "parameters": {
        "final\_top\_k": 100,
        "searches": \[ { "feature\_uri": "mixpeek://multimodal\_extractor@v1/vertex\_multimodal\_embedding",
          "query": { "input\_mode": "text", "value": "{{INPUT.query}}" }, "top\_k": 100 } \] } } },
    { "stage\_name": "one\_task", "stage\_type": "filter",
      "config": { "stage\_id": "attribute\_filter", "parameters": {
        "field": "metadata.task", "operator": "eq", "value": "{{INPUT.task}}" } } },
    { "stage\_name": "confirm", "stage\_type": "filter",
      "config": { "stage\_id": "llm\_filter", "parameters": {
        "criteria": "{{INPUT.criteria}}",
        "include\_media": true,
        "include\_reasoning": true } } },
    { "stage\_name": "moments", "stage\_type": "reduce",
      "config": { "stage\_id": "moment\_group", "parameters": {
        "parent\_field": "source\_object\_id",
        "time\_field": "start\_time",
        "merge\_tolerance\_ms": 1000,
        "max\_moments\_per\_parent": 3,
        "score\_strategy": "max",
        "output\_mode": "annotated" } } }
  \]
}

What each stage is for:  
  
*   **Search** ranks the windows by how well they match the sentence. It always returns a best match,
even in a task where nothing failed, which is why the next two stages exist.*   **One task** keeps the confirm stage's input small. [LLM Filter](https://mixpeek.com/docs/retrieval/stages/llm-filter)
calls a model once per window that reaches it.*   **Confirm** sends each window to the model as video (`include_media: true`) with your question
and keeps the ones it answers yes to. `include_reasoning` returns the model's reason, which is useful to a labeler.*   **Moments** merges neighboring confirmed windows into one moment per episode file, with start and
end times. [Moment Group](https://mixpeek.com/docs/retrieval/stages/moment-group) needs `time_field: "start_time"` for a text query; with its default the moments come back empty.  
  

## How do I turn the results into a labeling list?

  
  
Run the retriever once per kind of episode and task, and write every moment to a CSV:

import csv, os, requests

API = "https://api.mixpeek.com"
HEADERS = {"Authorization": f"Bearer {os.environ\['MIXPEEK\_API\_KEY'\]}",
           "X-Namespace": os.environ\["MIXPEEK\_NAMESPACE"\]}
RETRIEVER = "episodes-to-label"  # or its ret\_ id

KINDS = {
    "failed\_grasp": ("gripper closes but the object slips out",
                     "Keep only clips where the gripper closes on the object and the object falls or slips out"),
    "handoff": ("a person's hand takes the object from the gripper",
                "Keep only clips where the object passes from the robot to a person's hand"),
    "off\_task": ("the arm is still and a person rearranges the table",
                 "Keep only clips where the robot is not working on the task: the arm is idle, "
                 "the object is out of view, or a person is resetting the scene or adjusting the camera"),
}
TASKS = \["pick\_cup", "open\_drawer"\]

with open("label\_queue.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(\["kind", "task", "episode\_id", "camera", "start\_s", "end\_s", "score", "reason"\])
    for task in TASKS:
        for kind, (query, criteria) in KINDS.items():
            r = requests.post(f"{API}/v1/retrievers/{RETRIEVER}/execute", headers=HEADERS, timeout=300,
                              json={"inputs": {"query": query, "criteria": criteria, "task": task}})
            r.raise\_for\_status()
            out = r.json()
            confirm = out.get("stage\_statistics", {}).get("stages", {}).get("confirm", {})
            reasons = {d\["document\_id"\]: d.get("reason", "")
                       for d in confirm.get("metadata", {}).get("llm\_decisions", \[\])}
            for doc in out\["documents"\]:
                meta = doc.get("metadata", {})
                for m in doc.get("moments", \[\]):
                    w.writerow(\[kind, task, meta.get("episode\_id"), meta.get("camera"),
                                m\["start\_ms"\] / 1000, m\["end\_ms"\] / 1000, round(m\["score"\], 4),
                                reasons.get(doc\["document\_id"\], "")\])

The results are in `documents`. The confirm stage's yes-or-no decisions, with the model's reason for each, are in `stage_statistics` under the stage's name. Print one response before you trust the column mapping, since where your episode fields land depends on how you uploaded them. Each row is an episode, a camera, and the seconds to look at. Send the rows to your labeling tool as its task list, with the time range so a labeler starts at the event.  
  

## How do I drop near-duplicate demonstrations?

  
  
Export the collection with its embeddings and compare episodes in DuckDB. The [export endpoint](https://mixpeek.com/docs/api-reference/collections/export-collection) writes Parquet to a storage connection you own (with write access turned on), and `include_vectors` adds a file of embeddings keyed by `document_id`:

POST /v1/collections/robot-episodes/export
{
  "format": "parquet",
  "include\_vectors": true,
  "destination": { "connection\_id": "conn\_training\_bucket", "prefix": "exports/2026-11-21/" }
}

The export writes a `manifest.json` under the prefix that names every file. Average each episode's window embeddings, then list pairs of episodes in the same task whose averages are almost identical:

INSTALL httpfs;
LOAD httpfs;
CREATE OR REPLACE SECRET s3 (TYPE s3, PROVIDER credential\_chain);

-- File names come from manifest.json under the export prefix.
CREATE VIEW segments AS
  SELECT \* FROM read\_parquet('s3://training-bucket/exports/2026-11-21/<documents file>.parquet');
CREATE VIEW vectors AS
  SELECT \* FROM read\_parquet('s3://training-bucket/exports/2026-11-21/<vectors file>.parquet');

WITH dims AS (
  SELECT s."metadata.episode\_id" AS episode, s."metadata.task" AS task,
         unnest(v.vector\_0) AS x, generate\_subscripts(v.vector\_0, 1) AS i
  FROM vectors v JOIN segments s USING (document\_id)
  WHERE s."metadata.camera" = 'wrist'
),
episodes AS (
  SELECT episode, task, list(x ORDER BY i)::FLOAT\[1408\] AS e
  FROM (SELECT episode, task, i, avg(x) AS x FROM dims GROUP BY ALL)
  GROUP BY episode, task
)
SELECT a.episode AS keep, b.episode AS near\_duplicate, a.task,
       round(array\_cosine\_similarity(a.e, b.e), 4) AS similarity
FROM episodes a JOIN episodes b ON a.task = b.task AND a.episode < b.episode
WHERE array\_cosine\_similarity(a.e, b.e) > 0.98
ORDER BY similarity DESC;

The multimodal embedding is 1,408 floats; run `DESCRIBE vectors` and \`SELECT len(vector\_0) FROM vectors LIMIT 1\` to confirm which column holds it. The 0.98 cutoff is a starting point. Open a dozen pairs around it and move it until the pairs above it look like the same demonstration recorded twice. An episode that appears in the failed-grasp, handoff or off-task list stays in the queue even if it has a near duplicate, since those lists are about what happened, and two episodes can look alike up to the moment one of them fails.  
  

## What does it cost?

  
  
Indexing is per minute of video per camera. At the rate in Mixpeek's public features API on 2026-11-11, video search is $0.066 a minute ($3.96 an hour). DROID's 350 hours from the wrist camera alone come to about $1,386; all three cameras triple it.  
  
The confirm stage is a model call per window that reaches it, after the task filter. With `final_top_k` at 100, one search for one task sends at most 100 windows to it. The export and the DuckDB queries run in your own bucket and on your own machine.  
  

## What does Mixpeek not do here?

  
  
Mixpeek finds and ranks the episodes. Labeling happens in your labeling tool: Mixpeek draws no boxes, masks, keypoints or 3D cuboids. It reads video files, so each camera stream needs to be a video file with the episode fields attached as object metadata. Robot log formats and sensor streams other than video stay in your own pipeline, and the time ranges this search returns are positions in the video file.  
  

## Frequently Asked Questions

  
  

### How do I decide which robot episodes to label?

  
  
Label the episodes that differ from the rest: failures, handoffs and recoveries, plus a sample of clean successes. Find the first group by searching the episode video in plain language and confirming each match with a vision model, and thin the clean successes by removing episodes whose embeddings are nearly identical to another episode of the same task.  
  

### Can I search robot video in S3 without downloading it?

  
  
Yes. Register each video's S3 or GCS location as an object in a Mixpeek bucket, and Mixpeek reads the file from there to index it. The index holds windows, embeddings and your metadata, and the video stays in your bucket.  
  

### How do I find near-duplicate demonstrations?

  
  
Average each episode's window embeddings into one vector, then compare episodes of the same task by cosine similarity. Pairs above a cutoff you have checked by eye, such as 0.98, are candidates to skip. DuckDB does this over a Parquet export with `array_cosine_similarity`.  
  

### How do I find off-task frames in teleoperation data?

  
  
Search for what off-task looks like ("the arm is still and a person rearranges the table") and confirm each match with a yes-or-no question to a vision model. The confirm step matters here because a still arm also appears at the start of most clean episodes, and the question can say which stillness you mean.  
  

### What window length should I use?

  
  
Three seconds is a good start for manipulation. Grasps, slips and handoffs take about a second, and a window much longer than the event averages it away.  
  

## Related reading

  
  
*   [How do I find failed grasps and handoffs across thousands of robot episodes?](https://mixpeek.com/guides/how-to-find-failed-grasps-and-handoffs-in-robot-and-egocentric-video)
*   [How do I find specific moments across thousands of hours of robot or dashcam recordings?](https://mixpeek.com/guides/how-to-find-moments-in-thousands-of-hours-of-dashcam-or-robot-footage)
*   [How to stop duplicate videos showing up in search results](https://mixpeek.com/guides/how-to-stop-duplicate-videos-showing-up-in-search-results)

## Search your episodes before you label them

Mixpeek reads robot and egocentric video where it sits in S3, GCS or any bucket, indexes short timestamped windows with your episode fields, and returns the episodes and time ranges that match each search.

[Index your episodes](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide&utm_campaign=guide_cta&utm_content=robot_labeling)[See pricing](/pricing)

[Explore MVS](/mvs)[See pricing](/pricing)

## Related guides

[

Video Understanding

### How Do I Find Failed Grasps and Handoffs Across Thousands of Robot Episodes?

Cut each episode's camera video into 2 to 5 second windows, search them in plain language, confirm every match by sending the clip to a vision model with a yes-or-no question, and merge what survives into timestamped moments per episode. Then export the segments, embeddings and lineage as Parquet and count failures by task, robot and camera in DuckDB.

Read guide →](/guides/how-to-find-failed-grasps-and-handoffs-in-robot-and-egocentric-video)[

Video Understanding

### How Do I Find Specific Moments Across Thousands of Hours of Robot or Dashcam Recordings?

Narrow by telemetry first, skip idle footage, then search what is left by meaning. Cut continuous recordings into fixed 10 to 30 second segments, embed each segment, store its timestamps next to the vehicle or robot's sensor data, and filter before you rank. For 10,000 hours, that is 1.2 million vectors and about 15 GB of index, next to 36 TB of footage.

Read guide →](/guides/how-to-find-moments-in-thousands-of-hours-of-dashcam-or-robot-footage)[

Video Understanding

### How Do I Search Video From Snowflake?

Keep the campaign rows in Snowflake and index the video files they point to scene by scene in Mixpeek, with the warehouse key on every scene. Then load the scene matches back into a Snowflake table and answer questions like "which ads show the product in the first three seconds" with a SQL join.

Read guide →](/guides/how-to-search-video-from-snowflake)