NEWVectors or files. Pick a path.Start →
    Video Understanding
    8 min read
    Updated 2026-11-21

    How Do I Find the Robot Episodes Worth Labeling?

    Search robot and egocentric episodes where they sit in S3 or GCS before labeling any of them: index short windows with the episode fields, search for failed grasps, handoffs and off-task stretches, confirm each match with a vision model, and drop near-duplicate demonstrations by comparing episode embeddings in DuckDB. The output is a list of episodes and time ranges for your labeling tool.

    Ethan Steininger
    By , Founder & CEO, Mixpeek
    Robotics
    Physical AI
    Training Data
    Data Curation
    Egocentric Video
    Video Search
    DuckDB

    How do I find the robot episodes worth labeling?



    Search the episodes before you label any of them. Index each episode's camera video in short windows where it already sits in S3 or GCS, with the episode ID, task and robot on every window. Then run one search per kind of episode you want labeled (failed grasps, handoffs, off-task stretches), confirm each match by sending the clip to a vision model with a yes-or-no question, and drop near-duplicate demonstrations by comparing episode embeddings. What comes out is a short list of episodes and time ranges, and that list is what goes to your labeling tool.

    Why select episodes before labeling them?



    Most episodes in a demonstration archive are clean successes of the same task, recorded again and again. Labeling all of them pays the same per-episode cost for the hundredth clean pick as for the first failed one. The episodes that teach a policy something new are the rare ones: a grasp that slips, an object passed to a person, a stretch where the arm stops while the operator resets the scene.

    Those events are short. A failed grasp lasts about a second, and episodes are often under a minute: DROID, a public robot manipulation dataset, holds 76,000 trajectories and 350 hours of interaction (DROID), about 17 seconds per trajectory. Finding them by watching means watching everything, so the search has to work on what happens inside the video.

    What should I search for?



    Kind of episodeSearch queryConfirm question for the vision model
    Failed graspgripper closes but the object slips outKeep only clips where the gripper closes on the object and the object falls or slips out
    Handoffa person's hand takes the object from the gripperKeep only clips where the object passes from the robot to a person's hand
    Off-taskthe arm is still and a person rearranges the tableKeep only clips where the robot is not working on the task: the arm is idle, the object is out of view, or a person is resetting the scene or adjusting the camera
    Near-duplicate demosno query: compare episode embeddingsnone: pick a similarity cutoff by eye


    The first three are searches. The fourth compares episodes with each other, so it runs in SQL over an export of the embeddings.

    How do I index the episodes?



    Upload one video file per camera per episode as an object in a bucket, with the episode fields as metadata. Then create a collection that splits video by time. The multimodal extractor writes start_time and end_time on every window, and field_passthrough copies your episode fields onto each one:
    {
      "collection_name": "robot-episodes",
      "source": { "type": "bucket", "bucket_ids": ["bkt_episodes"] },
      "feature_extractor": {
        "feature_extractor_name": "multimodal_extractor",
        "version": "v1",
        "input_mappings": { "video": "video_url" },
        "field_passthrough": [
          { "source_path": "metadata.episode_id" },
          { "source_path": "metadata.task" },
          { "source_path": "metadata.robot_id" },
          { "source_path": "metadata.camera" }
        ],
        "parameters": {
          "split_method": "time",
          "time_split_interval": 3,
          "run_multimodal_embedding": true,
          "run_video_description": true,
          "enable_thumbnails": true
        }
      }
    }
    Three seconds keeps a one-second event in its own window. Index the camera that sees the gripper first; for many manipulation setups that is the wrist camera, and it is the cheapest way to start. How do I find failed grasps and handoffs across thousands of robot episodes? covers window length, multiple cameras and egocentric video in more depth.

    What retriever finds the episodes?



    One retriever serves every row of the table above. The query and the confirm question are inputs, so each call can ask for a different kind of episode:
    {
      "retriever_name": "episodes-to-label",
      "collection_identifiers": ["robot-episodes"],
      "input_schema": {
        "query": { "type": "text", "required": true },
        "criteria": { "type": "text", "required": true },
        "task": { "type": "text", "required": true }
      },
      "stages": [
        { "stage_name": "search", "stage_type": "filter",
          "config": { "stage_id": "feature_search", "parameters": {
            "final_top_k": 100,
            "searches": [ { "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
              "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 } ] } } },
        { "stage_name": "one_task", "stage_type": "filter",
          "config": { "stage_id": "attribute_filter", "parameters": {
            "field": "metadata.task", "operator": "eq", "value": "{{INPUT.task}}" } } },
        { "stage_name": "confirm", "stage_type": "filter",
          "config": { "stage_id": "llm_filter", "parameters": {
            "criteria": "{{INPUT.criteria}}",
            "include_media": true,
            "include_reasoning": true } } },
        { "stage_name": "moments", "stage_type": "reduce",
          "config": { "stage_id": "moment_group", "parameters": {
            "parent_field": "source_object_id",
            "time_field": "start_time",
            "merge_tolerance_ms": 1000,
            "max_moments_per_parent": 3,
            "score_strategy": "max",
            "output_mode": "annotated" } } }
      ]
    }
    What each stage is for:

  1. Search ranks the windows by how well they match the sentence. It always returns a best match,
  2. even in a task where nothing failed, which is why the next two stages exist.
  3. One task keeps the confirm stage's input small. LLM Filter
  4. calls a model once per window that reaches it.
  5. Confirm sends each window to the model as video (include_media: true) with your question
  6. and keeps the ones it answers yes to. include_reasoning returns the model's reason, which is useful to a labeler.
  7. Moments merges neighboring confirmed windows into one moment per episode file, with start and
  8. end times. Moment Group needs time_field: "start_time" for a text query; with its default the moments come back empty.

    How do I turn the results into a labeling list?



    Run the retriever once per kind of episode and task, and write every moment to a CSV:
    import csv, os, requests
    
    API = "https://api.mixpeek.com"
    HEADERS = {"Authorization": f"Bearer {os.environ['MIXPEEK_API_KEY']}",
               "X-Namespace": os.environ["MIXPEEK_NAMESPACE"]}
    RETRIEVER = "episodes-to-label"  # or its ret_ id
    
    KINDS = {
        "failed_grasp": ("gripper closes but the object slips out",
                         "Keep only clips where the gripper closes on the object and the object falls or slips out"),
        "handoff": ("a person's hand takes the object from the gripper",
                    "Keep only clips where the object passes from the robot to a person's hand"),
        "off_task": ("the arm is still and a person rearranges the table",
                     "Keep only clips where the robot is not working on the task: the arm is idle, "
                     "the object is out of view, or a person is resetting the scene or adjusting the camera"),
    }
    TASKS = ["pick_cup", "open_drawer"]
    
    with open("label_queue.csv", "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(["kind", "task", "episode_id", "camera", "start_s", "end_s", "score", "reason"])
        for task in TASKS:
            for kind, (query, criteria) in KINDS.items():
                r = requests.post(f"{API}/v1/retrievers/{RETRIEVER}/execute", headers=HEADERS, timeout=300,
                                  json={"inputs": {"query": query, "criteria": criteria, "task": task}})
                r.raise_for_status()
                out = r.json()
                confirm = out.get("stage_statistics", {}).get("stages", {}).get("confirm", {})
                reasons = {d["document_id"]: d.get("reason", "")
                           for d in confirm.get("metadata", {}).get("llm_decisions", [])}
                for doc in out["documents"]:
                    meta = doc.get("metadata", {})
                    for m in doc.get("moments", []):
                        w.writerow([kind, task, meta.get("episode_id"), meta.get("camera"),
                                    m["start_ms"] / 1000, m["end_ms"] / 1000, round(m["score"], 4),
                                    reasons.get(doc["document_id"], "")])
    The results are in documents. The confirm stage's yes-or-no decisions, with the model's reason for each, are in stage_statistics under the stage's name. Print one response before you trust the column mapping, since where your episode fields land depends on how you uploaded them. Each row is an episode, a camera, and the seconds to look at. Send the rows to your labeling tool as its task list, with the time range so a labeler starts at the event.

    How do I drop near-duplicate demonstrations?



    Export the collection with its embeddings and compare episodes in DuckDB. The export endpoint writes Parquet to a storage connection you own (with write access turned on), and include_vectors adds a file of embeddings keyed by document_id:
    POST /v1/collections/robot-episodes/export
    {
      "format": "parquet",
      "include_vectors": true,
      "destination": { "connection_id": "conn_training_bucket", "prefix": "exports/2026-11-21/" }
    }
    The export writes a manifest.json under the prefix that names every file. Average each episode's window embeddings, then list pairs of episodes in the same task whose averages are almost identical:
    INSTALL httpfs;
    LOAD httpfs;
    CREATE OR REPLACE SECRET s3 (TYPE s3, PROVIDER credential_chain);
    
    -- File names come from manifest.json under the export prefix.
    CREATE VIEW segments AS
      SELECT * FROM read_parquet('s3://training-bucket/exports/2026-11-21/<documents file>.parquet');
    CREATE VIEW vectors AS
      SELECT * FROM read_parquet('s3://training-bucket/exports/2026-11-21/<vectors file>.parquet');
    
    WITH dims AS (
      SELECT s."metadata.episode_id" AS episode, s."metadata.task" AS task,
             unnest(v.vector_0) AS x, generate_subscripts(v.vector_0, 1) AS i
      FROM vectors v JOIN segments s USING (document_id)
      WHERE s."metadata.camera" = 'wrist'
    ),
    episodes AS (
      SELECT episode, task, list(x ORDER BY i)::FLOAT[1408] AS e
      FROM (SELECT episode, task, i, avg(x) AS x FROM dims GROUP BY ALL)
      GROUP BY episode, task
    )
    SELECT a.episode AS keep, b.episode AS near_duplicate, a.task,
           round(array_cosine_similarity(a.e, b.e), 4) AS similarity
    FROM episodes a JOIN episodes b ON a.task = b.task AND a.episode < b.episode
    WHERE array_cosine_similarity(a.e, b.e) > 0.98
    ORDER BY similarity DESC;
    The multimodal embedding is 1,408 floats; run DESCRIBE vectors and `SELECT len(vector_0) FROM vectors LIMIT 1` to confirm which column holds it. The 0.98 cutoff is a starting point. Open a dozen pairs around it and move it until the pairs above it look like the same demonstration recorded twice. An episode that appears in the failed-grasp, handoff or off-task list stays in the queue even if it has a near duplicate, since those lists are about what happened, and two episodes can look alike up to the moment one of them fails.

    What does it cost?



    Indexing is per minute of video per camera. At the rate in Mixpeek's public features API on 2026-11-11, video search is $0.066 a minute ($3.96 an hour). DROID's 350 hours from the wrist camera alone come to about $1,386; all three cameras triple it.

    The confirm stage is a model call per window that reaches it, after the task filter. With final_top_k at 100, one search for one task sends at most 100 windows to it. The export and the DuckDB queries run in your own bucket and on your own machine.

    What does Mixpeek not do here?



    Mixpeek finds and ranks the episodes. Labeling happens in your labeling tool: Mixpeek draws no boxes, masks, keypoints or 3D cuboids. It reads video files, so each camera stream needs to be a video file with the episode fields attached as object metadata. Robot log formats and sensor streams other than video stay in your own pipeline, and the time ranges this search returns are positions in the video file.

    Frequently Asked Questions



    How do I decide which robot episodes to label?



    Label the episodes that differ from the rest: failures, handoffs and recoveries, plus a sample of clean successes. Find the first group by searching the episode video in plain language and confirming each match with a vision model, and thin the clean successes by removing episodes whose embeddings are nearly identical to another episode of the same task.

    Can I search robot video in S3 without downloading it?



    Yes. Register each video's S3 or GCS location as an object in a Mixpeek bucket, and Mixpeek reads the file from there to index it. The index holds windows, embeddings and your metadata, and the video stays in your bucket.

    How do I find near-duplicate demonstrations?



    Average each episode's window embeddings into one vector, then compare episodes of the same task by cosine similarity. Pairs above a cutoff you have checked by eye, such as 0.98, are candidates to skip. DuckDB does this over a Parquet export with array_cosine_similarity.

    How do I find off-task frames in teleoperation data?



    Search for what off-task looks like ("the arm is still and a person rearranges the table") and confirm each match with a yes-or-no question to a vision model. The confirm step matters here because a still arm also appears at the start of most clean episodes, and the question can say which stillness you mean.

    What window length should I use?



    Three seconds is a good start for manipulation. Grasps, slips and handoffs take about a second, and a window much longer than the event averages it away.

    Related reading



  9. How do I find failed grasps and handoffs across thousands of robot episodes?
  10. How do I find specific moments across thousands of hours of robot or dashcam recordings?
  11. How to stop duplicate videos showing up in search results
  12. Managed Mixpeek

    Put multimodal search to work

    Connect a bucket and Mixpeek runs the whole multimodal search pipeline for you: extraction, indexing, and search over your own objects. No models to wire up, nothing to host.

    Start with Managed
    MVS · bring your own

    Already have vectors?

    Keep your embeddings on your own cloud and run dense, sparse, and BM25 search directly on object storage. From $25/mo.

    Start with MVS

    Search your episodes before you label them

    Mixpeek reads robot and egocentric video where it sits in S3, GCS or any bucket, indexes short timestamped windows with your episode fields, and returns the episodes and time ranges that match each search.

    Index your episodesSee pricing