How Do I Find Failed Grasps and Handoffs Across Thousands of Robot Episodes?

How do I find failed grasps and handoffs across thousands of robot episodes?

Index each episode's camera video in short windows, 2 to 5 seconds, with the episode's metadata (task, robot, camera, operator) on every window. Search the windows in plain language, such as "gripper closes but the cup slips out" or "person takes the screwdriver from the robot's hand". Then send each top match to a vision-language model as video with a yes-or-no question, keep the ones it confirms, and merge neighboring windows into one moment per episode with a start and end time. Export what you kept to your own bucket as Parquet and answer the counting questions ("which task fails most, on which robot") in SQL.

Why are manipulation episodes harder to search than dashcam footage?

The event is short and the episode is short. A failed grasp lasts about a second inside an episode of a few seconds to a few minutes. DROID, a public robot manipulation dataset, holds 76,000 trajectories and 350 hours of interaction (DROID), which works out to about 17 seconds per trajectory. A 30 second segment would hold the whole episode, and a search would return the episode with no idea where the slip happened.

The words that matter describe motion and outcome: "slips", "misses", "drops", "hands over", "regrasps". A single frame shows a gripper near a cup whether the grasp worked or not. And there are often several cameras. DROID records with two external stereo cameras and one on the wrist, and a slip that is plain from the wrist camera can be invisible from across the table.

Egocentric video brings the same problems at a larger scale. Ego4D has 3,670 hours of first-person video from 923 participants in 74 locations across 9 countries (Ego4D), and the moments people search for there are hand-object interactions of a second or two.

For telemetry-first search over long continuous recordings, such as a fleet's dashcams or a robot running all day, see How do I find specific moments across thousands of hours of robot or dashcam recordings? This guide covers short, task-shaped episodes.

What does the pipeline look like?

Step	What it does	Why
1. One video file per camera per episode	Each file carries episode ID, task, robot, camera and operator as metadata	Every later filter and SQL query groups on these fields
2. Short fixed windows	Split by time every 2 to 5 seconds, with a written description of each window	Puts the event in its own window instead of averaging it into the episode
3. Search by meaning, filtered by task	Text query against the window embeddings, limited to one task or robot	Ranks thousands of windows instead of the whole archive
4. Confirm with a vision model	Send each top window as video with a yes-or-no question	Similarity search always returns a best match, even when nothing failed
5. Merge into moments	Adjacent confirmed windows become one moment with start and end times	One row per event instead of one per window
6. Export and count	Parquet with embeddings and lineage, queried in DuckDB	Failure rates by task, robot and camera, and a training set you can trace

How do I set it up in Mixpeek?

Upload each camera file as an object in a bucket, with its metadata, then create a collection that splits video by time. The multimodal extractor takes time_split_interval in seconds and writes start_time and end_time on every segment. field_passthrough copies your episode fields onto each segment:

{
  "collection_name": "robot-episodes",
  "source": { "type": "bucket", "bucket_ids": ["bkt_episodes"] },
  "feature_extractor": {
    "feature_extractor_name": "multimodal_extractor",
    "version": "v1",
    "input_mappings": { "video": "video_url" },
    "field_passthrough": [
      { "source_path": "metadata.episode_id" },
      { "source_path": "metadata.task" },
      { "source_path": "metadata.robot_id" },
      { "source_path": "metadata.camera" }
    ],
    "parameters": {
      "split_method": "time",
      "time_split_interval": 3,
      "run_multimodal_embedding": true,
      "run_video_description": true,
      "enable_thumbnails": true
    }
  }
}

For egocentric video with narration, also set run_transcription so a search can match what the wearer said as well as what the camera saw.

Then build a retriever with four stages: search the windows, keep one task, confirm each match with LLM Filter, and merge with Moment Group:

{
  "retriever_name": "failed-grasps",
  "collection_identifiers": ["robot-episodes"],
  "input_schema": {
    "query": { "type": "text", "required": true },
    "task": { "type": "text", "required": true }
  },
  "stages": [
    { "stage_name": "search", "stage_type": "filter",
      "config": { "stage_id": "feature_search", "parameters": {
        "final_top_k": 100,
        "searches": [ { "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
          "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 } ] } } },
    { "stage_name": "one_task", "stage_type": "filter",
      "config": { "stage_id": "attribute_filter", "parameters": {
        "field": "metadata.task", "operator": "eq", "value": "{{INPUT.task}}" } } },
    { "stage_name": "confirm", "stage_type": "filter",
      "config": { "stage_id": "llm_filter", "parameters": {
        "criteria": "Keep only clips where the gripper closes on the object and the object falls or slips out",
        "include_media": true,
        "include_reasoning": true } } },
    { "stage_name": "moments", "stage_type": "reduce",
      "config": { "stage_id": "moment_group", "parameters": {
        "parent_field": "source_object_id",
        "time_field": "start_time",
        "merge_tolerance_ms": 1000,
        "max_moments_per_parent": 3,
        "score_strategy": "max",
        "output_mode": "annotated" } } }
  ]
}

Three settings are easy to get wrong:

include_media: true on the filter. Without it the model judges each window by its text

fields only. With it, a video segment goes to Gemini as video, which is what you need to tell a slip from a clean grasp.

time_field: "start_time" for a text query. Moment Group's default reads intervals that

exist only when the query is a file. With a text query and the default, the moments come back empty.

Keep the filter's input small. LLM Filter calls a model per document, and its docs advise

narrowing first when more than 100 documents reach it. Filter by task or robot, or lower final_top_k, before the confirm stage.

For a handoff, change the query and the criteria: "a person's hand takes the object from the gripper" and "Keep only clips where the object passes from the robot to a person's hand". The same retriever then finds handoffs instead of drops.

How do I count failures by task and robot in DuckDB?

Export the collection as Parquet into a storage connection you own. The export endpoint writes Parquet by default. include_vectors adds a second Parquet file of embeddings, and include_lineage adds each segment's processing steps and the SHA-256 of its source file:

POST /v1/collections/robot-episodes/export
{
  "format": "parquet",
  "include_vectors": true,
  "include_lineage": true,
  "destination": { "connection_id": "conn_training_bucket", "prefix": "exports/2026-11-07/" }
}

The destination must be an S3 or GCS connection with write access turned on (PATCH /v1/organizations/connections/{connection_id} with {"write_enabled": true}). The export writes a manifest.json under the prefix that lists every file with its size and checksum. Read the file names from it.

Nested fields are flattened with dots, so metadata.task becomes a column named "metadata.task". The vectors file has a document_id column and one vector_N column per embedding. Run DESCRIBE once to see the exact names:

INSTALL httpfs;
LOAD httpfs;
CREATE OR REPLACE SECRET s3 (TYPE s3, PROVIDER credential_chain);

-- File names come from manifest.json under the export prefix.
CREATE VIEW segments AS
  SELECT * FROM read_parquet('s3://training-bucket/exports/2026-11-07/<documents file>.parquet');
CREATE VIEW vectors AS
  SELECT * FROM read_parquet('s3://training-bucket/exports/2026-11-07/<vectors file>.parquet');
DESCRIBE segments;

-- How much indexed video each task and robot has
SELECT "metadata.task" AS task, "metadata.robot_id" AS robot,
       count(*) AS windows, round(sum(end_time - start_time) / 3600, 1) AS hours
FROM segments
GROUP BY ALL
ORDER BY hours DESC;

The retriever's confirmed moments are not part of the collection export. Save each run's results (document ID, episode, start and end time, and the filter's reasoning) to a CSV or Parquet file and join it in:

CREATE VIEW failures AS SELECT * FROM read_csv('failed_grasps_2026-11-07.csv');

-- Failure rate per task and robot, counted by episode
SELECT s."metadata.task" AS task, s."metadata.robot_id" AS robot,
       count(DISTINCT f.episode_id) AS failed_episodes,
       count(DISTINCT s."metadata.episode_id") AS episodes,
       round(100.0 * count(DISTINCT f.episode_id) / count(DISTINCT s."metadata.episode_id"), 1) AS pct
FROM segments s
LEFT JOIN failures f ON f.document_id = s.document_id
GROUP BY ALL
ORDER BY pct DESC;

The embeddings also answer "find more like this one" in SQL. DuckDB's array_cosine_similarity needs fixed-size arrays, so cast the vector column to its dimension. The multimodal embedding is 1,408 floats; check which vector_N holds it with len(vector_0):

WITH v AS (SELECT document_id, vector_0::FLOAT[1408] AS e FROM vectors),
     q AS (SELECT e FROM v WHERE document_id = 'doc_a_confirmed_slip')
SELECT s."metadata.episode_id", s.start_time, s.end_time,
       array_cosine_similarity(v.e, q.e) AS similarity
FROM v JOIN segments s USING (document_id), q
ORDER BY similarity DESC
LIMIT 50;

The lineage columns make the kept set traceable. source_content_hash ties each row to the exact bytes of its source video, so a training run can pin this export and check later that nothing under it changed.

What does it cost?

At the rates in Mixpeek's public features API on 2026-11-07: video search is $0.05 a minute ($3 an hour) and multimodal understanding $0.0292 a minute. Rates apply per camera stream, so a dataset's hours multiply by its camera count.

Archive	Camera hours	At $3 an hour
DROID, 350 hours, three cameras	1,050	$3,150
DROID, wrist camera only	350	$1,050
Ego4D, 3,670 hours, one camera	3,670	$11,010

Indexing only the camera that sees the gripper is the biggest saving. The LLM Filter stage is a separate model call per window that reaches it, which is another reason to filter by task first. The short-window pipeline above uses the multimodal extractor, so run the preflight estimate on a sample of episodes before you index the archive.

What does Mixpeek not do here?

Mixpeek reads video, images, audio and documents. It does not parse MCAP, ROS bag or LeRobot files, so convert each camera topic to a video file and carry the episode fields as object metadata. It has no synchronized multi-sensor timeline viewer and no pixel, mask or 3D labeling. Teams that need those use a labeling tool on the episodes this pipeline selects. Our comparison with FiftyOne covers where each tool fits.

Frequently Asked Questions

How do I find failed grasps in robot demonstration videos?

Split each episode's video into 2 to 5 second windows, embed them, and search with a sentence that describes the failure, such as "gripper closes and the object slips out". Then ask a vision-language model a yes-or-no question about each top match, sending the clip itself, and keep only the ones it confirms. Filter by task first so the model checks a few hundred windows instead of the whole archive.

What segment length works for robot manipulation video?

Two to five seconds. Grasps, slips and handoffs take about a second, and episodes are often under a minute: DROID averages about 17 seconds per trajectory. Thirty second segments, the usual default for long video, would put most episodes in a single segment and lose the timestamp of the event.

Can I search egocentric video for hand-object interactions?

Yes, with the same windows and a query about the hands, such as "cuts the onion" or "picks up the phone". If the wearer narrates, add transcription so the search can match speech as well as the image.

How do I query video features with SQL?

Export the collection as Parquet with embeddings and lineage, then read the files with DuckDB, Spark, BigQuery or Snowflake. DuckDB reads Parquet straight from S3 through its httpfs extension, and array_cosine_similarity ranks segments by embedding similarity inside a query.

Does Mixpeek read MCAP or LeRobot datasets?

No. Export each camera topic to a video file, keep the episode's timestamps, and attach the episode ID, task and robot as metadata on the object. The search then filters and groups on those fields.

How Do I Find Failed Grasps and Handoffs Across Thousands of Robot Episodes?