How do I find the robot episodes worth labeling?
Search the episodes before you label any of them. Index each episode's camera video in short windows where it already sits in S3 or GCS, with the episode ID, task and robot on every window. Then run one search per kind of episode you want labeled (failed grasps, handoffs, off-task stretches), confirm each match by sending the clip to a vision model with a yes-or-no question, and drop near-duplicate demonstrations by comparing episode embeddings. What comes out is a short list of episodes and time ranges, and that list is what goes to your labeling tool.
Why select episodes before labeling them?
Most episodes in a demonstration archive are clean successes of the same task, recorded again and again. Labeling all of them pays the same per-episode cost for the hundredth clean pick as for the first failed one. The episodes that teach a policy something new are the rare ones: a grasp that slips, an object passed to a person, a stretch where the arm stops while the operator resets the scene.
Those events are short. A failed grasp lasts about a second, and episodes are often under a minute: DROID, a public robot manipulation dataset, holds 76,000 trajectories and 350 hours of interaction (DROID), about 17 seconds per trajectory. Finding them by watching means watching everything, so the search has to work on what happens inside the video.
What should I search for?
| Kind of episode | Search query | Confirm question for the vision model |
|---|---|---|
| Failed grasp | gripper closes but the object slips out | Keep only clips where the gripper closes on the object and the object falls or slips out |
| Handoff | a person's hand takes the object from the gripper | Keep only clips where the object passes from the robot to a person's hand |
| Off-task | the arm is still and a person rearranges the table | Keep only clips where the robot is not working on the task: the arm is idle, the object is out of view, or a person is resetting the scene or adjusting the camera |
| Near-duplicate demos | no query: compare episode embeddings | none: pick a similarity cutoff by eye |
The first three are searches. The fourth compares episodes with each other, so it runs in SQL over an export of the embeddings.
How do I index the episodes?
Upload one video file per camera per episode as an object in a bucket, with the episode fields as metadata. Then create a collection that splits video by time. The multimodal extractor writes
start_time and end_time on every window, and field_passthrough copies your episode fields
onto each one:{
"collection_name": "robot-episodes",
"source": { "type": "bucket", "bucket_ids": ["bkt_episodes"] },
"feature_extractor": {
"feature_extractor_name": "multimodal_extractor",
"version": "v1",
"input_mappings": { "video": "video_url" },
"field_passthrough": [
{ "source_path": "metadata.episode_id" },
{ "source_path": "metadata.task" },
{ "source_path": "metadata.robot_id" },
{ "source_path": "metadata.camera" }
],
"parameters": {
"split_method": "time",
"time_split_interval": 3,
"run_multimodal_embedding": true,
"run_video_description": true,
"enable_thumbnails": true
}
}
}What retriever finds the episodes?
One retriever serves every row of the table above. The query and the confirm question are inputs, so each call can ask for a different kind of episode:
{
"retriever_name": "episodes-to-label",
"collection_identifiers": ["robot-episodes"],
"input_schema": {
"query": { "type": "text", "required": true },
"criteria": { "type": "text", "required": true },
"task": { "type": "text", "required": true }
},
"stages": [
{ "stage_name": "search", "stage_type": "filter",
"config": { "stage_id": "feature_search", "parameters": {
"final_top_k": 100,
"searches": [ { "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
"query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 100 } ] } } },
{ "stage_name": "one_task", "stage_type": "filter",
"config": { "stage_id": "attribute_filter", "parameters": {
"field": "metadata.task", "operator": "eq", "value": "{{INPUT.task}}" } } },
{ "stage_name": "confirm", "stage_type": "filter",
"config": { "stage_id": "llm_filter", "parameters": {
"criteria": "{{INPUT.criteria}}",
"include_media": true,
"include_reasoning": true } } },
{ "stage_name": "moments", "stage_type": "reduce",
"config": { "stage_id": "moment_group", "parameters": {
"parent_field": "source_object_id",
"time_field": "start_time",
"merge_tolerance_ms": 1000,
"max_moments_per_parent": 3,
"score_strategy": "max",
"output_mode": "annotated" } } }
]
}include_media: true) with your questioninclude_reasoning returns the model's reason, which is
useful to a labeler.
time_field: "start_time" for a text query; with its default the moments come back empty.How do I turn the results into a labeling list?
Run the retriever once per kind of episode and task, and write every moment to a CSV:
import csv, os, requests
API = "https://api.mixpeek.com"
HEADERS = {"Authorization": f"Bearer {os.environ['MIXPEEK_API_KEY']}",
"X-Namespace": os.environ["MIXPEEK_NAMESPACE"]}
RETRIEVER = "episodes-to-label" # or its ret_ id
KINDS = {
"failed_grasp": ("gripper closes but the object slips out",
"Keep only clips where the gripper closes on the object and the object falls or slips out"),
"handoff": ("a person's hand takes the object from the gripper",
"Keep only clips where the object passes from the robot to a person's hand"),
"off_task": ("the arm is still and a person rearranges the table",
"Keep only clips where the robot is not working on the task: the arm is idle, "
"the object is out of view, or a person is resetting the scene or adjusting the camera"),
}
TASKS = ["pick_cup", "open_drawer"]
with open("label_queue.csv", "w", newline="") as f:
w = csv.writer(f)
w.writerow(["kind", "task", "episode_id", "camera", "start_s", "end_s", "score", "reason"])
for task in TASKS:
for kind, (query, criteria) in KINDS.items():
r = requests.post(f"{API}/v1/retrievers/{RETRIEVER}/execute", headers=HEADERS, timeout=300,
json={"inputs": {"query": query, "criteria": criteria, "task": task}})
r.raise_for_status()
out = r.json()
confirm = out.get("stage_statistics", {}).get("stages", {}).get("confirm", {})
reasons = {d["document_id"]: d.get("reason", "")
for d in confirm.get("metadata", {}).get("llm_decisions", [])}
for doc in out["documents"]:
meta = doc.get("metadata", {})
for m in doc.get("moments", []):
w.writerow([kind, task, meta.get("episode_id"), meta.get("camera"),
m["start_ms"] / 1000, m["end_ms"] / 1000, round(m["score"], 4),
reasons.get(doc["document_id"], "")])documents. The confirm stage's yes-or-no decisions, with the model's reason
for each, are in stage_statistics under the stage's name. Print one response before you trust the
column mapping, since where your episode fields land depends on how you uploaded them. Each row is an episode,
a camera, and the seconds to look at. Send the rows to your labeling tool as its task list, with
the time range so a labeler starts at the event.How do I drop near-duplicate demonstrations?
Export the collection with its embeddings and compare episodes in DuckDB. The export endpoint writes Parquet to a storage connection you own (with write access turned on), and
include_vectors adds
a file of embeddings keyed by document_id:POST /v1/collections/robot-episodes/export
{
"format": "parquet",
"include_vectors": true,
"destination": { "connection_id": "conn_training_bucket", "prefix": "exports/2026-11-21/" }
}manifest.json under the prefix that names every file. Average each episode's
window embeddings, then list pairs of episodes in the same task whose averages are almost
identical:INSTALL httpfs;
LOAD httpfs;
CREATE OR REPLACE SECRET s3 (TYPE s3, PROVIDER credential_chain);
-- File names come from manifest.json under the export prefix.
CREATE VIEW segments AS
SELECT * FROM read_parquet('s3://training-bucket/exports/2026-11-21/<documents file>.parquet');
CREATE VIEW vectors AS
SELECT * FROM read_parquet('s3://training-bucket/exports/2026-11-21/<vectors file>.parquet');
WITH dims AS (
SELECT s."metadata.episode_id" AS episode, s."metadata.task" AS task,
unnest(v.vector_0) AS x, generate_subscripts(v.vector_0, 1) AS i
FROM vectors v JOIN segments s USING (document_id)
WHERE s."metadata.camera" = 'wrist'
),
episodes AS (
SELECT episode, task, list(x ORDER BY i)::FLOAT[1408] AS e
FROM (SELECT episode, task, i, avg(x) AS x FROM dims GROUP BY ALL)
GROUP BY episode, task
)
SELECT a.episode AS keep, b.episode AS near_duplicate, a.task,
round(array_cosine_similarity(a.e, b.e), 4) AS similarity
FROM episodes a JOIN episodes b ON a.task = b.task AND a.episode < b.episode
WHERE array_cosine_similarity(a.e, b.e) > 0.98
ORDER BY similarity DESC;DESCRIBE vectors and `SELECT len(vector_0) FROM
vectors LIMIT 1` to confirm which column holds it. The 0.98 cutoff is a starting point. Open a
dozen pairs around it and move it until the pairs above it look like the same demonstration
recorded twice. An episode that appears in the failed-grasp, handoff or off-task list stays in the
queue even if it has a near duplicate, since those lists are about what happened, and two
episodes can look alike up to the moment one of them fails.What does it cost?
Indexing is per minute of video per camera. At the rate in Mixpeek's public features API on 2026-11-11, video search is $0.066 a minute ($3.96 an hour). DROID's 350 hours from the wrist camera alone come to about $1,386; all three cameras triple it.
The confirm stage is a model call per window that reaches it, after the task filter. With
final_top_k at 100, one search for one task sends at most 100 windows to it. The export and the
DuckDB queries run in your own bucket and on your own machine.What does Mixpeek not do here?
Mixpeek finds and ranks the episodes. Labeling happens in your labeling tool: Mixpeek draws no boxes, masks, keypoints or 3D cuboids. It reads video files, so each camera stream needs to be a video file with the episode fields attached as object metadata. Robot log formats and sensor streams other than video stay in your own pipeline, and the time ranges this search returns are positions in the video file.
Frequently Asked Questions
How do I decide which robot episodes to label?
Label the episodes that differ from the rest: failures, handoffs and recoveries, plus a sample of clean successes. Find the first group by searching the episode video in plain language and confirming each match with a vision model, and thin the clean successes by removing episodes whose embeddings are nearly identical to another episode of the same task.
Can I search robot video in S3 without downloading it?
Yes. Register each video's S3 or GCS location as an object in a Mixpeek bucket, and Mixpeek reads the file from there to index it. The index holds windows, embeddings and your metadata, and the video stays in your bucket.
How do I find near-duplicate demonstrations?
Average each episode's window embeddings into one vector, then compare episodes of the same task by cosine similarity. Pairs above a cutoff you have checked by eye, such as 0.98, are candidates to skip. DuckDB does this over a Parquet export with
array_cosine_similarity.How do I find off-task frames in teleoperation data?
Search for what off-task looks like ("the arm is still and a person rearranges the table") and confirm each match with a yes-or-no question to a vision model. The confirm step matters here because a still arm also appears at the start of most clean episodes, and the question can say which stillness you mean.
What window length should I use?
Three seconds is a good start for manipulation. Grasps, slips and handoffs take about a second, and a window much longer than the event averages it away.