Multimodal
Multimodal Extractor
Unified embeddings for video, audio, image, and text: scene/silence chunking, Whisper transcription, thumbnails, and Gemini vision.
Note: This playground provides simulated output to showcase functionality. No input data is processed or stored on our servers. Use this demo to explore the feature extractor's capabilities before integrating it into your application.
Input
Enter a URL to a video file
Drag and drop a video file here, or click to browse
Output
{
"start_time": "Start time of the segment in seconds, SOURCE-ABSOLUTE (measured from the start of the full source video). See segment_time_frame.",
"end_time": "End time of the segment in seconds, SOURCE-ABSOLUTE. See segment_time_frame.",
"start_frame": "Start frame number of the segment (start_time * fps)",
"end_frame": "End frame number of the segment (end_time * fps)",
"segment_time_frame": "Reference frame for start_time/end_time/start_frame/end_frame: source_absolute means measured from the start of the full source video.",
"cue_time_frame": "Reference frame for transcription_segments cue start/end: scene_relative means measured from this segment's start_time. source-absolute cue time = start_time + cue.",
"fps": "PLAYBACK frame rate of the source video (same as source_fps; BACKE-3446). Null when unknown.",
"source_fps": "Original source video frame rate before preprocessing (e.g. 29.97, 30, 23.976). Null when unknown.",
"analysis_fps": "Frame rate of the preprocessed analysis copy the extractor sampled (target_fps). Null when unknown.",
"duration": "Total source video duration in seconds. Null when unknown, never a fabricated 0.",
"width": "Source frame width in pixels, probed pre-preprocess (UF-37).",
"height": "Source frame height in pixels, probed pre-preprocess (UF-37).",
"aspect_ratio": "Source width/height. Null when geometry is unknown.",
"transcription": "Transcription of audio",
"transcription_segments": "Segment-level transcription with start/end timestamps (seconds). Cue start/end are SCENE-RELATIVE to this segment's start_time (see cue_time_frame); source-absolute time = start_time + cue.start.",
"description": "Generated segment description",
"ocr_text": "OCR text from video frames. Empty string = ran and found none; null = failed or not run (BACKE-3446).",
"json_output": "Raw JSON from underlying models",
"thumbnail_url": "Thumbnail image URL",
"source_video_url": "Original source video URL",
"video_segment_url": "Video segment URL",
"multimodal_extractor_v2_multimodal_embedding": "Dense vector embeddings (3072D) via Gemini Embedding 2 for multimodal content.",
"multimodal_extractor_v2_transcription_embedding": "Dense vector embeddings (1024D) for transcription text via E5-Large.",
"multimodal_extractor_v2_ocr_embedding": "Dense vector embeddings (1024D) for OCR text via E5-Large.",
"multimodal_extractor_v2_description_embedding": "Dense vector embeddings (1024D) for the segment description via E5-Large.",
"internal_metadata": "Internal processing metadata"
}Ready to run Multimodal Extractor on your data? Spin it up in Studio: no infra to host.
Run this in StudioMultimodal Extractor: common questions
- What is the Multimodal Video/Audio/Image (Vertex v1 · Gemini v2) extractor used for?
- Unified multimodal search with higher-dimensional embeddings and native multimodal understanding.
- How much does the Multimodal Video/Audio/Image (Vertex v1 · Gemini v2) extractor cost?
- 200 cr/minute, 5 cr/image, 2 cr/1k_tokens. Credits are billed per unit processed; see mixpeek.com/pricing for the credit rate.
- What file types does Multimodal Video/Audio/Image (Vertex v1 · Gemini v2) support?
- Video, Image, Audio, Text. It produces embeddings from gemini-embedding-2, multilingual_e5_large_instruct_v1, multilingual_e5_large_instruct_ocr_v1, multilingual_e5_large_instruct_description_v1.
Already have embeddings? Skip extraction: search your own vectors with MVS, from $25/mo for up to 1M vectors.
Try MVS →Recent updates
Full changelog- Aug 30, 2026multimodal_extractor_v2 was skipping every image and text objectA collection built on multimodal_extractor_v2 over images produced zero documents and reported no failures, which reads as an empty source rather than a bug. The media-streaming wrapper around the extractor checked for a video segment column, found the placeholder that image and text rows carry for schema alignment in mixed batches, and treated each row as a failed video segment: it passed the row through without running the extractor, so those objects never reached the path that would have downloaded and embedded them. The wrapper now splits three ways, routing rows positively identified as image or text to the extractor directly. The same skip was behind image description and OCR going missing, so both come back with it.
- Aug 27, 2026The transcription_language you set now reaches WhisperBoth multimodal extractors published transcription_language in their public parameter schema with a default of "en", and neither pipeline passed it to Whisper. Setting "ru" on a Russian corpus returned a 200 and transcribed with whatever Whisper guessed from the audio. Nothing errored, because Whisper auto-detects when given no hint, which is why the parameter survived through v1 and v2 looking like it worked. Both legs now forward it. That matters more than it sounds: production runs the API leg and self-hosted tenants run the local one, so wiring either alone would have left the fix inert for real traffic. The local leg had nowhere to put a language at all and gained a config field, passing the hint only when you actually name a language so auto-detect stays the default.
- May 9, 2026Multimodal Extractor v2 with Gemini Embedding 2New multimodal extractor generates 3072-dimensional embeddings using Gemini Embedding 2, enabling richer cross-modal search across text, images, and video.