Multimodal
Multimodal Extractor
Unified embeddings for video, audio, image, and text: scene/silence chunking, Whisper transcription, thumbnails, and Gemini vision.
Note: This playground provides simulated output to showcase functionality. No input data is processed or stored on our servers. Use this demo to explore the feature extractor's capabilities before integrating it into your application.
Input
Enter a URL to a video file
Drag and drop a video file here, or click to browse
Output
{
"start_time": "Start time of the segment in seconds, SOURCE-ABSOLUTE (measured from the start of the full source video). See segment_time_frame.",
"end_time": "End time of the segment in seconds, SOURCE-ABSOLUTE. See segment_time_frame.",
"start_frame": "Start frame number of the segment (start_time * fps)",
"end_frame": "End frame number of the segment (end_time * fps)",
"segment_time_frame": "Reference frame for start_time/end_time/start_frame/end_frame: source_absolute means measured from the start of the full source video.",
"cue_time_frame": "Reference frame for transcription_segments cue start/end: scene_relative means measured from this segment's start_time. source-absolute cue time = start_time + cue.",
"fps": "PLAYBACK frame rate of the source video (same as source_fps; BACKE-3446). Null when unknown.",
"source_fps": "Original source video frame rate before preprocessing (e.g. 29.97, 30, 23.976). Null when unknown.",
"analysis_fps": "Frame rate of the preprocessed analysis copy the extractor sampled (target_fps). Null when unknown.",
"duration": "Total source video duration in seconds. Null when unknown, never a fabricated 0.",
"width": "Source frame width in pixels, probed pre-preprocess (UF-37).",
"height": "Source frame height in pixels, probed pre-preprocess (UF-37).",
"aspect_ratio": "Source width/height. Null when geometry is unknown.",
"transcription": "Transcription of audio",
"transcription_segments": "Segment-level transcription with start/end timestamps (seconds). Cue start/end are SCENE-RELATIVE to this segment's start_time (see cue_time_frame); source-absolute time = start_time + cue.start.",
"description": "Generated segment description",
"ocr_text": "OCR text from video frames. Empty string = ran and found none; null = failed or not run (BACKE-3446).",
"json_output": "Raw JSON from underlying models",
"thumbnail_url": "Thumbnail image URL",
"source_video_url": "Original source video URL",
"video_segment_url": "Video segment URL",
"multimodal_extractor_v2_multimodal_embedding": "Dense vector embeddings (3072D) via Gemini Embedding 2 for multimodal content.",
"multimodal_extractor_v2_transcription_embedding": "Dense vector embeddings (1024D) for transcription text via E5-Large.",
"multimodal_extractor_v2_ocr_embedding": "Dense vector embeddings (1024D) for OCR text via E5-Large.",
"multimodal_extractor_v2_description_embedding": "Dense vector embeddings (1024D) for the segment description via E5-Large.",
"internal_metadata": "Internal processing metadata"
}Ready to run Multimodal Extractor on your data? Spin it up in Studio: no infra to host.
Run this in StudioMultimodal Extractor: common questions
- What is the Multimodal Video/Audio/Image (Vertex v1 · Gemini v2) extractor used for?
- Unified multimodal search with higher-dimensional embeddings and native multimodal understanding.
- How much does the Multimodal Video/Audio/Image (Vertex v1 · Gemini v2) extractor cost?
- 200 cr/minute, 5 cr/image, 2 cr/1k_tokens. Credits are billed per unit processed; see mixpeek.com/pricing for the credit rate.
- What file types does Multimodal Video/Audio/Image (Vertex v1 · Gemini v2) support?
- Video, Image, Audio, Text. It produces embeddings from google/gemini-embedding-2, intfloat/multilingual-e5-large-instruct.
Already have embeddings? Skip extraction: search your own vectors with MVS, from $25/mo for up to 1M vectors.
Try MVS →Related to Multimodal Extractor
Pages on the same stages, extractors and patterns.
Recipe
Reverse Video Search
Search your video library by submitting a video clip as the query. The pipeline decomposes the query video into scene-level embeddings and matches them against your indexed video collection to find visually and semantically similar content.
Learn more
Recipe
Video Semantic Search Pipeline
Build a production-ready video search engine that lets users find specific moments across thousands of hours of video using natural language queries.
Learn more
Recipe
Video Keyframe Extraction Pipeline
Extract representative keyframes from videos with scene detection. Generate thumbnails, visual summaries, and frame-level embeddings.
Learn more
Recipe
Hierarchical Classification
Assign content to multi-level category hierarchies using embedding-based classification. Define your taxonomy once, then classify new content automatically with confidence scores.
Learn more
Recipe
Video Transcription & Indexing Pipeline
Automatically transcribe video content with speaker identification, timestamps, and full-text indexing for downstream search and analytics.
Learn more
Recipe
Brand Safety & Ad Verification Pipeline
Brand safety pipeline for ad networks. Scores video and image creatives against the GARM brand safety floor and suitability categories, which vendors still use after GARM wound down in 2024, before an ad is served.
Learn more
Recent updates
Full changelog- Sep 25, 2026Setting a description_prompt on the multimodal extractor now produces descriptionsmultimodal_extractor v2 stored a custom description_prompt and then ignored it unless run_video_description was also set, so batches completed with no descriptions and no warning. A custom prompt now turns video descriptions on. An explicit run_video_description still wins.
- Sep 24, 2026Plain-text files embed correctly with the multimodal extractormultimodal_extractor v2 sent text files to the embedding model as raw bytes. The model rejected them, and every row of a text-file collection failed with an empty embedding. Text files are now sent as text.
- Aug 30, 2026multimodal_extractor_v2 was skipping every image and text objectA collection built on multimodal_extractor_v2 over images produced zero documents and reported no failures, which reads as an empty source rather than a bug. The media-streaming wrapper around the extractor checked for a video segment column, found the placeholder that image and text rows carry for schema alignment in mixed batches, and treated each row as a failed video segment: it passed the row through without running the extractor, so those objects never reached the path that would have downloaded and embedded them. The wrapper now splits three ways, routing rows positively identified as image or text to the extractor directly. The same skip was behind image description and OCR going missing, so both come back with it.
- Aug 27, 2026The transcription_language you set now reaches WhisperBoth multimodal extractors published transcription_language in their public parameter schema with a default of "en", and neither pipeline passed it to Whisper. Setting "ru" on a Russian corpus returned a 200 and transcribed with whatever Whisper guessed from the audio. Nothing errored, because Whisper auto-detects when given no hint, which is why the parameter survived through v1 and v2 looking like it worked. Both legs now forward it. That matters more than it sounds: production runs the API leg and self-hosted tenants run the local one, so wiring either alone would have left the fix inert for real traffic. The local leg had nowhere to put a language at all and gained a config field, passing the hint only when you actually name a language so auto-detect stays the default.