videoprism-base-f16r288
by google
A frozen video encoder that beat every prior foundation model on 31 of 33 benchmarks without fine-tuning
google/videoprism-base-f16r288mixpeek://video_extractor@v1/google_videoprism_base_v1Deploy videoprism-base-f16r288
Single-tenantMixpeek has no managed extractor for this model. On a single-tenant deployment you upload the weights and a custom plugin serves them next to the rest of your pipeline.
Overview
Video retrieval usually starts with a choice nobody wants to make: embed single frames and lose motion, or run a heavy video-language model on every clip. VideoPrism is Google DeepMind's answer to the first half of that. It takes video frames and returns spatiotemporal embeddings that feed a classifier, a retriever or an LLM, and it is designed to be used frozen. On 33 public video understanding benchmarks across four task categories, one frozen checkpoint outperformed the previous best foundation models on 31, with no fine-tuning on the target datasets.
The base checkpoint is small enough to be practical at corpus scale: 114M parameters and a 458MB file. It pairs with a text encoder in the CLIP fashion when you want text-to-video retrieval, and stands alone when you want clustering, classification or temporal localisation over your own labels.
Architecture
A ViViT-style factorised video encoder: a Vision Transformer image encoder initialised from CoCa followed by four temporal-attention Transformer layers. Input is (num_frames, 288, 288, 3) and output is (num_frames * 16 * 16, feature_channels), reshapeable to (num_frames, 16, 16, C) for spatiotemporal features. Trained at 16 frames for the base variant, and the temporal positional embeddings interpolate so other frame counts work.
How it runs
Mixpeek SDK Integration
# Frames in, spatiotemporal embeddings out. Decide your pooling before you
# index: per-clip keeps the moment, per-video loses it.
import numpy as np
from videoprism import models as vp
from mixpeek import Mixpeek
flax_model = vp.get_model("videoprism_public_v1_base")
state = vp.load_pretrained_weights("videoprism_public_v1_base")
mx = Mixpeek(api_key="API_KEY")
# frames: (16, 288, 288, 3), values in [0, 1]
embeddings, _ = vp.forward_fn(flax_model)(state, frames[None, ...])
clip_vector = np.asarray(embeddings[0]).mean(axis=0).tolist()
mx.documents.upsert(
collection_id="col_your_collection",
documents=[{
"document_id": "match-0412-clip-00:14:32",
"payload": {
"source_key": "matches/0412.mp4",
"start_sec": 872,
"end_sec": 880,
},
"vectors": {"video_embedding": clip_vector},
}],
)Capabilities
- Spatiotemporal embeddings from raw frames, usable frozen
- Text-to-video retrieval when paired with the matching text encoder
- Temporal and spatiotemporal localisation with a box proposal on top
- Video classification by training only a lightweight head
- 114M parameters, small enough to run across a large archive
Use Cases on Mixpeek
Frequently Asked Questions
How is this different from embedding one frame per second with CLIP?
Frame embeddings have no notion of order, so a jump and a landing look the same as a landing and a jump. VideoPrism attends across time, which is what makes actions and short events separable. Frame embeddings remain fine for questions about what appears in a video and weak for questions about what happens in it.
Do I index one vector per video or one per clip?
Per clip, almost always. A single vector for a 40-minute recording averages everything into a blur and can only tell you the file is relevant. Segment on scene boundaries or a fixed window, index each segment with its timestamps, and a search returns the moment.
Can I search it with text out of the box?
Not with this checkpoint alone. The base and large video encoders produce video embeddings only. The LvT variants add a text tower trained in the same space, which is what gives you text-to-video retrieval by cosine similarity.
Does Mixpeek run this model?
Not as a managed extractor. Run it over your clips and upsert the vectors with their timestamps, or use a managed video extractor if you would rather not operate an encoder.
Explore on Mixpeek
Compare alternatives in this category
Hand-picked tools & platforms compared
Deep-dive technical guide
See how Mixpeek runs models as extractors
Store & search embeddings at scale
Usage-based pricing for pipelines
Compare models, APIs & infrastructure
Specification
Research Paper
VideoPrism: A Foundational Visual Encoder for Video Understanding
arxiv.orgBuild a pipeline with videoprism-base-f16r288
Add this model to a processing pipeline alongside other extractors. Combine with retrieval stages for end-to-end search.
Run it on your own data, free