pe-av-large
by facebook
Joint audio-video-text embeddings from Meta's Perception Encoder family
facebook/pe-av-largemixpeek://audio_extractor@v1/facebook_pe_av_large_v1Deploy pe-av-large
Single-tenantMixpeek has no managed extractor for this model. On a single-tenant deployment you upload the weights and a custom plugin serves them next to the rest of your pipeline.
Overview
PE-AV Large embeds audio, video, synchronized audio-video, and text into one shared retrieval space. It is useful when the same event is expressed through motion, sound, or language, such as a siren, a crowd reaction, a machine failure, or a tennis serve.
On Mixpeek, PE-AV Large gives agents a single evidence channel for audiovisual retrieval. Instead of searching transcripts, frames, and audio fingerprints separately, an agent can retrieve clips where the sound and visual motion jointly match the query, then pass the top results to a reasoning model.
Architecture
Perception Encoder audio-video model with roughly 2.2B parameters. The model aligns raw audio, video frames, audio-video pairs, and text through contrastive training so cross-modal retrieval works across all supported input combinations.
Mixpeek SDK Integration
// No extractor parameter takes a Hugging Face model id (checked against
// GET /v1/discovery/extractors, which returns 13), so pe-av-large runs
// on your side and the output is upserted through POST
// /v1/namespaces/{namespace_id}/documents/upsert. On Enterprise the other
// path is to upload the weights instead: POST /v1/namespaces/{id}/models
// accepts the huggingface format and a custom plugin loads them.
const res = await fetch(
"https://api.mixpeek.com/v1/namespaces/ns_your_namespace/documents/upsert",
{
method: "POST",
headers: {
Authorization: "Bearer API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
collection_id: "col_your_collection",
documents: [
{
document_id: "asset-00412",
// The vector name has to match a vector index on the collection.
vectors: { "audio-embedding": yourVector },
payload: { source_key: "archive/2026/asset-00412" },
},
],
}),
},
);
// Managed alternative, if this exact model is not the requirement:
// audio_fingerprint_extractor@v1 runs laion/clap-htsat-tiny
// (512-d) over a bucket, with no inference of your own.Capabilities
- Text-to-video, text-to-audio, and text-to-audio-video retrieval
- Joint embeddings for synchronized sound and motion
- Useful for clips where audio carries the key signal
- Apache 2.0 license
Use Cases on Mixpeek
Performance
Common Pipeline Companions
Explore on Mixpeek
Compare alternatives in this category
Hand-picked tools & platforms compared
Deep-dive technical guide
See how Mixpeek runs models as extractors
Store & search embeddings at scale
Usage-based pricing for pipelines
Compare models, APIs & infrastructure
Specification
Research Paper
PE Audio Video
arxiv.orgBuild a pipeline with pe-av-large
Add this model to a processing pipeline alongside other extractors. Combine with retrieval stages for end-to-end search.
Run it on your own data, free