pe-av-large

by facebook

Joint audio-video-text embeddings from Meta's Perception Encoder family

4Kdl/month

61likes

2.2Bparams

HuggingFace Use in Pipeline

Identifiers

Model ID

facebook/pe-av-large

Feature URI

mixpeek://audio_extractor@v1/facebook_pe_av_large_v1

Overview

PE-AV Large embeds audio, video, synchronized audio-video, and text into one shared retrieval space. It is useful when the same event is expressed through motion, sound, or language, such as a siren, a crowd reaction, a machine failure, or a tennis serve.

On Mixpeek, PE-AV Large gives agents a single evidence channel for audiovisual retrieval. Instead of searching transcripts, frames, and audio fingerprints separately, an agent can retrieve clips where the sound and visual motion jointly match the query, then pass the top results to a reasoning model.

Architecture

Perception Encoder audio-video model with roughly 2.2B parameters. The model aligns raw audio, video frames, audio-video pairs, and text through contrastive training so cross-modal retrieval works across all supported input combinations.

Mixpeek SDK Integration

import { Mixpeek } from "mixpeek";

const mx = new Mixpeek({ apiKey: "API_KEY" });

await mx.collections.ingest({
  collection_id: "av-memory",
  source: { url: "s3://camera-footage/" },
  feature_extractors: [{
    feature: "audio_embeddings",
    model: "facebook/pe-av-large"
  }]
});