WeMM-Video 35B-A3B
by tencent
Tencent's open video model that watches the frames and listens to the soundtrack together: text, images, audio and video in, text out, with 3B of 35B parameters active
tencent/WeMM-Video 35B-A3BDeploy WeMM-Video 35B-A3B
Single-tenantMixpeek has no managed extractor for this model. On a single-tenant deployment you upload the weights and a custom plugin serves them next to the rest of your pipeline.
Overview
WeMM-Video-35B-A3B is the mixture-of-experts checkpoint in Tencent's WeMM-Video family, released on Hugging Face on November 17, 2026 under the Apache 2.0 license. It takes text, images, audio and video and writes text in English or Chinese. What sets it apart from most open vision-language models is the soundtrack: given a video file, it can read the frames and the original audio together, so it can describe an event, transcribe the speech and reason about what is seen and heard in one pass.
It has 35B parameters in total with about 3B active per token, and its BF16 weights are 67.39 GiB. Tencent also publishes FP8, AWQ 4-bit, GGUF Q4_K_M and NVFP4 versions, plus dense 9B and 2B siblings in the same family. The technical report (arXiv 2611.26409) describes pretraining on about 1.2 trillion multimodal tokens.
For a media pipeline, it can turn a clip into a description of what happens and what is said, which you can then index and search as text.
Architecture
Separate vision and audio encoders turn frames and sound into tokens, which a mixture-of-experts language model (35B total, about 3B active per token) processes alongside text. Video frames carry timestamps. Post-training used multi-teacher on-policy distillation, an anti-repetition preference step (MrDPO) and specialist reinforcement learning.
Mixpeek SDK Integration
# WeMM-Video is not a managed extractor. Serve it with Tencent's patched vLLM
# (see https://github.com/Tencent/WeMM-Video/blob/main/docs/serving.md), which
# the card runs across two GPUs:
# vllm serve tencent/WeMM-Video-35BA3B --trust-remote-code --dtype bfloat16 \
# --max-model-len 8192 --tensor-parallel-size 2
# Then describe each clip, frames and sound together, and store the text.
import requests
from openai import OpenAI
llm = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
def describe(clip_url):
r = llm.chat.completions.create(
model="tencent/WeMM-Video-35BA3B",
messages=[{"role": "user", "content": [
{"type": "video_url", "video_url": {"url": clip_url, "use_audio": True}},
{"type": "text", "text": "Describe what happens and summarize what is said."},
]}],
max_tokens=200,
)
return r.choices[0].message.content
requests.post(
"https://api.mixpeek.com/v1/namespaces/ns_your_namespace/documents/upsert",
headers={"Authorization": "Bearer API_KEY"},
json={"collection_id": "col_clip_descriptions",
"documents": [{"document_id": "clip-042",
"payload": {"description": describe("https://example.com/clip-042.mp4"),
"video": "clip-042.mp4"}}]},
)Capabilities
- Video with its original soundtrack, silent video, audio, images and text as input
- Text output in English or Chinese, with an instruct mode and a thinking mode
- About 3B active parameters per token out of 35B
- BF16 weights of 67.39 GiB, with FP8, AWQ, GGUF and NVFP4 versions
- Apache 2.0 license
Use Cases on Mixpeek
Performance
Tencent's serving recipes use tensor parallelism across two GPUs and say to size memory for the full expert weights, the media inputs and the generation cache. Tencent reports its benchmark results as charts on the model card. We have not measured it.
Common Pipeline Companions
Frequently Asked Questions
What is WeMM-Video?
A family of open video understanding models from Tencent, released November 17, 2026 under Apache 2.0. The models take text, images, audio and video, including a video's own soundtrack, and write text in English or Chinese. The 35B-A3B checkpoint is the mixture-of-experts model, with about 3B of its 35B parameters active per token; 9B and 2B versions are also published.
Can WeMM-Video understand the audio in a video?
Yes. Given a video file with sound, it reads the frames and the original soundtrack together, so it can describe an event and transcribe or summarize the speech in one answer. In the serving API, set use_audio to true on the video_url input; set it to false to send the frames alone.
What hardware does WeMM-Video-35B-A3B need?
The BF16 weights are 67.39 GiB, and Tencent's vLLM and SGLang commands split them across two GPUs with tensor parallelism. Only about 3B parameters are active per token, which keeps generation fast, but all 35B must fit in GPU memory along with the media inputs and cache. The FP8, AWQ 4-bit, GGUF and NVFP4 versions need less memory, and the dense 9B and 2B models less again.
Can I use WeMM-Video commercially?
The model card releases it under the Apache License 2.0, which allows commercial use. The card also points to a collection license and third-party attribution file; read both before you ship it.
How do I use WeMM-Video with Mixpeek?
Serve it with Tencent's patched vLLM, send each clip with its soundtrack, and upsert the description it writes into a Mixpeek collection. A retriever then searches those descriptions by meaning, next to any scene embeddings, faces or transcripts in the same namespace.
Specification
Research Paper
WeMM-Video: Pushing the Frontiers of Open-Source Omni-Modal Video Understanding
arxiv.orgBuild a pipeline with WeMM-Video 35B-A3B
Add this model to a processing pipeline alongside other extractors. Combine with retrieval stages for end-to-end search.
Run it on your own data, free