NEWVectors or files. Pick a path.Start →
    Models/tencent/WeMM-Video 35B-A3B
    Apache 2.0

    WeMM-Video 35B-A3B

    by tencent

    Tencent's open video model that watches the frames and listens to the soundtrack together: text, images, audio and video in, text out, with 3B of 35B parameters active

    27likes
    35B total (about 3B active)params
    Identifiers
    Model ID
    tencent/WeMM-Video 35B-A3B

    Deploy WeMM-Video 35B-A3B

    Single-tenant

    Mixpeek has no managed extractor for this model. On a single-tenant deployment you upload the weights and a custom plugin serves them next to the rest of your pipeline.

    Overview

    WeMM-Video-35B-A3B is the mixture-of-experts checkpoint in Tencent's WeMM-Video family, released on Hugging Face on November 17, 2026 under the Apache 2.0 license. It takes text, images, audio and video and writes text in English or Chinese. What sets it apart from most open vision-language models is the soundtrack: given a video file, it can read the frames and the original audio together, so it can describe an event, transcribe the speech and reason about what is seen and heard in one pass.

    It has 35B parameters in total with about 3B active per token, and its BF16 weights are 67.39 GiB. Tencent also publishes FP8, AWQ 4-bit, GGUF Q4_K_M and NVFP4 versions, plus dense 9B and 2B siblings in the same family. The technical report (arXiv 2611.26409) describes pretraining on about 1.2 trillion multimodal tokens.

    For a media pipeline, it can turn a clip into a description of what happens and what is said, which you can then index and search as text.

    Architecture

    Separate vision and audio encoders turn frames and sound into tokens, which a mixture-of-experts language model (35B total, about 3B active per token) processes alongside text. Video frames carry timestamps. Post-training used multi-teacher on-policy distillation, an anti-repetition preference step (MrDPO) and specialist reinforcement learning.

    Mixpeek SDK Integration

    # WeMM-Video is not a managed extractor. Serve it with Tencent's patched vLLM
    # (see https://github.com/Tencent/WeMM-Video/blob/main/docs/serving.md), which
    # the card runs across two GPUs:
    #   vllm serve tencent/WeMM-Video-35BA3B --trust-remote-code --dtype bfloat16 \
    #     --max-model-len 8192 --tensor-parallel-size 2
    # Then describe each clip, frames and sound together, and store the text.
    import requests
    from openai import OpenAI
    
    llm = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
    
    def describe(clip_url):
        r = llm.chat.completions.create(
            model="tencent/WeMM-Video-35BA3B",
            messages=[{"role": "user", "content": [
                {"type": "video_url", "video_url": {"url": clip_url, "use_audio": True}},
                {"type": "text", "text": "Describe what happens and summarize what is said."},
            ]}],
            max_tokens=200,
        )
        return r.choices[0].message.content
    
    requests.post(
        "https://api.mixpeek.com/v1/namespaces/ns_your_namespace/documents/upsert",
        headers={"Authorization": "Bearer API_KEY"},
        json={"collection_id": "col_clip_descriptions",
              "documents": [{"document_id": "clip-042",
                             "payload": {"description": describe("https://example.com/clip-042.mp4"),
                                         "video": "clip-042.mp4"}}]},
    )

    Capabilities

    • Video with its original soundtrack, silent video, audio, images and text as input
    • Text output in English or Chinese, with an instruct mode and a thinking mode
    • About 3B active parameters per token out of 35B
    • BF16 weights of 67.39 GiB, with FP8, AWQ, GGUF and NVFP4 versions
    • Apache 2.0 license

    Use Cases on Mixpeek

    Describing what happens and what is said in each clip of a video library, so it can be searched as text
    Transcribing speech in footage while also describing the scene it was spoken in
    Answering questions about a clip that need both the picture and the sound
    Writing summaries of recorded meetings, lectures or broadcasts from the video file itself

    Performance

    Input SizeText, images, audio and video; the card's vLLM command sets an 8,192-token context and samples video at 2 fps, 4 to 16 frames per clip
    GPU LatencyInput dependent
    GPU ThroughputBatch dependent
    GPU MemoryModel dependent

    Tencent's serving recipes use tensor parallelism across two GPUs and say to size memory for the full expert weights, the media inputs and the generation cache. Tencent reports its benchmark results as charts on the model card. We have not measured it.

    Frequently Asked Questions

    What is WeMM-Video?

    A family of open video understanding models from Tencent, released November 17, 2026 under Apache 2.0. The models take text, images, audio and video, including a video's own soundtrack, and write text in English or Chinese. The 35B-A3B checkpoint is the mixture-of-experts model, with about 3B of its 35B parameters active per token; 9B and 2B versions are also published.

    Can WeMM-Video understand the audio in a video?

    Yes. Given a video file with sound, it reads the frames and the original soundtrack together, so it can describe an event and transcribe or summarize the speech in one answer. In the serving API, set use_audio to true on the video_url input; set it to false to send the frames alone.

    What hardware does WeMM-Video-35B-A3B need?

    The BF16 weights are 67.39 GiB, and Tencent's vLLM and SGLang commands split them across two GPUs with tensor parallelism. Only about 3B parameters are active per token, which keeps generation fast, but all 35B must fit in GPU memory along with the media inputs and cache. The FP8, AWQ 4-bit, GGUF and NVFP4 versions need less memory, and the dense 9B and 2B models less again.

    Can I use WeMM-Video commercially?

    The model card releases it under the Apache License 2.0, which allows commercial use. The card also points to a collection license and third-party attribution file; read both before you ship it.

    How do I use WeMM-Video with Mixpeek?

    Serve it with Tencent's patched vLLM, send each clip with its soundtrack, and upsert the description it writes into a Mixpeek collection. A retriever then searches those descriptions by meaning, next to any scene embeddings, faces or transcripts in the same namespace.

    Specification

    Organizationtencent
    Retriever-
    Parameters35B total (about 3B active)
    LicenseApache 2.0
    Downloads/moN/A
    Likes27

    Research Paper

    WeMM-Video: Pushing the Frontiers of Open-Source Omni-Modal Video Understanding

    arxiv.org

    Build a pipeline with WeMM-Video 35B-A3B

    Add this model to a processing pipeline alongside other extractors. Combine with retrieval stages for end-to-end search.

    Run it on your own data, free