clef-omni
by Cloudflare
Cloudflare's 30B-A3B decision model that hears as well as sees: ask typed questions about text, images, audio or video and get a probability for every answer
Cloudflare/clef-omniDeploy clef-omni
Single-tenantMixpeek has no managed extractor for this model. On a single-tenant deployment you upload the weights and a custom plugin serves them next to the rest of your pipeline.
Overview
Clef-Omni turns a situation and a set of typed questions into decisions, like Clef, and adds audio. You pass the state as text or JSON plus any images, audio clips or videos, ask yes/no, pick-one or score questions, and it returns a probability for every allowed answer of every question in one forward pass, with no generated text to parse. Cloudflare released it on 9 October 2026 under Apache-2.0, post-trained from Qwen3-Omni-30B-A3B-Instruct, a mixture-of-experts model with about 3B parameters active per token.
Video files are sampled at 2 frames per second, and their soundtracks are heard alongside the frames, so one call can answer whether a fan is running and whether it sounds normal. On Cloudflare's own Decision Index 0.2.1 run it scores 94.8 macro-F1 on BANKING77 and 97.7 on CLINC150 with out-of-scope queries, slightly ahead of Clef, and 47.4 on GPQA Diamond, well behind a general model.
The scores are self-reported, and the backbone needs about 64 GB of GPU memory in bfloat16. It is also served on Workers AI as @cf/cloudflare/clef-omni.
Architecture
The thinker of Qwen3-Omni-30B-A3B-Instruct, with its vision and audio encoders, reads the encoded record: the state, images, 16 kHz audio and video frames sampled at 2 fps with their soundtrack. The base model's speech-output weights are shipped but not loaded. A small transformer head reads the backbone's final hidden states, routes evidence from the state to each question, and scores all options of all questions jointly, producing one logit per allowed option; a softmax per question gives the probabilities. Records are bounded at 64,000 tokens by default.
Mixpeek SDK Integration
# Ask Clef-Omni typed questions about each video (frames and soundtrack), then store
# the answers in metadata so Mixpeek search can filter or group by them.
import requests
for clip in clips: # [{"url": "s3://...", "answers": {"has_speech": True, "setting": "outdoor"}}]
requests.post(
"https://api.mixpeek.com/v1/buckets/bkt_your_bucket/objects",
headers={"Authorization": "Bearer API_KEY", "X-Namespace": "ns_your_namespace"},
json={
"blobs": [{"property": "video", "type": "video", "url": clip["url"]}],
"metadata": clip["answers"],
},
)Capabilities
- Typed questions: true/false, pick one of named options, or score on an ordered scale
- Reads text, JSON, images, audio and video, with a video's soundtrack heard alongside its frames
- Returns a probability for every allowed option of every question in one forward pass
- Apache-2.0 open weights, also served on Cloudflare Workers AI
Use Cases on Mixpeek
Benchmarks
| Dataset | Metric | Score | Source |
|---|---|---|---|
| BANKING77 | Macro-F1 | 94.8 | Model card: Cloudflare/clef-omni (self-reported, Decision Index 0.2.1 run; Clef 94.2) |
| CLINC150 + out-of-scope | Macro-F1 | 97.7 | Model card (self-reported; Clef 97.4) |
| MMLU | Accuracy | 92.7 | Model card (self-reported) |
| GPQA Diamond | Accuracy | 47.4 | Model card (self-reported; Jev 78.3) |
Performance
Tested by Cloudflare with torch 2.11 and transformers 5.10.2 on a single H200; the backbone needs about 64 GB of GPU memory in bfloat16. We have not measured it.
Common Pipeline Companions
Frequently Asked Questions
What does Clef-Omni do?
You give it a situation as text or JSON plus images, audio or video, and a set of typed questions, and it returns a probability for every allowed answer to every question in one pass. It does not write free text, so there is nothing to parse.
How is Clef-Omni different from Clef?
Clef-Omni adds audio: it reads audio clips and hears a video's soundtrack alongside its frames. It is built on Qwen3-Omni-30B-A3B, a mixture-of-experts model, where Clef uses the dense Qwen3.8-27B. On the card's Decision Index run each leads on different tasks, for example Clef-Omni on BANKING77 (94.8 against 94.2) and Clef on ToolRet (69.2 against 66.6).
How does Clef-Omni handle video?
Video files are sampled at 2 frames per second, and when every video in the record has a soundtrack, the audio is heard alongside the frames. You can also pass frames yourself as RGB arrays sampled at 2 fps.
What hardware does Clef-Omni need?
Cloudflare tested it on a single H200; the backbone needs about 64 GB of GPU memory in bfloat16. It is also available as a hosted model on Workers AI.
How do I use Clef-Omni answers in Mixpeek search?
Store each answer in the object's metadata and filter or group a retriever by it, as in the example on this page. Mixpeek taxonomies can also assign labels inside the pipeline.
Specification
Research Paper
Clef-omni with full multimodality (Cloudflare blog)
arxiv.orgBuild a pipeline with clef-omni
Add this model to a processing pipeline alongside other extractors. Combine with retrieval stages for end-to-end search.
Run it on your own data, free