Youtu-Parsing-Omni
by tencent
Tencent's 5B parser that turns a document page, image, chart, audio clip or video into one structured JSON, with boxes, timestamps, OCR and transcripts
tencent/Youtu-Parsing-OmniDeploy Youtu-Parsing-Omni
Single-tenantMixpeek has no managed extractor for this model. On a single-tenant deployment you upload the weights and a custom plugin serves them next to the rest of your pipeline.
Overview
Youtu-Parsing-Omni is a 5B-parameter model from Tencent Youtu Lab that reads one input (a document page, a photo, a chart or flowchart, a geometry figure, an audio clip or a video with its soundtrack) and writes one JSON object describing it. A task prompt picks the output: layout elements with bounding boxes and reading order for a document, entities and captions for a photo, timed segments with speakers and speech transcripts for audio, and timed segments with actions, on-screen text and speech for video. It was released in October 2026 with a technical report.
The model card reports 96.96 Overall on OmniDocBench v1.6, the highest in its own comparison table, and a 75.08 average on OmniParsingBench, behind Gemini-3-Pro at 77.44 and ahead of the other open-weight models it lists. These are Tencent's numbers. The license is Tencent's own: it follows Apache-style terms but states the model is not intended for use within the European Union.
For retrieval, the value is that every modality comes back in the same envelope. A video's segments carry start and end timestamps, an event description and the speech in that span, so each segment can be stored as one searchable document that points back to a time range.
Architecture
One shared encoder (Youtu-Omni-Encoder, initialized from a pre-trained text language model) replaces separate vision and audio towers. Thin modality stems map pixels and log-mel frames to tokens; a bidirectional Transformer contextualizes images, audio chunks and video under one (t, h, w) positional encoding, with fusion layers that let a frame attend to the audio of the same instant. Projectors feed a Youtu-LLM decoder (hidden size 2560 per config.json) that emits OmniSchema JSON. Post-trained with SFT, schema-routed RLVR and an on-policy distillation method the card calls OSAD. About 5B parameters in BF16. Loads with trust_remote_code; served through a vLLM plugin (vllm-plugin-vita-omni) or Transformers 5.10.2.
Mixpeek SDK Integration
# Youtu-Parsing-Omni runs on your side. Serve it as the model card shows:
# MODEL=tencent/Youtu-Parsing-Omni DATA_ROOT=/path/to/media GPU=0 bash scripts/vllm.sh
# then parse a recording with the card's example client:
# python examples/infer_vllm.py --task textrich_video --media /path/to/media/lecture.mp4
# The text-rich video schema returns segments with start_timestamp,
# end_timestamp, visual.event_description and audio.asr_text.
import json, requests
parsed = json.loads(model_json) # the model's fenced json block, parsed
for seg in parsed["segments"]:
text = " ".join(filter(None, [
seg["visual"].get("event_description"),
seg.get("audio", {}).get("asr_text"),
]))
# One object per segment. A collection with features: ["text_search"]
# embeds block_text; list source_key, start, end and speaker in its
# field_passthrough so a hit can jump to the right moment.
requests.post(
"https://api.mixpeek.com/v1/buckets/bkt_your_bucket/objects",
headers={"Authorization": "Bearer API_KEY", "X-Namespace": "ns_your_namespace"},
json={
"key_prefix": "/lectures/2026-10",
"blobs": [{"property": "block_text", "type": "text", "data": text}],
"source_key": "lectures/2026-10/lecture.mp4",
"start": seg["start_timestamp"], "end": seg["end_timestamp"],
"speaker": seg.get("audio", {}).get("speaker_id"),
},
)Capabilities
- Eight task prompts: document, natural image, chart, flowchart, geometry figure, audio, natural video and text-rich video
- Document parsing with bounding boxes, reading order, LaTeX formulas, OTSL tables, Markdown charts and Mermaid flowcharts
- Audio parsing into timed segments with speaker, ASR transcript and acoustic events
- Video parsing that reads frames and the audio track together, with camera motion and actions per segment
- One JSON envelope for every input, with a global_description on every output
- vLLM plugin and evaluation scripts in Tencent's youtu-parsing repository
Use Cases on Mixpeek
Benchmarks
| Dataset | Metric | Score | Source |
|---|---|---|---|
| OmniDocBench v1.6 | Overall | 96.96 | Model card: tencent/Youtu-Parsing-Omni (self-reported) |
| OmniDocBench v1.6 | Table TEDS | 96.79 | Model card (self-reported) |
| OmniParsingBench | Average | 75.08 | Model card (self-reported; Gemini-3-Pro 77.44) |
| OmniParsingBench | Audio | 78.08 | Model card (self-reported) |
| OmniParsingBench | Text-rich video | 75.62 | Model card (self-reported) |
| ChemOCR | [email protected] | 54.2 | Model card (self-reported) |
Performance
The card's serving settings are a 65,536-token context, one sequence at a time, BF16 and 0.70 GPU memory utilization, with greedy decoding. It publishes no latency or throughput figures, and we have not measured it. vLLM serving and Transformers inference pin different transformers versions (5.2.0 and 5.10.2), so the card installs them in separate environments.
Common Pipeline Companions
The text embedder behind Mixpeek's text_search feature, for the parsed segments and elements
A document-only OCR model with labeled boxes, under Apache 2.0 with no territorial limit
A dedicated diarizer to compare against the speaker ids in the audio output
Frequently Asked Questions
What is Youtu-Parsing-Omni?
A 5B-parameter parsing model from Tencent Youtu Lab. It takes a document page, image, chart, geometry figure, audio clip or video and returns one structured JSON object with layout elements, text, tables, bounding boxes, timestamps, transcripts and captions, depending on the task prompt.
Can Youtu-Parsing-Omni parse video and audio?
Yes. The audio task returns timed segments with speaker ids, ASR transcripts and acoustic events. The two video tasks read frames and the audio track together and return segments with actions, on-screen text, camera motion and speech, plus a Markdown report for text-heavy videos.
How does Youtu-Parsing-Omni score on document parsing?
The model card reports 96.96 Overall on OmniDocBench v1.6, the top score in its own table, with TeleOCR at 96.91 and PaddleOCR-VL-1.6 at 96.34. Tencent ran these evaluations and has released its OmniDocBench evaluation code.
Can I use Youtu-Parsing-Omni in the EU?
The license says the model is not intended for use within the European Union, and that clause prevails over the rest of the terms. Read the LICENSE file in the repository before deploying it.
How do I search what Youtu-Parsing-Omni extracts?
Run the model, then upload each element or video segment to a Mixpeek bucket as a text object with its page, bounding box or time range as metadata. A collection with the text_search feature embeds the text and a retriever searches it, returning the region or time range that matched.
Specification
Research Paper
Youtu-Parsing-Omni: One Encoder, One Schema, Every Modality (technical report)
arxiv.orgBuild a pipeline with Youtu-Parsing-Omni
Add this model to a processing pipeline alongside other extractors. Combine with retrieval stages for end-to-end search.
Run it on your own data, free