ColUltraVBERT
by athrael-soju
A 255M late-interaction retriever that searches PDF pages as images, scoring 86.4 nDCG@5 on ViDoRe V1 under an MIT license.
athrael-soju/ColUltraVBERTDeploy ColUltraVBERT
Single-tenantMixpeek has no managed extractor for this model. On a single-tenant deployment you upload the weights and a custom plugin serves them next to the rest of your pipeline.
Overview
ColUltraVBERT is a visual document retriever released in October 2026 and fine-tuned from UltraVBERT. It embeds each query token and each page-image token as a 768-dimension vector and ranks pages by MaxSim, the ColPali approach, so pages are searched without OCR. Its official MTEB results put it at a mean nDCG@5 of 86.4 on ViDoRe V1 and 54.8 on ViDoRe V2, against 81.4 and 40.7 for ColModernVBERT. Vectors can be cut to 128, 256, 320 or 512 dimensions for a smaller index.
Architecture
UltraVBERT's 768-d token states pass through a linear projection to 768-d retrieval vectors, L2-normalized, with padding dropped. Pages are resized to at most 3,072 px on the long edge and split into up to 25 tiles of 512 px plus a downscaled full-page view, for at most 1,664 image vectors per page. The released weights merge four LoRA fine-tunes trained on 499,704 query-page pairs with Matryoshka losses at five sizes.
Mixpeek SDK Integration
# Embed pages and queries (from the ColUltraVBERT model card).
# Requires sentence-transformers>=6.1 and transformers>=5.21.
from PIL import Image
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("athrael-soju/ColUltraVBERT")
pages = [Image.open("page1.png"), Image.open("page2.png")]
queries = ["What was the 2023 revenue?", "Which table lists regional margins?"]
page_embeddings = model.encode_document(pages)
query_embeddings = model.encode_query(queries)
scores = model.similarity(query_embeddings, page_embeddings) # one row per query, one column per pageCapabilities
- Text-to-page retrieval over PDF page images, with no OCR step
- Multi-vector (late-interaction) scoring with MaxSim
- Matryoshka truncation to 128, 256, 320 or 512 dimensions
- Queries in English, Spanish, German, French, Italian and Portuguese
- Loads through sentence-transformers MultiVectorEncoder
Use Cases on Mixpeek
Benchmarks
| Dataset | Metric | Score | Source |
|---|---|---|---|
| ViDoRe V1 | Mean nDCG@5 | 86.4 | https://huggingface.co/athrael-soju/ColUltraVBERT |
| ViDoRe V2 | Mean nDCG@5 | 54.8 | https://huggingface.co/athrael-soju/ColUltraVBERT |
| ViDoRe V3 Computer Science | nDCG@10 | 66.5 | https://huggingface.co/athrael-soju/ColUltraVBERT |
Performance
We have not measured it. Scores are from the model card and its MTEB submissions.
Common Pipeline Companions
Frequently Asked Questions
What is ColUltraVBERT?
A 255M-parameter retriever that finds the PDF pages answering a question by comparing the question's tokens with the page image's tokens. It needs no OCR and is released under MIT.
How does ColUltraVBERT compare with ColPali-style models?
It uses the same late-interaction scoring. On ViDoRe V1 its mean nDCG@5 is 86.4 against 81.4 for ColModernVBERT, a model of similar size, and on ViDoRe V2 it is 54.8 against 40.7, per the official MTEB results on its card.
How big is a ColUltraVBERT index?
Up to 1,664 image vectors per page at 768 dimensions. Truncating each vector to 128 dimensions cuts storage about sixfold, at some cost in accuracy that the card's Matryoshka training is meant to limit.
Specification
Build a pipeline with ColUltraVBERT
Add this model to a processing pipeline alongside other extractors. Combine with retrieval stages for end-to-end search.
Run it on your own data, free