Served on MixpeekEmbeddingRecommendedVerified 2026-11-05
ColQwen2 v1.0 (merged)
vidore/colqwen2-v1.0-merged
ColQwen2 v1.0 (merged) is a image, document and text embedding model (Apache-2.0). Served on Mixpeek, scaling to zero when idle.
A Ray Serve app in Mixpeek's inference registry hosts it. It starts at zero replicas and wakes on first use.
New opens the Designer in Studio. For an existing namespace, open it in Studio and use its Designer tab.
No demo for ColQwen2 v1.0 (merged) yet
A video, image, slideshow or playground goes here once we record one on our own data.
What it is good and bad at
- Best at
- Text search over document page images (PDF pages, slides, infographics, tables) by late interaction, with no OCR step.
- Worst at
- Video frames and product photos: committed eval nDCG@10 0.133 on MSR-VTT keyframes and 0.170 on product images, below single-vector SigLIP. About 750 x 128-d vectors per page make storage and cold reads expensive.
- Task
- Embedding
- Input
- Image, Document, Text
- Output
- Multi-vector
- Use cases
- Document search, OCR
- Why we recommend it
- Served late-interaction page retriever with a committed ViDoRe evaluation.
Templates that use it
No template uses this model yet. Browse templates.
Input and output
Input
- image
- max_visual_tokens
- 768
- patch
- 28 px merged patches
- max_pixels
- 602112
- text
- query text, one vector per token
Output
- vector
- dims
- 128
- distance
- cosine
- type
- multi_dense
How Mixpeek wires it
- Extractors
colqwen_extractor@v1fixed
- Feature URIs
mixpeek://colqwen_extractor@v1/colqwen2_v1
- Vector indexes
colqwen_extractor_v1_tokens- Retriever stages
- maxsim_rerank, feature_search
- Combines with
- PDF pages or frames -> ColQwen2 patch matrices -> maxsim_rerank over a SigLIP or E5 first stage
How it runs
- Status
- Served on Mixpeek Ray Serve app in the inference registry; colqwen_extractor batch calls it (ColQwen2EmbeddingBatchService, a ServeProxyProcessor). The embedding registry marks it PREVIEW.
- Serve apps
vidore__colqwen2- Runtime
- transformers Qwen2VLModel + Qwen2VLImageProcessor + AutoTokenizer, 1_Dense projection to 128 (ColQwen2Encoder, engine code); no colpali-engine dependency
- Weights
- safetensors, no remote code
- Parameters
- 2,209 M
- Size on disk
- 4,419 MB
- Preprocessing
- RGB, resized by the Qwen2-VL processor to at most 768 visual tokens; bf16 (or fp16) on CUDA, fp32 on CPU.
- Postprocessing
- Project every image patch or query token to 128 dims and L2-normalize; one matrix per page, scored by MaxSim.
- Serving
- both, batch 8
- Hardware
- Runs on CPU (slower without a GPU)
Scale to zero
- Minimum replicas
- 0
- Maximum replicas
- 1
- Idle timeout
- 300 s
- Memory
- 12 GB
- CPUs
- 4
- GPUs
- 0
- Cold start, node warm
- Not recorded
- Cold start, new node
- Not recorded
Measured cost
Unmeasured
No committed per-item throughput, latency, memory and cost measurement of the Serve app.
Our evaluations
| Dataset and metric | Score | Baseline | Date |
|---|---|---|---|
ViDoRe v1 + v2 document pages, 1,500 English queries (93,992-item mixed corpus) nDCG@10, exact f32 MaxSim (vidore/colqwen2-v1.0 via colpali-engine, bf16 on L4) (k=10) research/goal-specs/mvs-scale-from-zero/RESULTS-MULTIMODAL-LI.md | 0.696 | SigLIP base alone, top 10: 0.135 | 2026-10-15 |
MSR-VTT 1K-A captions over scene-cut keyframes (1,000 queries) nDCG@10, exact f32 MaxSim (k=10) research/goal-specs/mvs-scale-from-zero/RESULTS-MULTIMODAL-LI.md | 0.133 | SigLIP base alone, top 10: 0.14 | 2026-10-15 |
Marqo/KAGL product images, 1,000 product-title queries nDCG@10, exact f32 MaxSim (k=10) research/goal-specs/mvs-scale-from-zero/RESULTS-MULTIMODAL-LI.md | 0.17 | SigLIP base alone, top 10: 0.293 | 2026-10-15 |
Questions
- What is ColQwen2 v1.0 (merged) best at?
- Text search over document page images (PDF pages, slides, infographics, tables) by late interaction, with no OCR step.
- What are the limits of ColQwen2 v1.0 (merged)?
- Video frames and product photos: committed eval nDCG@10 0.133 on MSR-VTT keyframes and 0.170 on product images, below single-vector SigLIP. About 750 x 128-d vectors per page make storage and cold reads expensive.
- Can ColQwen2 v1.0 (merged) be used commercially?
- Yes. Apache-2.0 on the repo; the base Qwen2-VL-2B is Apache-2.0.
- How does ColQwen2 v1.0 (merged) run on Mixpeek?
- Ray Serve app in the inference registry; colqwen_extractor batch calls it (ColQwen2EmbeddingBatchService, a ServeProxyProcessor). The embedding registry marks it PREVIEW.