colqwen-omni-v0.1

Name: colqwen-omni-v0.1
Author: vidore

by vidore

Omnimodal ColBERT retrieval for documents, audio, and video search

3Kdl/month

95likes

~3Bparams

HuggingFace Run it on your own data, free

Identifiers

Model ID

vidore/colqwen-omni-v0.1

Feature URI

mixpeek://image_extractor@v1/vidore_colqwen_omni_v1

Overview

ColQwen Omni extends the ColPali paradigm to all modalities (documents, audio, and video) using ColBERT-style multi-vector representations built on Qwen2.5-Omni-3B. Unlike dense single-vector models, multi-vector retrieval preserves fine-grained token-level matching, delivering higher precision on complex queries.

On Mixpeek, ColQwen Omni powers late-interaction retrieval across document pages, audio recordings, and video content. Its zero-shot audio retrieval (no audio training data needed) makes it especially useful for indexing podcasts, meetings, and lecture recordings alongside visual content.

Architecture

Qwen2.5-Omni-3B-Instruct fine-tuned for ColBERT-style multi-vector output. Dynamic image resolution (max 1024 patches). Audio/video towers frozen during training: audio retrieval is zero-shot. Trained with colpali-engine 0.3.11 on 127K query-page pairs.

Mixpeek SDK Integration

// No extractor parameter takes a Hugging Face model id (checked against
// GET /v1/discovery/extractors, which returns 13), so colqwen-omni-v0.1 runs
// on your side and the output is upserted through POST
// /v1/namespaces/{namespace_id}/documents/upsert. On Enterprise the other
// path is to upload the weights instead: POST /v1/namespaces/{id}/models
// accepts the huggingface format and a custom plugin loads them.
const res = await fetch(
  "https://api.mixpeek.com/v1/namespaces/ns_your_namespace/documents/upsert",
  {
    method: "POST",
    headers: {
      Authorization: "Bearer API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      collection_id: "col_your_collection",
      documents: [
        {
          document_id: "asset-00412",
          // The vector name has to match a vector index on the collection.
          vectors: { "multimodal-embedding": yourVector },
          payload: { source_key: "archive/2026/asset-00412" },
        },
      ],
    }),
  },
);

// Managed alternative, if this exact model is not the requirement:
// multimodal_extractor@v2 runs google/gemini-embedding-2
// (3072-d) over a bucket, with no inference of your own.