colnomic-embed-multimodal-7b

Name: colnomic-embed-multimodal-7b
Author: nomic-ai

by nomic-ai

Late-interaction multimodal embeddings: SOTA visual document retrieval without OCR

585dl/month

107likes

7Bparams

HuggingFace Run it on your own data, free

Identifiers

Model ID

nomic-ai/colnomic-embed-multimodal-7b

Feature URI

mixpeek://image_extractor@v1/nomic_colnomic_multimodal_7b_v1

Overview

ColNomic Embed Multimodal 7B is Nomic AI's multi-vector late-interaction embedding model that processes text, images, PDFs, and charts without requiring OCR or image captioning as a preprocessing step. Fine-tuned from Qwen2.5-VL-7B-Instruct, it produces multiple token-level embeddings per document instead of a single vector, enabling fine-grained matching between query tokens and document tokens at retrieval time.

The model achieves 62.7 NDCG@5 on Vidore-v2, a 2.8-point improvement over the previous state-of-the-art for visual document retrieval. On Mixpeek, ColNomic powers high-precision document search where the visual layout of PDFs, slides, and charts carries meaning that traditional text-only embeddings miss: tables, diagrams, and mixed text-image pages are all searchable directly from their rendered appearance.

Architecture

Multi-vector late-interaction architecture fine-tuned from Qwen2.5-VL-7B-Instruct. Produces multiple token-level embeddings per document for fine-grained matching. Uses same-source sampling to generate harder in-batch negatives during training. Processes interleaved text and image inputs natively.

Mixpeek SDK Integration

// No extractor parameter takes a Hugging Face model id (checked against
// GET /v1/discovery/extractors, which returns 13), so colnomic-embed-multimodal-7b runs
// on your side and the output is upserted through POST
// /v1/namespaces/{namespace_id}/documents/upsert. On Enterprise the other
// path is to upload the weights instead: POST /v1/namespaces/{id}/models
// accepts the huggingface format and a custom plugin loads them.
const res = await fetch(
  "https://api.mixpeek.com/v1/namespaces/ns_your_namespace/documents/upsert",
  {
    method: "POST",
    headers: {
      Authorization: "Bearer API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      collection_id: "col_your_collection",
      documents: [
        {
          document_id: "asset-00412",
          // The vector name has to match a vector index on the collection.
          vectors: { "image-embedding": yourVector },
          payload: { source_key: "archive/2026/asset-00412" },
        },
      ],
    }),
  },
);

// Managed alternative, if this exact model is not the requirement:
// image_extractor@v1 runs google/siglip-base-patch16-224
// (768-d) over a bucket, with no inference of your own.