colpali-v1.2

by vidore

Late-interaction visual document retrieval: search PDFs as images, not extracted text

133Kdl/month

112likes

3Bparams

HuggingFace Run on your data

Identifiers

Model ID

vidore/colpali-v1.2

Feature URI

mixpeek://image_extractor@v1/vidore_colpali_v1

Overview

ColPali retrieves documents by embedding the page image directly, skipping OCR and layout parsing entirely. Instead of one vector per page it produces a grid of patch embeddings and scores a query against all of them with ColBERT-style late interaction, so a match can come from a chart, a table, a diagram or a stamp that text extraction would have flattened or never seen.

This matters because the standard document pipeline, OCR then chunk then embed the text, silently loses exactly the content people search for in technical and financial documents. ColPali treats the page as what it is, a picture, and lets the retrieval model decide what is salient.

The cost is index size: many vectors per page rather than one. That trade is the central design decision in visual document retrieval.

Architecture

PaliGemma-3B vision-language backbone with a LoRA adapter, producing multi-vector page embeddings. Query-document scoring uses ColBERT-style late interaction (MaxSim over patch embeddings) rather than a single dot product, which is what preserves region-level detail.

Mixpeek SDK Integration

import { Mixpeek } from "mixpeek";

const mx = new Mixpeek({ apiKey: "API_KEY" });

// Managed: create a collection over a bucket; Mixpeek runs this model's extractor
const collection = await mx.collections.create({
  namespace_id: "my-namespace",
  collection_name: "my-collection",
  source: { type: "bucket", bucket_ids: ["bkt_your_bucket"] },
  feature_extractor: {
    feature_extractor_name: "document",
    version: "v1",
    parameters: { model_id: "vidore/colpali-v1.2" },
  },
});