tipsv2-so400m14

by google

Contrastive image-text encoder that also returns per-patch spatial features

236Kdl/month

860Mparams

HuggingFace Run it on your own data, free

Identifiers

Model ID

google/tipsv2-so400m14

Feature URI

mixpeek://image_extractor@v1/google_siglip2_so400m_v1

Overview

TIPSv2 stands for Text-Image Pre-training with Spatial awareness. It is a contrastive vision-language family in the CLIP and SigLIP lineage, with one property that separates it for retrieval work: a forward pass returns both a global image embedding and a grid of per-patch features aligned to the same text space.

That matters because a single vector per image forces a choice. Pool everything into one embedding and you can search a million images cheaply, but you lose where in the frame the match happened. TIPSv2 gives you the pooled CLS token at 1152 dimensions for recall, and 1024 patch tokens at the same width when you need to know which region answered the query. Google DeepMind ships DPT head variants on top of the same backbones for zero-shot segmentation.

The SO400m/14 checkpoint carries 412M vision parameters and 448M text parameters. Smaller B/14 and L/14 checkpoints and a larger g/14 exist if this size does not fit your latency budget.

Architecture

Vision transformer with patch size 14 at 448x448 input, trained with a contrastive image-text objective. Exposes encode_image returning a CLS token of width 1152 alongside 1024 patch tokens of the same width, plus a text tower of 448M parameters projecting into the shared space.

Mixpeek SDK Integration

import { Mixpeek } from "mixpeek";

const mx = new Mixpeek({ apiKey: "API_KEY" });

// Managed: create a collection over a bucket; Mixpeek runs this model's extractor
const collection = await mx.collections.create({
  namespace_id: "my-namespace",
  collection_name: "my-collection",
  source: { type: "bucket", bucket_ids: ["bkt_your_bucket"] },
  feature_extractor: {
    feature_extractor_name: "image_embedding",
    version: "v1",
    parameters: { model_id: "google/tipsv2-so400m14" },
  },
});