jina-clip-v2

Name: jina-clip-v2
Author: jinaai

by jinaai

Text and images in one space across 89 languages, truncatable from 1024 dimensions down to 64

73Kdl/month

343likes

865Mparams

HuggingFace Run it on your own data, free

Identifiers

Model ID

jinaai/jina-clip-v2

Feature URI

mixpeek://image_extractor@v1/jina_clip_v2

Deploy jina-clip-v2

Single-tenant

Mixpeek has no managed extractor for this model. On a single-tenant deployment you upload the weights and a custom plugin serves them next to the rest of your pipeline.

Read the customer-hosted guide Talk to us about single-tenant

Overview

A CLIP model that only speaks English quietly breaks any catalogue whose queries do not. jina-clip-v2 puts text and images in a shared space with a multilingual text tower covering 89 languages, so a Spanish query reaches an image captioned in English and a Japanese query reaches the same image. The text side also works as a standalone long-context dense retriever at up to 8,192 tokens, which means one model covers text-to-text, text-to-image and image-to-image without a second encoder.

The Matryoshka training is the part that shows up on your storage bill. Output dimensions truncate from 1,024 down to 64 with graceful degradation, so you can index at 1,024 for quality or at 256 for a quarter of the vector storage and decide per collection rather than per model. Note the licence: CC BY-NC 4.0 is non-commercial, and commercial use goes through Jina's API or one of the cloud marketplace listings.

Architecture

A dual-tower CLIP: a Jina-XLM-RoBERTa text encoder (561M parameters, 24 layers, 8,192-token input, mean pooling, FlashAttention2) and an EVA02-L vision encoder (304M parameters, 24 layers, 512x512 input at patch size 14, CLS pooling, xFormers), trained jointly so the two towers share one space. Matryoshka representation learning makes any prefix of the 1,024-dimensional output a usable vector down to 64.

Mixpeek SDK Integration

# Pick the truncation dimension ONCE per collection. Changing it later means
# reindexing, because a 256-d vector cannot be compared with a 1024-d one.
from transformers import AutoModel
from mixpeek import Mixpeek

model = AutoModel.from_pretrained("jinaai/jina-clip-v2", trust_remote_code=True)
mx = Mixpeek(api_key="API_KEY")

DIM = 512  # anywhere from 64 to 1024
vec = model.encode_image([image_url], truncate_dim=DIM)[0].tolist()

mx.namespaces.documents.upsert(
    namespace_id="ns_your_namespace",
    documents=[{
        "document_id": "sku-88213",
        "payload": {"source_key": "catalog/88213.jpg", "locale": "es-MX"},
        "vectors": {"image_embedding": vec},
    }],
)

Capabilities

Cross-modal retrieval across 89 languages in one shared space
Long-context text embedding at up to 8,192 tokens
Matryoshka output dimensions from 1,024 down to 64
Text-to-text, text-to-image and image-to-image from a single model
ONNX and bfloat16 paths for cheaper inference

Use Cases on Mixpeek

Multilingual product catalogues where queries and captions differ in language

Media libraries searched by description in whatever language the user types

Mixed corpora of documents and images behind one retrieval stage

Storage-constrained indexes that need a smaller vector without a smaller model

Frequently Asked Questions

Can I use this commercially?

Not under the weights licence. CC BY-NC 4.0 covers non-commercial use only, so a production catalogue needs Jina's hosted API or one of the AWS, Azure and Google Cloud marketplace listings. Check this before you build an index on it, because a licence problem discovered after ingestion means re-embedding everything.

What do I lose by truncating to 256 dimensions?

Less than the ratio suggests, which is the point of Matryoshka training: the early dimensions carry the most signal by construction. Measure recall at your own k on a sample before committing, since the acceptable loss depends on how crowded your corpus is. A catalogue of near-identical products will feel the truncation sooner than a diverse media archive.

Does 89 languages mean the quality is the same in all of them?

No. Coverage and quality are different claims. The model card reports up to 4% improvement over nllb-clip-large-siglip on multilingual image retrieval, which is an average across languages, and high-resource languages sit above that average. Evaluate on the languages your users type.