GTE-ModernColBERT-v1
by lightonai
Late interaction retrieval model with record-breaking long-context performance
lightonai/GTE-ModernColBERT-v1mixpeek://text_extractor@v1/lighton_gte_moderncolbert_v1Overview
GTE-ModernColBERT-v1 is a ColBERT-style late interaction retrieval model built on the ModernBERT architecture. Instead of compressing an entire document into a single vector, it produces 128-dimensional embeddings for every token, then scores query-document pairs using MaxSim: for each query token, find the best-matching document token and sum the scores. This token-level matching preserves fine-grained detail that single-vector models lose.
The model's standout capability is long-context retrieval. On the LongEmbed benchmark (documents up to 32K tokens), it scores 88.39 mean: roughly 10 points above the previous state of the art. It also outperforms ColBERT-small on BEIR while supporting documents up to 32K tokens natively. Trained in just 15K steps on MS MARCO using LightOn's PyLate library, it demonstrated that the ModernBERT + ColBERT recipe produces competitive results with minimal training compute.
Architecture
ModernBERT encoder (from Alibaba-NLP/gte-modernbert-base) with a linear projection layer (768 → 128 dimensions, no bias, no activation). Produces per-token 128-dim embeddings. Default query length 32 tokens, document length up to 32K tokens. Scoring via MaxSim operator. Trained with knowledge distillation on MS MARCO using PyLate.
Mixpeek SDK Integration
// No extractor parameter takes a Hugging Face model id (checked against
// GET /v1/discovery/extractors, which returns 13), so GTE-ModernColBERT-v1 runs
// on your side and the output is upserted through POST
// /v1/namespaces/{namespace_id}/documents/upsert. On Enterprise the other
// path is to upload the weights instead: POST /v1/namespaces/{id}/models
// accepts the huggingface format and a custom plugin loads them.
const res = await fetch(
"https://api.mixpeek.com/v1/namespaces/ns_your_namespace/documents/upsert",
{
method: "POST",
headers: {
Authorization: "Bearer API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
collection_id: "col_your_collection",
documents: [
{
document_id: "asset-00412",
// The vector name has to match a vector index on the collection.
vectors: { "text-embedding": yourVector },
payload: { source_key: "archive/2026/asset-00412" },
},
],
}),
},
);
// Managed alternative, if this exact model is not the requirement:
// text_extractor@v1 runs intfloat/multilingual-e5-large-instruct
// (1024-d) over a bucket, with no inference of your own.Capabilities
- Late interaction retrieval with per-token 128-dim embeddings
- Long-context support up to 32K tokens (tested to 32,768)
- 88.39 mean on LongEmbed benchmark (~10 points above prior SOTA)
- 54.75 NDCG@10 on BEIR: outperforms ColBERT-small
- Apache 2.0 license, reproducible training with PyLate
Use Cases on Mixpeek
Benchmarks
| Dataset | Metric | Score | Source |
|---|---|---|---|
| BEIR (15 datasets) | NDCG@10 | 54.75 | LightOn, 2025: Model Card |
| LongEmbed (32K context) | Mean Score | 88.39 | LightOn, 2025: Blog Post |
| NanoBEIR | NDCG@10 | 67.58 | LightOn, 2025: Model Card |
Performance
Common Pipeline Companions
Explore on Mixpeek
Compare alternatives in this category
Hand-picked tools & platforms compared
Deep-dive technical guide
See how Mixpeek runs models as extractors
Store & search embeddings at scale
Usage-based pricing for pipelines
Compare models, APIs & infrastructure
Specification
Research Paper
LightOn Releases GTE-ModernColBERT, First SOTA Late-Interaction Model Trained on PyLate
arxiv.orgBuild a pipeline with GTE-ModernColBERT-v1
Add this model to a processing pipeline alongside other extractors. Combine with retrieval stages for end-to-end search.
Run it on your own data, free