UltraBERT-Embed-150M
by dmis-lab
A 151M-parameter English text and code embedding model with a 32K-token context, 768-dimensional vectors and instruction-following queries
dmis-lab/UltraBERT-Embed-150MDeploy UltraBERT-Embed-150M
Single-tenantMixpeek has no managed extractor for this model. On a single-tenant deployment you upload the weights and a custom plugin serves them next to the rest of your pipeline.
Overview
UltraBERT-Embed-150M is a text embedding model from DMIS Lab, released on 10 October 2026 under Apache 2.0. It turns English text or code into a 768-dimensional vector and reads up to 32,768 tokens, so a long contract, a transcript or a source file can be embedded whole instead of cut into chunks first. Queries take a task instruction in the E5-mistral style (Instruct: {task}\nQuery: ) and documents are embedded raw, so one model covers retrieval, classification, clustering and similarity.
The card reports 67.24 on MTEB(eng, v2) across 41 tasks, which it says is the best of any dense model under 200M parameters as of 11 October 2026, ahead of gte-modernbert-base (66.26). It also reports 73.30 on MTEB(Code, v1) and 79.30 on LongEmbed, including 99.0 on passkey retrieval. It was trained with contrastive losses on public pairs only, without distilling a larger embedding model.
For search, it is a small model to embed transcripts, OCR text and captions extracted from video, images and PDFs, where documents are often longer than the 512 or 8,192 tokens older small encoders accept. The scores are self-reported and we have not measured it.
Architecture
A 24-layer encoder (hidden size 768, 6 attention heads of dimension 128, SiLU feed-forward of size 1024, RMSNorm) built on the UltraBERT-150M pre-trained model. Two of every three layers use local attention over a 128-token window and every third layer attends globally, with RoPE base 10,000 on local layers and 2,560,000 on global ones, which is how it reaches a 32,768-token context. The embedding is the CLS token, compared with cosine similarity. Training ran in two contrastive stages: about 500M weakly supervised text and code pairs with in-batch negatives, then about 5.7M examples with mined hard negatives and query-side instructions, averaged over several checkpoints.
Mixpeek SDK Integration
# Embed each transcript whole with UltraBERT-Embed-150M (up to 32K tokens), then
# upsert the vectors into a Mixpeek Vector Store namespace you query directly.
from sentence_transformers import SentenceTransformer
from mixpeek import Mixpeek
model = SentenceTransformer("dmis-lab/UltraBERT-Embed-150M", trust_remote_code=True)
client = Mixpeek(api_key="YOUR_KEY")
docs = [{"id": "ep-017", "text": open("ep-017.txt").read()}]
vectors = model.encode([d["text"] for d in docs]) # documents take no instruction
client.namespaces.upsert(
namespace="transcripts",
documents=[{"id": d["id"], "dense_embedding": v.tolist(), "text": d["text"]}
for d, v in zip(docs, vectors)],
)Capabilities
- 768-dimensional embeddings for English text and code
- Inputs up to 32,768 tokens, embedded whole
- Instruction on the query side; documents embedded raw, labels with a Label: prefix
- Sentence Transformers 6+ with trust_remote_code; Apache 2.0
Use Cases on Mixpeek
Benchmarks
| Dataset | Metric | Score | Source |
|---|---|---|---|
| MTEB(eng, v2) | Mean over 41 tasks | 67.24 | Model card: dmis-lab/UltraBERT-Embed-150M (self-reported; gte-modernbert-base 66.26) |
| MTEB(eng, v2) Retrieval | Mean over 10 tasks | 57.20 | Model card (self-reported) |
| MTEB(Code, v1) | Mean over 12 tasks | 73.30 | Model card (self-reported; gte-modernbert-base 71.66) |
| LongEmbed | Mean over 6 tasks | 79.30 | Model card (self-reported; passkey 99.0) |
Performance
FlashAttention 2 is recommended so Sentence Transformers can pack batches without padding; SDPA also works. Scores are self-reported. We have not measured it.
Common Pipeline Companions
Frequently Asked Questions
What is UltraBERT-Embed-150M?
A 151M-parameter English text and code embedding model from DMIS Lab, released on 10 October 2026 under Apache 2.0. It outputs 768-dimensional vectors and accepts inputs up to 32,768 tokens.
How long can the input be?
Up to 32,768 tokens. Two of every three layers attend locally over a 128-token window and every third layer attends globally, which keeps long inputs affordable. The card reports 79.30 on the LongEmbed benchmark.
Do I need to add an instruction to queries?
Yes for queries, no for documents. Queries take Instruct: {task}\nQuery: , and prompt_name="query" applies the default retrieval instruction. Documents are embedded raw, and label phrases take the prefix Label: .
How does it compare with gte-modernbert-base?
On the card's MTEB(eng, v2) table it scores 67.24 against 66.26 at a similar size (151M against 149M parameters), and 73.30 against 71.66 on MTEB(Code, v1). Its context is 32,768 tokens against 8,192. The figures are self-reported.
How do I use UltraBERT-Embed-150M with Mixpeek?
Embed your documents with it and upsert the vectors into a Mixpeek Vector Store namespace, as in the example on this page, then query with vectors from the same model. MVS can combine those dense vectors with BM25 over the same text in one hybrid query.
Specification
Research Paper
UltraBERT: Beyond the Limit of BERT Pre-Training (arXiv 2610.15028)
arxiv.orgBuild a pipeline with UltraBERT-Embed-150M
Add this model to a processing pipeline alongside other extractors. Combine with retrieval stages for end-to-end search.
Run it on your own data, free