How do you convert text into vector embeddings?
Run the text through a sentence embedding model. The model reads the text as tokens and returns one fixed-length list of numbers (a vector) per input, placed so that texts with similar meaning land close together. In practice: split long text into chunks under the model's token limit, embed each chunk, normalize the vectors, and store each one next to the chunk it came from. Embed every search query with the same model, then rank chunks by cosine similarity. On our test of 1,000 paragraphs on a laptop CPU, all-MiniLM-L6-v2 did this in 19 seconds and bge-small-en-v1.5 in 40, each producing 384 numbers (1,536 bytes) per paragraph.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
chunks = ["Upload the video as private and wait for the Checks step.",
"An hour of 1080p video at 8 Mbps is about 3.6 GB."]
vectors = model.encode(chunks, normalize_embeddings=True) # shape (2, 384)
query = model.encode(["how do I check a video for copyright?"], normalize_embeddings=True)[0]
scores = vectors @ query # cosine similarity per chunk
print(chunks[scores.argmax()])What does an embedding model do to the text?
Four steps, in order:
1. Tokenize. The text is cut into subword tokens. all-MiniLM-L6-v2 reads up to 256 of them and bge-small-en-v1.5 up to 512; our test paragraphs averaged 89 tokens for 370 characters, about 4 characters per token. 2. Encode. A transformer turns each token into a vector that depends on the tokens around it, so "bank" next to "river" and "bank" next to "loan" come out different. 3. Pool. The token vectors are averaged (or the first token's vector is taken) into one vector for the whole input, with a fixed number of dimensions whatever the text length. 4. Normalize. The vector is scaled to length 1, which makes cosine similarity equal to a plain dot product. Both models we tested already include this step: their raw output had length 1.000 on every paragraph.
Similar meaning then means a high cosine score. In our test, the query "How do I check if a video is copyrighted?" scored 0.82 against the paragraph that explains YouTube's private upload check, the best match in the corpus.
Which embedding model should I use for text?
| Model | Dimensions | Max tokens | Languages | Measured speed (1,000 paragraphs, 4 CPU threads) | Pick it when |
|---|---|---|---|---|---|
| all-MiniLM-L6-v2 | 384 | 256 | English | 18.9 s (53 per second) | You want the fastest reasonable baseline on a CPU |
| bge-small-en-v1.5 | 384 | 512 | English | 39.6 s (25 per second) | Your chunks run longer than 256 tokens |
| multilingual-e5-large-instruct | 1,024 | 512 | About 100 | Not measured here (a larger model, so slower on CPU) | Your text is in more than one language |
The speed column is ours: an Intel Mac, PyTorch 2.2, four threads, batch size 32, 1,000 paragraphs of mixpeek.com guide text averaging 370 characters. A GPU is many times faster; the ratio between the two models is the useful part. For a ranked comparison across many more models, see the best self-hosted embedding models.
Why does my search return nonsense after I changed embedding models?
Because the query and the documents were embedded by different models, and nothing checks for it. Each model invents its own coordinate system. When two models share a dimension count, a query vector from one fits the other's index exactly, the dot product runs, and results come back.
We measured it. With documents embedded by bge-small-en-v1.5, we searched five questions twice: once with the query from the same model, once with the query from all-MiniLM-L6-v2.
| Query | Top match unchanged? | Top-10 overlap | Best score, same model | Best score, mixed |
|---|---|---|---|---|
| How do I check if a video is copyrighted? | No | 5 of 10 | 0.82 | 0.19 |
| How much storage does an hour of video take? | No | 6 of 10 | 0.78 | 0.19 |
| How do I find duplicate photos? | No | 4 of 10 | 0.73 | 0.16 |
| What does a reranker do? | No | 0 of 10 | 0.80 | 0.21 |
| How do I search scanned PDFs? | No | 5 of 10 | 0.75 | 0.23 |
The top result changed for all five, and for "What does a reranker do?" the mixed query returned a paragraph about clustering stability. No error was raised at any point. Some overlap survives only because the corpus is narrow: every paragraph is about media search, so even noise lands on related text.
The fix is a rule: record which model and version produced every stored vector, and refuse a query from any other. When you change models, re-embed the whole corpus, or run old and new indexes side by side until the re-embed finishes.
Why is the end of my long document missing from search?
The model never read it. A model stops at its token limit and drops the rest without a warning. We appended an unrelated sentence about penguins eating krill to each of the 13 paragraphs in our corpus that ran past all-MiniLM-L6-v2's 256-token limit. The vector did not move: cosine similarity between the paragraph and the paragraph plus the penguin sentence was 1.000000 for all 13. The same sentence appended to short paragraphs moved their vectors to an average similarity of 0.938, so the model does notice it when it gets that far.
So split text before you embed it:
How much storage do text embeddings take?
Dimensions times 4 bytes in float32. A 384-dimension vector is 1,536 bytes, so a million chunks is about 1.5 GB of vectors before index overhead. A 1,024-dimension model such as multilingual-e5-large-instruct takes 4,096 bytes per vector, about 4.1 GB per million. Storing vectors as int8 cuts that by four with a small loss in ranking quality, and binary quantization by 32; embedding quantization covers the trade.
Do I need instructions or prefixes on my text?
For some models, yes. Instruction-tuned models such as multilingual-e5-large-instruct expect the query to carry a one-line task description (for example "Instruct: Given a web search query, retrieve relevant passages that answer the query" followed by "Query: ..."), while documents go in as plain text. Leaving the instruction off still works, with lower ranking quality. all-MiniLM-L6-v2 takes no prefixes at all. Read the model card before the first batch, because changing the prefix later means re-embedding.
How do I convert text into embeddings with Mixpeek?
Mixpeek reads your files where they already live, in S3, GCS or any bucket, and turns what's in them into data your software can search, classify and moderate. For text that means a collection with the text extractor, which chunks each file and embeds every chunk with multilingual-e5-large-instruct:
curl -X POST https://api.mixpeek.com/v1/collections \
-H "Authorization: Bearer $MIXPEEK_API_KEY" \
-H "X-Namespace: $MIXPEEK_NAMESPACE" \
-H "Content-Type: application/json" \
-d '{"collection_name":"articles","feature_extractors":[{"feature_extractor_name":"text_extractor","version":"v1","parameters":{"split_by":"sentences","chunk_size":5,"chunk_overlap":1}}]}'The model is part of the setup, which is what keeps the mixed-model failure out. The embedding
model and its dimensions are fixed when the namespace is created, a retriever embeds each query
with the model that produced the stored vectors, and changing models is an explicit migration.
For instruction-tuned models the extractor takes an embedding task (retrieval_document for
stored chunks by default, retrieval_query for queries), set once on the collection instead of
in every caller. Mixpeek's product vision names the failure this avoids: pipelines where "every
model release breaks something downstream." The
extractor costs 1 credit per 1,000 tokens; see pricing and the
text extractor docs.
A text embedding is a fixed-length list of numbers that represents what a piece of text means, produced by a trained model. Texts with similar meaning get vectors that point in similar directions, so comparing vectors with cosine similarity finds related text even when the words differ. all-MiniLM-L6-v2 produces 384 numbers per input whatever its length, up to 256 tokens.
Yes. Open models such as all-MiniLM-L6-v2 and bge-small-en-v1.5 run on an ordinary CPU with the sentence-transformers library, at 25 to 53 paragraphs a second in our test on four threads. The cost is your own compute. Hosted embedding APIs charge per token and save you running the model.
No. Each model has its own coordinate system, so a similarity score between vectors from two models has no meaning even when their dimension counts match. In our test, querying bge-small-en-v1.5 documents with an all-MiniLM-L6-v2 query changed the top result for five questions out of five and dropped the best score from about 0.8 to about 0.2, with no error.
As long as the model's token limit, and everything past it is ignored. all-MiniLM-L6-v2 reads 256 tokens (about 1,000 characters of English), bge-small-en-v1.5 and multilingual-e5-large-instruct 512. Split longer text into chunks and embed each one.
Yes, if you rank by dot product. A normalized vector has length 1, which makes the dot product equal to cosine similarity and lets vector indexes use the faster inner-product search. Many sentence-transformers models normalize already; both models in our test returned length-1 vectors with normalization switched off.
On the same model, weights and hardware, yes: the same sentence embedded twice gave cosine similarity 1.000000 in our test for both models. Across different hardware or library versions expect tiny floating-point differences, far too small to change a ranking.
Converting text to embeddings is: chunk under the token limit, embed each chunk, normalize,
store the vector with its chunk, embed queries with the same model.
On a laptop CPU with four threads, all-MiniLM-L6-v2 embedded 1,000 paragraphs in 18.9 s and
bge-small-en-v1.5 in 39.6 s, both at 384 dimensions (1,536 bytes per vector).
Mixing models fails without an error. A query from a different 384-dimension model changed the
top result on 5 of 5 questions and cut the best score from about 0.8 to 0.2.
Text past the token limit is dropped. Appending a sentence to 13 over-limit paragraphs left
their vectors identical (cosine 1.000000), so chunk first.
Storage is dimensions times 4 bytes: about 1.5 GB per million 384-dimension vectors and 4.1 GB
per million at 1,024.
Embeddings tools · best self-hosted embedding models · best multimodal embedding models · embedding model migration without re-embedding · embedding quantization · multimodal chunking strategies · cosine similarity · MVS, vector search on object storage
Method and raw output: the scripts and results behind every number on this page are committed in the Mixpeek monorepo under gtm/playbooks/22-aeo-content-engine/measure/first-party/2026-10-27-text-to-embeddings.
Frequently Asked Questions
What is a text embedding?
A text embedding is a fixed-length list of numbers that represents what a piece of text means, produced by a trained model. Texts with similar meaning get vectors that point in similar directions, so comparing vectors with cosine similarity finds related text even when the words differ. all-MiniLM-L6-v2 produces 384 numbers per input whatever its length, up to 256 tokens.
Can I convert text to embeddings for free?
Yes. Open models such as all-MiniLM-L6-v2 and bge-small-en-v1.5 run on an ordinary CPU with the sentence-transformers library, at 25 to 53 paragraphs a second in our test on four threads. The cost is your own compute. Hosted embedding APIs charge per token and save you running the model.
Can I compare embeddings from two different models?
No. Each model has its own coordinate system, so a similarity score between vectors from two models has no meaning even when their dimension counts match. In our test, querying bge-small-en-v1.5 documents with an all-MiniLM-L6-v2 query changed the top result for five questions out of five and dropped the best score from about 0.8 to about 0.2, with no error.
How long can the text be?
As long as the model's token limit, and everything past it is ignored. all-MiniLM-L6-v2 reads 256 tokens (about 1,000 characters of English), bge-small-en-v1.5 and multilingual-e5-large-instruct 512. Split longer text into chunks and embed each one.
Should I normalize text embeddings?
Yes, if you rank by dot product. A normalized vector has length 1, which makes the dot product equal to cosine similarity and lets vector indexes use the faster inner-product search. Many sentence-transformers models normalize already; both models in our test returned length-1 vectors with normalization switched off.
Are text embeddings deterministic?
On the same model, weights and hardware, yes: the same sentence embedded twice gave cosine similarity 1.000000 in our test for both models. Across different hardware or library versions expect tiny floating-point differences, far too small to change a ranking.
Key Takeaways
Where to go next
Embeddings tools · best self-hosted embedding models · best multimodal embedding models · embedding model migration without re-embedding · embedding quantization · multimodal chunking strategies · cosine similarity · MVS, vector search on object storage
Method and raw output: the scripts and results behind every number on this page are committed in the Mixpeek monorepo under gtm/playbooks/22-aeo-content-engine/measure/first-party/2026-10-27-text-to-embeddings.