The Short Answer
Search stays stuck in one language when some stage of the pipeline compares words instead of meaning, or compares meaning with a model that only learned one language. The transcript of a Japanese video is Japanese text. A keyword stage matches tokens, so an English query scores zero against it. A monolingual embedding model puts that Japanese text somewhere arbitrary in vector space, so a semantic query lands nowhere near it either. The fix is to embed every language with one multilingual model so the query and the content share a space, let transcription detect the spoken language instead of forcing one, and stop expecting the keyword stage to carry cross-language matches. Images and video frames have no language at all, so a multilingual vision-language model searches them from any language with nothing to translate.
The three causes, and the fingerprint each one leaves
The same symptom, "it only works in English", comes from three different places in the pipeline. They are worth separating because the fix for one does nothing for the others.
1. A keyword stage is doing the matching
Full-text search and BM25 compare tokens. The English word "sanctions" and the Japanese word "制裁" share none, so the score is zero, and no amount of tuning changes that. If your hybrid retriever leans on its lexical half, or the semantic half was never wired up, every cross-language query hits a wall.
The fingerprint: same-language queries return good results, and cross-language queries return nothing at all. When something does come back, it is a segment where a proper name, a product code or a number happened to appear in both languages, because those are the only tokens that survive a language boundary.
2. The embedding model only learned one language
A lot of text encoders were trained on English. The original CLIP text tower learned from English captions, and a text embedding model trained on English web pages has seen very little Japanese. Given a Japanese query it still produces a vector, because it will tokenize anything, and that vector is noise relative to where the English content sits.
The fingerprint: cross-language queries return results, with scores that look normal, and the results are wrong. Nothing about the response tells you the model had no idea what it was reading. This is the only one of the three that a better model fixes.
3. Transcription was told the wrong language
Speech recognition models like Whisper detect the spoken language on their own when you do not tell them one. Force a language code that does not match the audio, which happens whenever a sample config with
"en" in it gets copied onto a Spanish or Farsi corpus, and the transcript comes out as garbled text shaped like the wrong language. Nothing errors: the collection processes and every document gets a transcript that is useless.The fingerprint is the one that separates this from the other two: queries fail on those files in every language, including the language the audio was recorded in. Open the stored transcript of a segment you can listen to and the damage is obvious.
Tell which one you have
Three checks, each under a minute.
| Check | What you do | What it tells you |
| 1. Ask the same question twice | Run one query in the content's language, then in another language | Good results then nothing: cause 1. Good results then wrong results: cause 2. Bad results both times: cause 3 |
| 2. Read the matched text | Print the text of the top result rather than its title | If the match hinges on a shared name or number, the lexical stage carried it |
| 3. Read one stored transcript | Pick a segment you can listen to and compare it to what was transcribed | Text in the wrong language, or fluent nonsense, means the transcription language was forced |
Fixing each one
For the keyword stage, keep it and stop asking it to do cross-language work. Lexical retrieval is what catches exact identifiers and names, so it stays in the retriever, and the dense stage becomes the one that carries meaning across languages. The consequence for a hybrid system is that fusion weights behave differently on cross-language queries than on same-language ones, because the lexical half contributes nothing to the former. Hybrid search: BM25 and vector fusion covers how to weight the two halves.
For the model, use one multilingual embedding model for the whole index. BAAI/bge-m3 and multilingual-e5-large-instruct both cover 100+ languages and place a sentence and its translation close together by construction, so a query in any of those languages finds content in all of them without a translation step or a language detector. Every vector in an index has to come from the same model, so switching means re-embedding the corpus. Do it once, and pick the model with that in mind.
For transcription, leave the language unset and let the model detect it per segment. Override only when detection is wrong on your audio, which happens with music beds under speech and heavy code-switching, and then set the code for the language spoken in the recording.
Video and images are the easy case
A frame has no language. A vision-language model embeds images and text into one shared space, so when its text side is multilingual you get cross-language image and video search with nothing to translate: a Spanish query lands next to the matching frame directly. What reintroduces language into video search is everything around the frames. The transcript is in whatever was spoken. On-screen text and an attached slide deck each bring their own. One video can carry three languages at once, which is exactly the case that defeats detect-the-language-and-route designs. Keep the visual signal and each text signal as separate retrieval stages, and a miss tells you which side it came from.
How this looks on Mixpeek
# 1. bucket: land every file, whatever language it was recorded in
POST /v1/buckets/{bucket_identifier}/objects
{"blobs":[{"property":"video","type":"video","data":"https://your-storage/interviews/tokyo-0417.mp4"}]}
# 2. collection: transcribe in the spoken language, embed with a multilingual model
POST /v1/collections
{"collection_name":"interviews",
"source":{"type":"bucket","bucket_ids":["{bucket_identifier}"]},
"feature_extractor":{"feature_extractor_name":"multimodal_extractor","version":"v2",
"parameters":{"run_transcription":true,"run_transcription_embedding":true}}}
# 3. retriever: ask in one language, match content in any
POST /v1/retrievers/{retriever_id}/execute
{"inputs":{"query":"where the guest explains why they left the company"}}transcription_language is left out on purpose. Unset, the multimodal extractor lets Whisper detect the language of each segment, so a bucket that mixes Japanese, Portuguese and English needs no configuration and no per-language collection. The transcript comes back in the language that was spoken. run_transcription_embedding embeds that transcript with multilingual E5, which is what makes an English query match a Japanese segment by meaning, and the frame embedding never had a language to begin with.Mixpeek returns matched text in the language it was spoken or written in, with no translation step. If a reader needs it in theirs, append an
llm_enrich stage to the retriever that writes a translation field beside the original content, which is the recipe in the retrieval cookbook. Translating at query time on the results that matched is far cheaper than translating a corpus at ingest, and it keeps the original wording where the search can still see it.Frequently Asked Questions
Will a multilingual embedding model fix all three causes?
It fixes the second one. A keyword stage still matches tokens no matter what model sits beside it, and a transcript that was forced into the wrong language is bad input that no embedding model can recover. Run the checks above before switching models, because a model swap means re-embedding the entire corpus, and it is a waste when the transcript was the problem.
Should I translate everything into English at ingest instead?
For a small number of languages it is a legitimate design, and it lets you use the strongest single-language model for English. The costs are a translation pass over the whole corpus, a repeat of that pass whenever the translation model changes, and the loss of the original wording, which is what a user who speaks that language wanted to see. A multilingual embedding model avoids all three, and translation of the few results that matched can happen at query time instead.
Why do results get worse for some languages than others?
Alignment quality follows how much parallel text existed for a language pair when the model was trained. The well-resourced European and East Asian languages sit tightly aligned; low-resource languages sit in roughly the right region rather than precisely on target, and precision at the top of the ranking suffers first. That is a property of the training data, and query-time tuning does not change it. Measure recall per language pair on your own content rather than trusting a headline count of supported languages.
The audio switches languages mid-recording. What happens?
Whisper detects per segment, so a recording that moves between languages gets each segment transcribed in the language spoken there, as long as nothing forced a single code for the whole file. Heavy code-switching inside a single segment is where detection is least reliable, and it is the one situation where overriding the language for that collection can beat detection.
My users type queries in their language but the content is all English. Does this still apply?
Yes, and it is the most common version of the problem. The query has to land in the same space as the content. A multilingual embedding model puts the Japanese query next to the English content it means, and a keyword stage still contributes nothing to that match. Run check 1 in reverse, an English query against English content and then a Japanese query against the same content, and the fingerprint tells you which cause you have.