Why Your Video Search Can't Find Words That Appear On Screen

The Short Answer

When your search misses text that is only shown on screen and never spoken, the search is reading a transcript, and a transcript only has what was SAID. Words that appear on screen without being spoken (a title card, a lower third, a disclaimer, a price on a product shot, a scrolling ticker) never enter it. To find them, run optical character recognition (OCR) on the video's frames, store the text it reads with the timestamp of each frame, and search that alongside the transcript. For one video, a tool that reads the frames is enough; for a library, the OCR text has to be part of the index.

On Mixpeek that pass is scrolling_text_extractor, which reads every overlay in a video, including text that scrolls. If you also want the spoken words, that is multimodal_extractor, which produces transcription, OCR and embeddings per segment. Running both against the same bucket gives you one index that answers "where does anyone say this" and another that answers "where does this appear on screen".

Why the transcript never had those words

A transcript comes from speech recognition, and speech recognition listens. A frame that shows "OFFER ENDS SUNDAY" in 90 point type produces no audio, so the transcript for that second is empty or holds whatever the voiceover happened to be saying over the top.

This is easy to miss because the failure is quiet. The search returns results, the results are relevant to the words that WERE spoken, and nothing anywhere reports that a whole class of text was never indexed. The first sign is usually a person who knows the clip exists, searches the exact phrase they can see with their own eyes, and gets nothing back.

The two kinds, and why the second one is harder

Static overlays sit still for a while: title cards, captions, disclaimers, calls to action, chyrons. Any frame that contains them contains the whole string, so sampling frames and reading them is enough.

Scrolling text never appears in full in any single frame. Tickers, banners, credits and terms-and-conditions crawls move across or up the screen, so a frame sampler gets fragments that overlap unpredictably and a naive OCR pass returns a soup of partial words.

scrolling_text_extractor handles them as two passes for exactly that reason. The static pass samples visually distinct keyframes, stitches them into one grid image, and makes a single VLM OCR call over the grid. The scrolling pass splits frames into strips, phase-correlates consecutive frames to detect per-strip pixel shift, merges strips with a consistent shift into bands, panoramic-stitches each band into one wide or tall image, and OCRs that. It then deduplicates repeated marquee loops and any overlap between the static and scrolling results.

What if I only need to find text in one video?

Pause on the frame and let your device read it: Live Text on recent iPhones, iPads and Macs, and Google Lens on Android and in Chrome, can select and copy text from a paused video frame. For a whole video, online video-OCR tools read every frame and return a searchable, timestamped list of the text they found, which is the quickest route when the question is about one file rather than a library.

How do I make on-screen text searchable across a whole library?

Every approach has the same four steps; the choices inside them decide the quality.

1. Sample frames. Reading every frame is wasteful because text usually stays on screen for seconds. Sample about once per second, and add a frame at every scene cut so short overlays are not skipped between samples. 2. Read the text. OCR each sampled frame. Printed overlays read well; stylized fonts, text over busy backgrounds and very small legal text read worse, and a vision-language model is more forgiving than classic OCR on those. 3. Collapse repeats. The same title card appears in dozens of consecutive samples. Merge identical text in consecutive frames into one entry with a start and end time, so one overlay is one result. 4. Index it with the transcript. Store the on-screen text next to the spoken words for the same time range, and search both. Keep them as separate fields as well, so you can ask "shown but never said" when that matters, for example when checking that a legal disclaimer was displayed.

Scrolling text needs its own handling, covered next.

Indexing it

The shape is the ordinary Mixpeek flow. A bucket holds the source videos, a collection runs an extractor over them, and a batch does the work.

# 1. A bucket for the videos
POST /v1/buckets
{"bucket_name": "ad-creative",
 "bucket_schema": {"properties": {"video": {"type": "video"}}}}

# 2. The videos themselves
POST /v1/buckets/{bucket_id}/objects

# 3. A collection that reads on-screen text
POST /v1/collections
{"collection_name": "onscreen-text",
 "source": {"bucket_id": "{bucket_id}"},
 "feature_extractors": [{"name": "scrolling_text_extractor", "version": "v1"}]}

# 4. Run it
POST /v1/buckets/{bucket_id}/batches

# 5. Read the results
GET /v1/collections/{collection_id}/documents

Processing runs at roughly 2 to 5 times realtime depending on resolution and the sampling rate you configure, so a one minute clip costs somewhere in the region of twelve to thirty seconds of extraction.

What comes back

One document per source video, carrying every string found:

Field

What it holds

onscreen_text	Everything found, static and scrolling, pipe separated
static_text	Overlays only
scrolling_text	The scrolling bands only
scroll_bands	Per band detail: axis, direction, and that band's text
bands_detected	How many scrolling bands were found

Keeping the three text fields apart is worth more than it looks. A disclaimer that crawls along the bottom of an ad is a compliance artifact and a title card is creative, and once they are merged into one string you cannot ask about either one without the other.

The part that surprises people

The extracted text is stored as payload, not as a vector. Semantic search over it uses the text_extractor vector, so if you want "find the ad that mentions a money back guarantee" rather than an exact string match, you index the extracted text with text_extractor as well.

That is a deliberate split. On-screen text is often short, uppercase and punctuation free, which embeds poorly on its own, and a lot of the questions people ask about it ("which spots carry the legal line", "how many have a CTA") are filters rather than similarity queries.

Which extractor for which question

You want

Use

Every word visible on screen, including tickers and credits	scrolling_text_extractor
The spoken words, plus per segment OCR and embeddings	multimodal_extractor
Text out of PDFs and scanned documents	document_graph_extractor
One extractor across images, video, audio and documents	universal_extractor

scrolling_text_extractor takes video only. For spoken content reach for multimodal_extractor with transcription enabled, and for documents reach for text_extractor.

Frequently Asked Questions

Why does my search miss text that is only shown on screen and never spoken?

Because the search index holds a transcript, which comes from the audio. Text that only appears in the picture produces no sound, so it is never in the transcript. Running OCR on the video frames and indexing that text with timestamps makes it searchable.

Does a transcript ever include text that is only shown on screen?

No. A transcript is produced by speech recognition, which listens to the audio track. A frame showing "OFFER ENDS SUNDAY" in large type produces no sound, so nothing about it reaches the transcript. The two are separate signals and each needs its own extraction pass.

What about burned-in subtitles or open captions?

Those are on-screen text, so they come from the OCR pass rather than the transcript. That matters when the audio is in one language and the burned-in subtitle is in another: the transcript holds the spoken language and onscreen_text holds the written one, and searching only the transcript misses every viewer who remembers the subtitle.

Can I search the extracted on-screen text semantically?

Not from scrolling_text_extractor alone. Its output is stored as payload, so it supports filtering and exact matching. For similarity search over that text, index it with text_extractor as well and query that vector.

Does it handle credits and legal crawls that never sit still?

Yes, that is the scrolling pass. It detects bands moving horizontally or vertically, reconstructs each band by stitching frames like a panorama, and OCRs the reconstructed image, then removes repeated marquee loops so a ticker that cycles four times is not returned four times.

How long does extraction take?

Roughly 2 to 5 times realtime, depending on the video resolution and the frame sampling rate you configure. A one minute clip lands somewhere around twelve to thirty seconds.

My video is a screen recording of a slide deck. Is that the same problem?

Same problem, and the on-screen pass is what reads the slides. If the deck also exists as a PDF, document_graph_extractor will give you better structure from the source file than OCR over a recording of it, so prefer the original when you have it.

Checking whether this is your problem

Take a clip where you can see the text with your own eyes. Search the exact phrase. If the transcript search returns nothing and you can read the words in the frame, the words were never indexed, and no ranking change or embedding swap will recover them. The fix is an extraction pass, not a retrieval one.