What is Optical Character Recognition (OCR)
Optical Character Recognition (OCR) - Turning text in images, scans and video frames into machine-readable text
Optical character recognition (OCR) is software that finds text in an image, a scanned page or a video frame and converts it into characters a computer can search, copy and index. A modern OCR system first detects where text sits, then recognizes the characters in each region, then rebuilds reading order and layout. Today the work is done by three kinds of models: classic engines such as Tesseract, deep learning pipelines such as PaddleOCR, and vision language models that read an image and return its text in one step.
How It Works
- Text detection: a model draws boxes around lines or words, including rotated or curved text
- Text recognition: each region is read into characters, usually by a sequence model that predicts the whole line at once
- Layout analysis: blocks are ordered into reading order and labeled as headings, paragraphs, tables or captions
- Post-processing: spelling and language models fix likely misreads, and each block gets a confidence score
Types of OCR Models
- Classic engines (Tesseract): free, run on a CPU, strong on clean printed text, weak on photos, handwriting and complex layouts
- Deep learning pipelines (PaddleOCR and similar): separate detection and recognition networks, good on photos, signs and many scripts, and fast enough for large batches
- Vision language models (Gemini, GPT-class and open models such as Qwen-VL): read the whole image at once, handle messy layouts, tables and handwriting better, and cost more per page
- Specialized models: receipts, IDs, math and music notation each have dedicated models that beat general ones on their format
Choosing the Best OCR Model
- Clean, born-digital PDFs: skip OCR and read the embedded text layer
- Scanned documents: a detection and recognition pipeline, with a vision language model to correct low-confidence blocks
- Photos, screenshots and ad creatives: a vision language model, since layouts vary and text sits on busy backgrounds
- Video: sample frames, remove near-duplicate frames, and read only frames where on-screen text changes; scrolling tickers need the frames stitched first
- Test on 50 to 100 of your own files and measure character or word error rate before choosing
OCR in Mixpeek
- Scanned PDFs: the document extractor runs OCR with layout recovery, corrects noisy scans with a vision language model, tags every block with a confidence grade, and embeds the text so a `feature_search` stage can search it
- Images and video frames: the multimodal extractor reads on-screen text when `run_ocr` is on and stores it with each segment's transcript and description
- Scrolling text in video: the scrolling text extractor detects tickers, credits and disclaimers that no single frame shows in full, stitches them into one image and reads it
- The retrieval cookbook has a ready retriever for searching OCR text from scanned PDFs
Common Pitfalls
- Running OCR on PDFs that already carry a text layer, which adds errors and cost
- Scanning below 150 DPI; 150 to 300 DPI originals give the best results
- Ignoring reading order, so two-column pages come out with lines interleaved
- Trusting low-confidence blocks in search or analytics without a flag
- Assuming handwriting and non-Latin scripts work as well as printed Latin text
Put multimodal search to work
Connect a bucket and Mixpeek runs the whole multimodal search pipeline for you: extraction, indexing, and search over your own objects. No models to wire up, nothing to host.
Start with ManagedAlready have vectors?
Keep your embeddings on your own cloud and run dense, sparse, and BM25 search directly on object storage. From $25/mo.
Start with MVS