Guides
Vendor-neutral, engineer-written guides to the concepts behind multimodal AI: perception, retrieval, embeddings, and the infrastructure agents use to see, hear, and search unstructured data. Learn the idea first; then see how Mixpeek applies it.
164 guides across 24 topics
Did my search get better? How to test a retrieval change before you ship it
To know whether a new embedding model, reranker or ranking tweak made search better, run the old and new system on the same set of judged queries, score both with recall@k, MRR and nDCG@k, and test whether the per-query differences are larger than noise. How to build the judged query set, which metric answers which question, what to do about unjudged results, how many queries you need, when an LLM judge can stand in for people, and how to confirm the result online after shipping. Includes a worked example and runnable Python.
AdCP, AAMP, ARTF and Agentic Audiences: what each one does
AdCP and AAMP are two separate efforts to standardize AI agents in advertising. AdCP is one protocol from AgenticAdvertising.org, carried over MCP or A2A, with tasks like get_signals and create_media_buy. AAMP is IAB Tech Lab's umbrella, which includes ARTF (agent containers inside an exchange that return bid request mutations) and Agentic Audiences (embeddings carried in OpenRTB). A dated comparison table, spec links, and what to do when a contextual embedding model changes.
Why CTV bid requests lack content categories, and how to fill them
Most CTV bid requests arrive with little or no content metadata because the publisher never classified the stream in IAB terms, privacy law limits show-level sharing, and some publishers hold their best show data back for direct deals. The fix is to classify each asset once at ingest from its frames, speech and on-screen text, store IAB Content Taxonomy 3.1 IDs keyed by content ID, and fill OpenRTB content.cat, cattax, genres and gtax by lookup when the request goes out.
Why Is My Image Embedding Model So Slow on CPU?
An image embedding model that crawls on CPU is usually running in bfloat16 on a processor with no hardware for it, or leaving most of its cores idle. PyTorch autocast on CPU defaults to bfloat16, and without AVX512-BF16 or AMX the math runs in software: we measured 13.0 s per image under autocast against 0.555 s in fp32 for SigLIP base. Run CPU inference in fp32, give each process one thread, and start one model copy per core.
Why Does My Content Moderation Flag So Many False Positives?
Content moderation flags too much when one confidence threshold does every job. Default cutoffs are low (Amazon Rekognition returns any moderation label at 50% confidence unless you set one), harmful content is rare so even an accurate classifier flags more safe items than real ones, and a video checked frame by frame gives every frame a chance to misfire. Set a threshold per category from your own labeled sample, send a middle band to review, and judge video by segment with context.
Why Does My Image Search Miss Small Objects in a Photo?
Image search misses small objects because most systems store one vector per photo, computed from a copy shrunk to about 224 pixels. A 100-pixel object in a 4000x3000 photo becomes 5 to 8 pixels, smaller than one of the model's image patches, and CLIP's center crop drops a quarter of a wide photo before the model sees it. The fix is to embed crops of each photo as well as the whole, and rank each photo by its best-matching crop.
Why Does AI Miss Things When I Ask It About a Long Video?
AI misses things in a long video because it only sees sampled frames. Gemini reads one frame per second by default, and Claude's and OpenAI's APIs take images only, capped at 600 and 1,500 per request, while an hour of video at one frame per second is 3,600 frames. The fix is to find the minutes your question is about first and send only those, at a higher frame rate.
How Do I Keep Footage of Minors Out of a Video Training Dataset?
To keep footage of minors out of a video training dataset, screen at ingest: detect and group faces across cuts, estimate each person's age, and send anyone near the cutoff or too unclear to age to human review. On our 650-face evaluation, a cutoff of 18 caught 82% of minors.
Which Retrieval Pipeline Steps Should Use a Decision Model Like Jev?
Many steps in a retrieval pipeline are decisions over a fixed set of options: keep or discard, pick a label, rate one to five, route to a branch. A decision model such as TypeSafe's Jev answers these as calibrated probabilities, in one fast call, at a fraction of the cost of generating text. This explains which steps fit, how to phrase them, and where a generative model is still the right tool.
How Do I Match the Right Video to Each Article on My Site?
To match the right video to each article, index videos as short scenes with transcript, on-screen text and an embedding, extract each article's entities, keywords and IAB category, and rank scenes by shared entities first. Return the matching seconds and why each was chosen.
How Do I Find Where Users Got Stuck Across Dozens of Usability Test Recordings?
To find where users got stuck across many usability test recordings, index them so each word is joined to the screen it was spoken over, then ask once across every session. Results open at the moment it happened, attributed to a participant, and grouping by participant shows who hit the problem.
How Do I Search Photos, Video and Audio on the Device, Without Uploading Them?
To search photos, video and audio on a phone or laptop without uploading them, embed each item with a multimodal model running on the device, keep the vectors in a local index, and search it with text, a photo or a voice memo. EmbeddingGemma 2 does this in about 567 MB of RAM.
All guides
Every guide in one place, 164 in total.
- Did my search get better? How to test a retrieval change before you ship it
- AdCP, AAMP, ARTF and Agentic Audiences: what each one does
- Why CTV bid requests lack content categories, and how to fill them
- Why Is My Image Embedding Model So Slow on CPU?
- Why Does My Content Moderation Flag So Many False Positives?
- Why Does My Image Search Miss Small Objects in a Photo?
- Why Does AI Miss Things When I Ask It About a Long Video?
- How Do You Run Agentic Search Over a Hierarchy of Clusters?
- Which Retrieval Pipeline Steps Should Use a Decision Model Like Jev?
- Why Do I Get Different Search Results Every Time I Run the Same Query?
- Why Does My Video Search Return the Wrong Scene?
- How Do I Stop Duplicate Videos Showing Up in My Search Results?
- How Do I Let an AI Agent Look Through My Own Video Library?
- How Do I Find Where Users Got Stuck Across Dozens of Usability Test Recordings?
- How Do I Search Photos, Video and Audio on the Device, Without Uploading Them?
- How Do I Match the Right Video to Each Article on My Site?
- How Do I Keep Footage of Minors Out of a Video Training Dataset?
- How Do I Get an Alert When My Security Camera Sees Something Specific?
- How Do I Find One Specific Moment in Hours of Footage?
- How Do I Find Where Someone Said Something in Our Meeting Recordings?
- How Do I Find Duplicate Photos, Including Edited and Resized Copies?
- How Do I Make Hundreds of Scanned PDFs Searchable?
- How Do I Find Every Ad We've Run That Shows a Specific Product?
- What Does 'Multimodal' Mean in Google Search Console?
- Why Does My Image Search Return Similar-Looking Items Instead of the Exact One?
- How Do I Find Out Where a Video Came From?
- Why Do My Photos Have No Date or Location, and Can I Get It Back?
- Why Can't My Search Find Exact Part Numbers, Codes or Names?
- Why Do My Search Results Get Worse as I Add More Data?
- Why Your Video Search Can't Find Words That Appear On Screen
- How to Prove Who Changed a Configuration, When, and Why
- Why Does My Search Find the Right Document but the Wrong Part of It?
- How Do You Search a 3D World You Generated?
- Search Every World You Generate with Atlas
- Search Every World You Generate with Runway
- Search Everything You Generate on fal
- Search Every Image You Generate with ChatGPT
- Why Does My Podcast Search Find the Right Episode but Not the Moment?
- Why Does My Search Only Work in One Language?
- Search Every Conversation You Have with GPT Live
- What Is a Semantic Contract? The Interface Between Extraction and Retrieval
- What Does a Semantic Layer Compile To When the Target Is Not SQL?
- How Should an AI Agent Choose Between Retrieval Capabilities?
- Should Semantic Features Be Materialized or Computed at Query Time?
- How Do You Model Unstructured Data From Raw Objects to Serving Contracts?
- What a Semantic Layer Looks Like in Production
- Compute Pushdown: Why Your Vector Search Over Object Storage Is Slow
- How to Merge Data Sources With Different Schemas Into One Searchable Index
- Write-Ahead Logs on Object Storage: How S3 Becomes a Database's Durability Layer
- Query Planning for Multimodal Retrieval: Stage Order, Filter Pushdown, and Candidate Budgets
- Multi-Stage Retrieval: Retrieve, Filter, Rerank
- Change Data Capture for a Search Index: Keeping Downstream Systems in Sync
- Query Rewriting, Expansion and HyDE: Fixing Retrieval Before It Searches
- Cross-Lingual Retrieval: Searching Content in Languages You Did Not Query In
- C2PA Content Credentials: How Media Provenance Actually Works
- How Do You Detect and Redact PII in Images, Video, and Audio?
- How Do You Delete Data From a Vector Index? Embeddings, Tombstones, and the Right to Be Forgotten
- Do You Need a Vector Database? When Brute Force, pgvector, and a Dedicated Store Each Win
- How to Build a Taxonomy From Unlabeled Data: Clustering, Labeling, and Promotion
- What Is Hybrid Search? BM25, Vector Retrieval, and How to Fuse Their Rankings
- What Is MUVERA? Turning Multi-Vector Retrieval Into a Single-Vector Search
- How Do You Run Your Own Model Inside a Managed Search Pipeline?
- How Do You Isolate Tenants in a Vector Index?
- What Does It Cost to Make a Video Library Searchable?
- How to Search, Deduplicate, and Moderate AI-Generated Video and Images (FLUX 3, Kling, Veo, Runway)
- How Does LoRA Fine-Tuning Work? (Adapters, QLoRA, DoRA, and Fine-Tuning Retrieval Models)
- How Do I Evaluate a RAG Pipeline? (Faithfulness, Answer Relevance, Hallucination, and Context Metrics)
- How Do I Ingest Millions of Files into a Search Index? (Images, Video, Documents at Scale)
- How Do AI Agents Search Big Datasets by Navigating Clusters? (Hierarchical Cluster Search)
- What Is Composite Clustering? Clustering Across Multiple Feature Spaces (and Clusters of Clusters)
- How Do I Filter Vector Search Results by Location? (Radius, Bounding Box, Polygon)
- How Do I Automatically Classify Content Against a Taxonomy?
- Brand Safety vs Brand Suitability: How AI Classifies Video for Advertisers
- How Do I Debug Bad Retrieval Results in RAG and Vector Search?
- Can You Run a Vector Database on S3? Object-Storage-Backed Vector Search, Explained
- How Do I Build a Deep Research Agent Over My Own Data?
- What Concepts Exist in My Data That Nobody Has Labeled Yet?
- Marengo 3.0 on Amazon Bedrock: Architecture, Benchmarks and Vector Storage
- How to Store TwelveLabs Marengo Embeddings in Your Own Vector Store
- Video Highlight Detection: How AI Finds the Best Moments
- How Much Does a Vector Database Cost? A 2026 Pricing Comparison
- How to Switch Embedding Models Without Re-Embedding Everything
- Reverse Video Search: How It Works and How to Build One
- Semantic Caching: How Agents Skip Work They Have Already Done
- Efficient Attention: How Models Read Hour-Long Video and Book-Length Documents
- Optical Context Compression: Reading Documents as Images, Not Text
- Multi-Object Tracking: How Agents Follow Objects Across Video Frames
- Monocular Depth Estimation: How Models Infer 3D From a Single Image
- Instance-Level Visual Matching: Finding the Same Object, Not Just Similar Ones
- Face Recognition and Identity Clustering: How Agents Recognize and Group People in Video
- Reasoning Rerankers: How Listwise LLM Rerankers Reorder Retrieval Results
- Retrieval Feedback Loops: Learning to Rank from Clicks, Outcomes, and Agent Interactions
- Matryoshka Representation Learning: Nested Embeddings for Adaptive Multimodal Retrieval
- Creative Ad Analysis for AI Agents: JEPA, Multi-Vector Retrieval, and Signal Fusion
- How to Check if an Image Is Copyrighted
- How Do I Find Out If Someone Reposted My Video?
- How to Check if a Video Is Copyrighted
- How to Check if a Song Is Copyrighted on YouTube
- How to Check if a Picture Is Copyrighted Before Using It
- The Complete Guide to Pre-Publication IP Clearance
- What Is a Multimodal Data Warehouse?
- How to Build a Multimodal Data Warehouse
- Multimodal Data Warehouse Architecture Deep Dive
- Vector Storage Tiering: The Architecture Guide to Hot, Warm, and Cold Vector Data
- How to Build MCP Tools for Multimodal AI Agents
- Embedding Portability and Versioning: Why Your Vectors Are Not as Portable as You Think
- How to Build a Multimodal RAG Pipeline
- Context Engineering for AI Agents
- Video Temporal Grounding: How AI Agents Find Specific Moments in Video
- Multimodal Perception for AI Agents: How to Give Your Agent Eyes, Ears, and Memory
- Open-Vocabulary Object Detection: Teaching AI to Find Anything You Describe
- Audio Feature Extraction: How AI Agents Learn to Hear
- Contrastive Learning: How CLIP, SigLIP, and CLAP Actually Work
- Multi-Stage Retrieval: How AI Agents Search Unstructured Data at Scale
- Visual Document Retrieval: How AI Agents Search Documents Without OCR
- Video Scene Segmentation: How AI Decomposes Continuous Video into Searchable Segments
- Omnimodal Embeddings: One Model for Text, Image, Audio, and Video Retrieval
- Speaker Diarization: How AI Agents Know Who Said What in Audio and Video
- Cross-Encoder Reranking: Why Two-Stage Retrieval Beats One-Stage Search
- MCP Tool Design for Multimodal Search
- Long-Context Video Understanding for Agent Perception
- Structured Extraction from Unstructured Documents: How Vision-Language Models Replace OCR Pipelines
- Late Interaction Retrieval: How ColBERT, ColPali, and ColQwen Search Without Losing Token-Level Detail
- Agentic Retrieval: How AI Agents Search Differently Than Humans
- Embedding Quantization & Compression: Making Vector Search Practical at Billion Scale
- Video RAG: Building Retrieval-Augmented Generation Over Video Corpora
- Evaluating Multimodal Retrieval: Metrics, Benchmarks, and Ground Truth
- Multimodal Chunking Strategies: How to Decompose Video, Audio, Images, and Documents for Search
- How to Build a Video Perception Layer for AI Agents
- Approximate Nearest Neighbor Search: The Algorithms Behind Fast Vector Retrieval
- Instruction-Tuned Embeddings: How Task Prompts Transform Retrieval Quality
- Multi-Index Search Architecture: How to Combine Visual, Audio, and Text Embeddings for Rich Media
- Embedding Space Geometry: Why Cosine Similarity Doesn't Always Mean What You Think
- Object Decomposition and Layered Indexing for AI Agent Perception
- Retrieval Control Planes for AI Agents: Streaming, Cancellation, and Budgets
- Video Anomaly Detection for AI Agents: From Signals to Searchable Events
- Agent Perception Evals: Testing Whether AI Agents Can See, Hear, and Search
- Audio-Visual Retrieval for AI Agents: How to Search What Happened, Not Just What Was Said
- Computer-Use Agent Memory: How to Search Screens, Tools, and UI State
- Production Ingestion Reliability for Agent Perception: Ledgers, Backfills, and Recall Checks
- Mask-Aware Retrieval for AI Agents: Segment First, Search Crops, Then Reason
- Forced Alignment for AI Agents: Word Timestamps, Diarization, and Audio Evidence Search
- Payload Projection for Agentic Vector Search: Field Selection, Evidence Handles, and Context Budgets
- Adaptive Indexing for Agentic Search: Query Logs, Payload Indexes, and Retrieval Routing
- Video Frame Sampling: How Many Frames to Embed and Which Ones to Keep
- Calibrating Similarity Scores: What Cosine Similarity Actually Means for Retrieval
- Chunk Contextualization: How Late Chunking and Contextual Retrieval Fix Broken RAG
- Budget-Aware Multi-Vector Retrieval: How to Make Late Interaction Affordable at Scale
- Diversity-Aware Retrieval: How Agents Avoid Twenty Near-Duplicate Results
- Query Transformation Pipelines: Turning a Raw Agent Question Into Searchable Intent
- Learned Sparse Retrieval and Dense-Sparse Hybrid: Why Agents Need Both Vocabularies
- The Modality Gap: Why Text-to-Image Search Underperforms and How Agents Fix It
- How OCR Actually Works: Detection, Recognition, Reading Order, and Tables
- How ASR Decoding Actually Works: Beam Search, LM Fusion, and Confidence
- Index Freshness and Incremental Updates: How Just-Ingested Content Becomes Searchable
- Filtered Vector Search: How Agents Combine Similarity with Hard Constraints
- How Vision-Language Models Fuse Image and Text Tokens
- Hybrid Search Fusion: How to Combine Dense and Lexical Retrieval Without Breaking Ranking
- Audio Fingerprinting: How Agents Recognize a Specific Recording in Noise
- Embedding Fine-Tuning and Distillation: Teaching an Agent to See and Hear Your Domain
- BM25 and the Inverted Index: The Lexical Retriever Every Hybrid Search Treats as a Black Box
- Perceptual Image Hashing: How Agents Recognize the Same Picture After It Has Been Re-Encoded, Cropped, and Recolored
- Streaming Video Understanding: How Agents Watch an Unbounded Live Feed in Real Time
- Why Does My Document Search Miss the Answer That Is in a Chart or Table?
From concept to production
These guides explain how multimodal perception and retrieval actually work. Mixpeek is the platform that runs them: point it at your storage and get back relevant, timestamped results.