Guides
Vendor-neutral, engineer-written guides to the concepts behind multimodal AI: perception, retrieval, embeddings, and the infrastructure agents use to see, hear, and search unstructured data. Learn the idea first; then see how Mixpeek applies it.
167 guides across 24 topics
Our Brand Safety Tool Flags Too Many False Positives on Video. How Do We Fix It?
A brand safety tool over-flags video when it judges words instead of what the video shows, scores single frames out of context, applies one floor to every brand, and runs on a low default threshold against content that is rarely unsafe. Measure precision per category on a labeled sample of what got blocked, then fix the cause: score segments with picture, speech and on-screen text together, grade treatment as well as topic, and set tiers per brand. Covers the causes, how to measure them, a fix table, and where GARM stands since it shut down in 2024.
Write-Ahead Logs on Object Storage: How S3 Becomes a Database's Durability Layer
A write-ahead log acknowledges a write the moment it is durable, before any index work happens. Move that log onto S3 or GCS and the bucket becomes the durability layer: recovery, read replicas and change feeds all become readers of one chain of sequenced objects. Covers what a WAL is, why object storage changes the design, the two coordination problems it creates (writer fencing and garbage collection that could delete the only copy of a write), how conditional writes solve them, and the shipped write path in Mixpeek MVS.
We Can't Tell What Content Our CTV Ads Ran Next To. How Do We Find Out?
If you cannot tell what content your CTV ads ran next to, the record was never written: most CTV impressions reach the buyer with an app ID at best, and no show, episode or scene. Ask sellers for show-level reporting in the deal, add the VAST [CONTENTID] and [MEDIAPLAYHEAD] macros to your impression trackers so each impression carries the content ID and the second it played at, and have whoever holds the video join those two values to a scene index of the content. Covers what each party can see, the macros, the join, and what to put in a deal.
Can you run vector search on NFS, Lustre or a local NVMe drive?
Yes on a local NVMe drive, which is what most vector databases expect. On NFS and Lustre it depends on the engine: Qdrant's docs rule NFS out, SQLite warns that its locks can fail on network filesystems, PostgreSQL allows NFS only with a hard mount, and Lustre is tuned for large files rather than the small random reads a graph index makes. How each engine treats a shared filesystem, why random-read latency decides it, which index designs tolerate slower storage, and a probe you can run on your own mount.
Did my search get better? How to test a retrieval change before you ship it
To know whether a new embedding model, reranker or ranking tweak made search better, run the old and new system on the same set of judged queries, score both with recall@k, MRR and nDCG@k, and test whether the per-query differences are larger than noise. How to build the judged query set, which metric answers which question, what to do about unjudged results, how many queries you need, when an LLM judge can stand in for people, and how to confirm the result online after shipping. Includes a worked example and runnable Python.
AdCP, AAMP, ARTF and Agentic Audiences: what each one does
AdCP and AAMP are two separate efforts to standardize AI agents in advertising. AdCP is one protocol from AgenticAdvertising.org, carried over MCP or A2A, with tasks like get_signals and create_media_buy. AAMP is IAB Tech Lab's umbrella, which includes ARTF (agent containers inside an exchange that return bid request mutations) and Agentic Audiences (embeddings carried in OpenRTB). A dated comparison table, spec links, and what to do when a contextual embedding model changes.
Why CTV bid requests lack content categories, and how to fill them
Most CTV bid requests arrive with little or no content metadata because the publisher never classified the stream in IAB terms, privacy law limits show-level sharing, and some publishers hold their best show data back for direct deals. The fix is to classify each asset once at ingest from its frames, speech and on-screen text, store IAB Content Taxonomy 3.1 IDs keyed by content ID, and fill OpenRTB content.cat, cattax, genres and gtax by lookup when the request goes out.
Why Is My Image Embedding Model So Slow on CPU?
An image embedding model that crawls on CPU is usually running in bfloat16 on a processor with no hardware for it, or leaving most of its cores idle. PyTorch autocast on CPU defaults to bfloat16, and without AVX512-BF16 or AMX the math runs in software: we measured 13.0 s per image under autocast against 0.555 s in fp32 for SigLIP base. Run CPU inference in fp32, give each process one thread, and start one model copy per core.
Why Does My Content Moderation Flag So Many False Positives?
Content moderation flags too much when one confidence threshold does every job. Default cutoffs are low (Amazon Rekognition returns any moderation label at 50% confidence unless you set one), harmful content is rare so even an accurate classifier flags more safe items than real ones, and a video checked frame by frame gives every frame a chance to misfire. Set a threshold per category from your own labeled sample, send a middle band to review, and judge video by segment with context.
Why Does My Image Search Miss Small Objects in a Photo?
Image search misses small objects because most systems store one vector per photo, computed from a copy shrunk to about 224 pixels. A 100-pixel object in a 4000x3000 photo becomes 5 to 8 pixels, smaller than one of the model's image patches, and CLIP's center crop drops a quarter of a wide photo before the model sees it. The fix is to embed crops of each photo as well as the whole, and rank each photo by its best-matching crop.
Why Does AI Miss Things When I Ask It About a Long Video?
AI misses things in a long video because it only sees sampled frames. Gemini reads one frame per second by default, and Claude's and OpenAI's APIs take images only, capped at 600 and 1,500 per request, while an hour of video at one frame per second is 3,600 frames. The fix is to find the minutes your question is about first and send only those, at a higher frame rate.
How Do I Keep Footage of Minors Out of a Video Training Dataset?
To keep footage of minors out of a video training dataset, screen at ingest: detect and group faces across cuts, estimate each person's age, and send anyone near the cutoff or too unclear to age to human review. On our 650-face evaluation, a cutoff of 18 caught 82% of minors.
All guides
Every guide in one place, 167 in total.
- Our Brand Safety Tool Flags Too Many False Positives on Video. How Do We Fix It?
- We Can't Tell What Content Our CTV Ads Ran Next To. How Do We Find Out?
- Can you run vector search on NFS, Lustre or a local NVMe drive?
- Did my search get better? How to test a retrieval change before you ship it
- AdCP, AAMP, ARTF and Agentic Audiences: what each one does
- Why CTV bid requests lack content categories, and how to fill them
- Why Is My Image Embedding Model So Slow on CPU?
- Why Does My Content Moderation Flag So Many False Positives?
- Why Does My Image Search Miss Small Objects in a Photo?
- Why Does AI Miss Things When I Ask It About a Long Video?
- How Do You Run Agentic Search Over a Hierarchy of Clusters?
- Which Retrieval Pipeline Steps Should Use a Decision Model Like Jev?
- Why Do I Get Different Search Results Every Time I Run the Same Query?
- Why Does My Video Search Return the Wrong Scene?
- How Do I Stop Duplicate Videos Showing Up in My Search Results?
- How Do I Let an AI Agent Look Through My Own Video Library?
- How Do I Find Where Users Got Stuck Across Dozens of Usability Test Recordings?
- How Do I Search Photos, Video and Audio on the Device, Without Uploading Them?
- How Do I Match the Right Video to Each Article on My Site?
- How Do I Keep Footage of Minors Out of a Video Training Dataset?
- How Do I Get an Alert When My Security Camera Sees Something Specific?
- How Do I Find One Specific Moment in Hours of Footage?
- How Do I Find Where Someone Said Something in Our Meeting Recordings?
- How Do I Find Duplicate Photos, Including Edited and Resized Copies?
- How Do I Make Hundreds of Scanned PDFs Searchable?
- How Do I Find Every Ad We've Run That Shows a Specific Product?
- What Does 'Multimodal' Mean in Google Search Console?
- Why Does My Image Search Return Similar-Looking Items Instead of the Exact One?
- How Do I Find Out Where a Video Came From?
- Why Do My Photos Have No Date or Location, and Can I Get It Back?
- Why Can't My Search Find Exact Part Numbers, Codes or Names?
- Why Do My Search Results Get Worse as I Add More Data?
- Why Your Video Search Can't Find Words That Appear On Screen
- How to Prove Who Changed a Configuration, When, and Why
- Why Does My Search Find the Right Document but the Wrong Part of It?
- How Do You Search a 3D World You Generated?
- Search Every World You Generate with Atlas
- Search Every World You Generate with Runway
- Search Everything You Generate on fal
- Search Every Image You Generate with ChatGPT
- Why Does My Podcast Search Find the Right Episode but Not the Moment?
- Why Does My Search Only Work in One Language?
- Search Every Conversation You Have with GPT Live
- What Is a Semantic Contract? The Interface Between Extraction and Retrieval
- What Does a Semantic Layer Compile To When the Target Is Not SQL?
- How Should an AI Agent Choose Between Retrieval Capabilities?
- Should Semantic Features Be Materialized or Computed at Query Time?
- How Do You Model Unstructured Data From Raw Objects to Serving Contracts?
- What a Semantic Layer Looks Like in Production
- Compute Pushdown: Why Your Vector Search Over Object Storage Is Slow
- How to Merge Data Sources With Different Schemas Into One Searchable Index
- Write-Ahead Logs on Object Storage: How S3 Becomes a Database's Durability Layer
- Query Planning for Multimodal Retrieval: Stage Order, Filter Pushdown, and Candidate Budgets
- Multi-Stage Retrieval: Retrieve, Filter, Rerank
- Change Data Capture for a Search Index: Keeping Downstream Systems in Sync
- Query Rewriting, Expansion and HyDE: Fixing Retrieval Before It Searches
- Cross-Lingual Retrieval: Searching Content in Languages You Did Not Query In
- C2PA Content Credentials: How Media Provenance Actually Works
- How Do You Detect and Redact PII in Images, Video, and Audio?
- How Do You Delete Data From a Vector Index? Embeddings, Tombstones, and the Right to Be Forgotten
- Do You Need a Vector Database? When Brute Force, pgvector, and a Dedicated Store Each Win
- How to Build a Taxonomy From Unlabeled Data: Clustering, Labeling, and Promotion
- What Is Hybrid Search? BM25, Vector Retrieval, and How to Fuse Their Rankings
- What Is MUVERA? Turning Multi-Vector Retrieval Into a Single-Vector Search
- How Do You Run Your Own Model Inside a Managed Search Pipeline?
- How Do You Isolate Tenants in a Vector Index?
- What Does It Cost to Make a Video Library Searchable?
- How to Search, Deduplicate, and Moderate AI-Generated Video and Images (FLUX 3, Kling, Veo, Runway)
- How Does LoRA Fine-Tuning Work? (Adapters, QLoRA, DoRA, and Fine-Tuning Retrieval Models)
- How Do I Evaluate a RAG Pipeline? (Faithfulness, Answer Relevance, Hallucination, and Context Metrics)
- How Do I Ingest Millions of Files into a Search Index? (Images, Video, Documents at Scale)
- How Do AI Agents Search Big Datasets by Navigating Clusters? (Hierarchical Cluster Search)
- What Is Composite Clustering? Clustering Across Multiple Feature Spaces (and Clusters of Clusters)
- How Do I Filter Vector Search Results by Location? (Radius, Bounding Box, Polygon)
- How Do I Automatically Classify Content Against a Taxonomy?
- Brand Safety vs Brand Suitability: How AI Classifies Video for Advertisers
- How Do I Debug Bad Retrieval Results in RAG and Vector Search?
- Can You Run a Vector Database on S3? Object-Storage-Backed Vector Search, Explained
- How Do I Build a Deep Research Agent Over My Own Data?
- What Concepts Exist in My Data That Nobody Has Labeled Yet?
- Marengo 3.0 on Amazon Bedrock: Architecture, Benchmarks and Vector Storage
- How to Store TwelveLabs Marengo Embeddings in Your Own Vector Store
- Video Highlight Detection: How AI Finds the Best Moments
- How Much Does a Vector Database Cost? A 2026 Pricing Comparison
- How to Switch Embedding Models Without Re-Embedding Everything
- Reverse Video Search: How It Works and How to Build One
- Semantic Caching: How Agents Skip Work They Have Already Done
- Efficient Attention: How Models Read Hour-Long Video and Book-Length Documents
- Optical Context Compression: Reading Documents as Images, Not Text
- Multi-Object Tracking: How Agents Follow Objects Across Video Frames
- Monocular Depth Estimation: How Models Infer 3D From a Single Image
- Instance-Level Visual Matching: Finding the Same Object, Not Just Similar Ones
- Face Recognition and Identity Clustering: How Agents Recognize and Group People in Video
- Reasoning Rerankers: How Listwise LLM Rerankers Reorder Retrieval Results
- Retrieval Feedback Loops: Learning to Rank from Clicks, Outcomes, and Agent Interactions
- Matryoshka Representation Learning: Nested Embeddings for Adaptive Multimodal Retrieval
- Creative Ad Analysis for AI Agents: JEPA, Multi-Vector Retrieval, and Signal Fusion
- How to Check if an Image Is Copyrighted
- How Do I Find Out If Someone Reposted My Video?
- How to Check if a Video Is Copyrighted
- How to Check if a Song Is Copyrighted on YouTube
- How to Check if a Picture Is Copyrighted Before Using It
- The Complete Guide to Pre-Publication IP Clearance
- What Is a Multimodal Data Warehouse?
- How to Build a Multimodal Data Warehouse
- Multimodal Data Warehouse Architecture Deep Dive
- Vector Storage Tiering: The Architecture Guide to Hot, Warm, and Cold Vector Data
- How to Build MCP Tools for Multimodal AI Agents
- Embedding Portability and Versioning: Why Your Vectors Are Not as Portable as You Think
- How to Build a Multimodal RAG Pipeline
- Context Engineering for AI Agents
- Video Temporal Grounding: How AI Agents Find Specific Moments in Video
- Multimodal Perception for AI Agents: How to Give Your Agent Eyes, Ears, and Memory
- Open-Vocabulary Object Detection: Teaching AI to Find Anything You Describe
- Audio Feature Extraction: How AI Agents Learn to Hear
- Contrastive Learning: How CLIP, SigLIP, and CLAP Actually Work
- Multi-Stage Retrieval: How AI Agents Search Unstructured Data at Scale
- Visual Document Retrieval: How AI Agents Search Documents Without OCR
- Video Scene Segmentation: How AI Decomposes Continuous Video into Searchable Segments
- Omnimodal Embeddings: One Model for Text, Image, Audio, and Video Retrieval
- Speaker Diarization: How AI Agents Know Who Said What in Audio and Video
- Cross-Encoder Reranking: Why Two-Stage Retrieval Beats One-Stage Search
- MCP Tool Design for Multimodal Search
- Long-Context Video Understanding for Agent Perception
- Structured Extraction from Unstructured Documents: How Vision-Language Models Replace OCR Pipelines
- Late Interaction Retrieval: How ColBERT, ColPali, and ColQwen Search Without Losing Token-Level Detail
- Agentic Retrieval: How AI Agents Search Differently Than Humans
- Embedding Quantization & Compression: Making Vector Search Practical at Billion Scale
- Video RAG: Building Retrieval-Augmented Generation Over Video Corpora
- Evaluating Multimodal Retrieval: Metrics, Benchmarks, and Ground Truth
- Multimodal Chunking Strategies: How to Decompose Video, Audio, Images, and Documents for Search
- How to Build a Video Perception Layer for AI Agents
- Approximate Nearest Neighbor Search: The Algorithms Behind Fast Vector Retrieval
- Instruction-Tuned Embeddings: How Task Prompts Transform Retrieval Quality
- Multi-Index Search Architecture: How to Combine Visual, Audio, and Text Embeddings for Rich Media
- Embedding Space Geometry: Why Cosine Similarity Doesn't Always Mean What You Think
- Object Decomposition and Layered Indexing for AI Agent Perception
- Retrieval Control Planes for AI Agents: Streaming, Cancellation, and Budgets
- Video Anomaly Detection for AI Agents: From Signals to Searchable Events
- Agent Perception Evals: Testing Whether AI Agents Can See, Hear, and Search
- Audio-Visual Retrieval for AI Agents: How to Search What Happened, Not Just What Was Said
- Computer-Use Agent Memory: How to Search Screens, Tools, and UI State
- Production Ingestion Reliability for Agent Perception: Ledgers, Backfills, and Recall Checks
- Mask-Aware Retrieval for AI Agents: Segment First, Search Crops, Then Reason
- Forced Alignment for AI Agents: Word Timestamps, Diarization, and Audio Evidence Search
- Payload Projection for Agentic Vector Search: Field Selection, Evidence Handles, and Context Budgets
- Adaptive Indexing for Agentic Search: Query Logs, Payload Indexes, and Retrieval Routing
- Video Frame Sampling: How Many Frames to Embed and Which Ones to Keep
- Calibrating Similarity Scores: What Cosine Similarity Actually Means for Retrieval
- Chunk Contextualization: How Late Chunking and Contextual Retrieval Fix Broken RAG
- Budget-Aware Multi-Vector Retrieval: How to Make Late Interaction Affordable at Scale
- Diversity-Aware Retrieval: How Agents Avoid Twenty Near-Duplicate Results
- Query Transformation Pipelines: Turning a Raw Agent Question Into Searchable Intent
- Learned Sparse Retrieval and Dense-Sparse Hybrid: Why Agents Need Both Vocabularies
- The Modality Gap: Why Text-to-Image Search Underperforms and How Agents Fix It
- How OCR Actually Works: Detection, Recognition, Reading Order, and Tables
- How ASR Decoding Actually Works: Beam Search, LM Fusion, and Confidence
- Index Freshness and Incremental Updates: How Just-Ingested Content Becomes Searchable
- Filtered Vector Search: How Agents Combine Similarity with Hard Constraints
- How Vision-Language Models Fuse Image and Text Tokens
- Hybrid Search Fusion: How to Combine Dense and Lexical Retrieval Without Breaking Ranking
- Audio Fingerprinting: How Agents Recognize a Specific Recording in Noise
- Embedding Fine-Tuning and Distillation: Teaching an Agent to See and Hear Your Domain
- BM25 and the Inverted Index: The Lexical Retriever Every Hybrid Search Treats as a Black Box
- Perceptual Image Hashing: How Agents Recognize the Same Picture After It Has Been Re-Encoded, Cropped, and Recolored
- Streaming Video Understanding: How Agents Watch an Unbounded Live Feed in Real Time
- Why Does My Document Search Miss the Answer That Is in a Chart or Table?
From concept to production
These guides explain how multimodal perception and retrieval actually work. Mixpeek is the platform that runs them: point it at your storage and get back relevant, timestamped results.