Guides
Vendor-neutral, engineer-written guides to the concepts behind multimodal AI: perception, retrieval, embeddings, and the infrastructure agents use to see, hear, and search unstructured data. Learn the idea first; then see how Mixpeek applies it.
121 guides across 18 topics
What Is a Semantic Contract? The Interface Between Extraction and Retrieval
A semantic contract is what an application binds to instead of an index: what one result unit is, which model produced it, what the score means, which fields are filterable and at what path, how fresh it is, and what happens when a stage fails. Includes what counts as a breaking change and a worked declaration.
What Does a Semantic Layer Compile To When the Target Is Not SQL?
A structured semantic layer compiles a named concept into SQL. Over unstructured data it compiles a named capability into a retrieval plan. The operator set, the guarantees a retrieval plan gives up, why the cost model is inverted, why the optimizer has no statistics, and why text-to-SQL does not cover this.
How Should an AI Agent Choose Between Retrieval Capabilities?
Capability-aware retrieval gives an agent a short roster of named, described, costed retrieval contracts instead of raw index access. What each declaration has to carry, how raw vector-search tools degrade silently, the three ways capability selection goes wrong, and how to make a bad choice observable.
Should Semantic Features Be Materialized or Computed at Query Time?
The latency arithmetic that decides it, what the query-time path can actually afford, why staleness and re-extraction are the price of materializing, the hybrid split every production system lands on, and the three problems live semantic federation across arbitrary sources still has to solve.
How Do You Model Unstructured Data From Raw Objects to Serving Contracts?
Five layers, each rebuildable from the one below: raw objects, units, features, entities, contracts. What each owns, why stable unit ids are the whole trick, a blast-radius table for every kind of change, how to keep a re-run from re-paying full extraction, and the layer almost everyone skips.
What a Semantic Layer Looks Like in Production
The semantic-layer architecture for unstructured data, mapped onto one shipped video-search deployment layer by layer: what each layer owns, what it prevented, and the three places this deployment does not prove the thesis. Uses Cutsio's RED R3D and ARRI RAW pipeline, with their published numbers.
How to Check if a Video Is Copyrighted
Five ways to check whether a video is copyrighted before you publish it, compared on what each one catches, what it costs and where it fails: YouTube Content ID, manual rights clearance, frame-by-frame reverse image search, audio fingerprinting, and automated multimodal scanning.
Compute Pushdown: Why Your Vector Search Over Object Storage Is Slow
Searching data that lives in S3 usually means downloading it first. Compute pushdown inverts that: send the predicate to the bytes instead of the bytes to the predicate. Covers what can and cannot be pushed down, why the network is the bottleneck rather than the math, the prior art from Parquet and S3 Select to Aurora, and how to tell whether pushdown would actually help your workload.
How to Merge Data Sources With Different Schemas Into One Searchable Index
Your content lives in three systems and none of them agree on field names. One calls it headline, one calls it name, one calls it asset_title. Covers the four ways teams reconcile that for search, how to design a canonical field set you will not regret, what to do when two sources claim the same target field, how to keep lineage back to the origin record after the merge, and when merging is the wrong answer.
Write-Ahead Logs on Object Storage: How S3 Becomes a Database's Durability Layer
A write-ahead log acknowledges a write the moment it is durable, before any index work happens. Move that log onto S3 or GCS and the bucket becomes the durability layer: recovery, read replicas and change feeds all become readers of one chain of sequenced objects. Covers what a WAL is, why object storage changes the design, the two coordination problems it creates (writer fencing and garbage collection that could delete the only copy of a write), how conditional writes solve them, and the shipped write path in Mixpeek MVS.
Query Planning for Multimodal Retrieval: Stage Order, Filter Pushdown, and Candidate Budgets
Why the same five retrieval stages can differ by two orders of magnitude in cost depending only on their order and their candidate budgets. Covers pre-filter versus post-filter and why naive filtering destroys recall on a graph index, how to measure selectivity when your vector store keeps no statistics, how to size the reranker budget from a recall curve instead of a guess, deduplicating before the expensive stage, and what changes when the rows are video scenes rather than text chunks.
Multi-Stage Retrieval: Retrieve, Filter, Rerank
How a production search pipeline chains retrieval, filtering and reranking, what each stage costs, how many candidates to pass between them, and whether to filter before or after the vector search.
All guides
Every guide in one place, 121 in total.
- What Is a Semantic Contract? The Interface Between Extraction and Retrieval
- What Does a Semantic Layer Compile To When the Target Is Not SQL?
- How Should an AI Agent Choose Between Retrieval Capabilities?
- Should Semantic Features Be Materialized or Computed at Query Time?
- How Do You Model Unstructured Data From Raw Objects to Serving Contracts?
- What a Semantic Layer Looks Like in Production
- Compute Pushdown: Why Your Vector Search Over Object Storage Is Slow
- How to Merge Data Sources With Different Schemas Into One Searchable Index
- Write-Ahead Logs on Object Storage: How S3 Becomes a Database's Durability Layer
- Query Planning for Multimodal Retrieval: Stage Order, Filter Pushdown, and Candidate Budgets
- Multi-Stage Retrieval: Retrieve, Filter, Rerank
- Change Data Capture for a Search Index: Keeping Downstream Systems in Sync
- Query Rewriting, Expansion and HyDE: Fixing Retrieval Before It Searches
- Cross-Lingual Retrieval: Searching Content in Languages You Did Not Query In
- C2PA Content Credentials: How Media Provenance Actually Works
- How Do You Detect and Redact PII in Images, Video, and Audio?
- How Do You Delete Data From a Vector Index? Embeddings, Tombstones, and the Right to Be Forgotten
- Do You Need a Vector Database? When Brute Force, pgvector, and a Dedicated Store Each Win
- How to Build a Taxonomy From Unlabeled Data: Clustering, Labeling, and Promotion
- What Is Hybrid Search? BM25, Vector Retrieval, and How to Fuse Their Rankings
- What Is MUVERA? Turning Multi-Vector Retrieval Into a Single-Vector Search
- How Do You Run Your Own Model Inside a Managed Search Pipeline?
- How Do You Isolate Tenants in a Vector Index?
- What Does It Cost to Make a Video Library Searchable?
- How to Search, Deduplicate, and Moderate AI-Generated Video and Images (FLUX 3, Kling, Veo, Runway)
- How Does LoRA Fine-Tuning Work? (Adapters, QLoRA, DoRA, and Fine-Tuning Retrieval Models)
- How Do I Evaluate a RAG Pipeline? (Faithfulness, Answer Relevance, Hallucination, and Context Metrics)
- How Do I Ingest Millions of Files into a Search Index? (Images, Video, Documents at Scale)
- How Do AI Agents Search Big Datasets by Navigating Clusters? (Hierarchical Cluster Search)
- What Is Composite Clustering? Clustering Across Multiple Feature Spaces (and Clusters of Clusters)
- How Do I Filter Vector Search Results by Location? (Radius, Bounding Box, Polygon)
- How Do I Automatically Classify Content Against a Taxonomy?
- Brand Safety vs Brand Suitability: How AI Classifies Video for Advertisers
- How Do I Debug Bad Retrieval Results in RAG and Vector Search?
- Can You Run a Vector Database on S3? Object-Storage-Backed Vector Search, Explained
- How Do I Build a Deep Research Agent Over My Own Data?
- What Concepts Exist in My Data That Nobody Has Labeled Yet?
- How to Store TwelveLabs Marengo Embeddings in Your Own Vector Store
- Video Highlight Detection: How AI Finds the Best Moments
- How Much Does a Vector Database Cost? A 2026 Pricing Comparison
- How to Switch Embedding Models Without Re-Embedding Everything
- Reverse Video Search: How It Works and How to Build One
- Semantic Caching: How Agents Skip Work They Have Already Done
- Efficient Attention: How Models Read Hour-Long Video and Book-Length Documents
- Optical Context Compression: Reading Documents as Images, Not Text
- Multi-Object Tracking: How Agents Follow Objects Across Video Frames
- Monocular Depth Estimation: How Models Infer 3D From a Single Image
- Instance-Level Visual Matching: Finding the Same Object, Not Just Similar Ones
- Face Recognition and Identity Clustering: How Agents Recognize and Group People in Video
- Reasoning Rerankers: How Listwise LLM Rerankers Reorder Retrieval Results
- Retrieval Feedback Loops: Learning to Rank from Clicks, Outcomes, and Agent Interactions
- Matryoshka Representation Learning: Nested Embeddings for Adaptive Multimodal Retrieval
- Creative Ad Analysis for AI Agents: JEPA, Multi-Vector Retrieval, and Signal Fusion
- How to Check if an Image Is Copyrighted
- How to Check if a Video Is Copyrighted
- How to Check if a Song Is Copyrighted on YouTube
- How to Check if a Picture Is Copyrighted Before Using It
- The Complete Guide to Pre-Publication IP Clearance
- What Is a Multimodal Data Warehouse?
- How to Build a Multimodal Data Warehouse
- Multimodal Data Warehouse Architecture Deep Dive
- Vector Storage Tiering: The Architecture Guide to Hot, Warm, and Cold Vector Data
- How to Build MCP Tools for Multimodal AI Agents
- Embedding Portability and Versioning: Why Your Vectors Are Not as Portable as You Think
- How to Build a Multimodal RAG Pipeline
- Context Engineering for AI Agents
- Video Temporal Grounding: How AI Agents Find Specific Moments in Video
- Multimodal Perception for AI Agents: How to Give Your Agent Eyes, Ears, and Memory
- Open-Vocabulary Object Detection: Teaching AI to Find Anything You Describe
- Audio Feature Extraction: How AI Agents Learn to Hear
- Contrastive Learning: How CLIP, SigLIP, and CLAP Actually Work
- Multi-Stage Retrieval: How AI Agents Search Unstructured Data at Scale
- Visual Document Retrieval: How AI Agents Search Documents Without OCR
- Video Scene Segmentation: How AI Decomposes Continuous Video into Searchable Segments
- Omnimodal Embeddings: One Model for Text, Image, Audio, and Video Retrieval
- Speaker Diarization: How AI Agents Know Who Said What in Audio and Video
- Cross-Encoder Reranking: Why Two-Stage Retrieval Beats One-Stage Search
- MCP Tool Design for Multimodal Search
- Long-Context Video Understanding for Agent Perception
- Structured Extraction from Unstructured Documents: How Vision-Language Models Replace OCR Pipelines
- Late Interaction Retrieval: How ColBERT, ColPali, and ColQwen Search Without Losing Token-Level Detail
- Agentic Retrieval: How AI Agents Search Differently Than Humans
- Embedding Quantization & Compression: Making Vector Search Practical at Billion Scale
- Video RAG: Building Retrieval-Augmented Generation Over Video Corpora
- Evaluating Multimodal Retrieval: Metrics, Benchmarks, and Ground Truth
- Multimodal Chunking Strategies: How to Decompose Video, Audio, Images, and Documents for Search
- How to Build a Video Perception Layer for AI Agents
- Approximate Nearest Neighbor Search: The Algorithms Behind Fast Vector Retrieval
- Instruction-Tuned Embeddings: How Task Prompts Transform Retrieval Quality
- Multi-Index Search Architecture: How to Combine Visual, Audio, and Text Embeddings for Rich Media
- Embedding Space Geometry: Why Cosine Similarity Doesn't Always Mean What You Think
- Object Decomposition and Layered Indexing for AI Agent Perception
- Retrieval Control Planes for AI Agents: Streaming, Cancellation, and Budgets
- Video Anomaly Detection for AI Agents: From Signals to Searchable Events
- Agent Perception Evals: Testing Whether AI Agents Can See, Hear, and Search
- Audio-Visual Retrieval for AI Agents: How to Search What Happened, Not Just What Was Said
- Computer-Use Agent Memory: How to Search Screens, Tools, and UI State
- Production Ingestion Reliability for Agent Perception: Ledgers, Backfills, and Recall Checks
- Mask-Aware Retrieval for AI Agents: Segment First, Search Crops, Then Reason
- Forced Alignment for AI Agents: Word Timestamps, Diarization, and Audio Evidence Search
- Payload Projection for Agentic Vector Search: Field Selection, Evidence Handles, and Context Budgets
- Adaptive Indexing for Agentic Search: Query Logs, Payload Indexes, and Retrieval Routing
- Video Frame Sampling: How Many Frames to Embed and Which Ones to Keep
- Calibrating Similarity Scores: What Cosine Similarity Actually Means for Retrieval
- Chunk Contextualization: How Late Chunking and Contextual Retrieval Fix Broken RAG
- Budget-Aware Multi-Vector Retrieval: How to Make Late Interaction Affordable at Scale
- Diversity-Aware Retrieval: How Agents Avoid Twenty Near-Duplicate Results
- Query Transformation Pipelines: Turning a Raw Agent Question Into Searchable Intent
- Learned Sparse Retrieval and Dense-Sparse Hybrid: Why Agents Need Both Vocabularies
- The Modality Gap: Why Text-to-Image Search Underperforms and How Agents Fix It
- How OCR Actually Works: Detection, Recognition, Reading Order, and Tables
- How ASR Decoding Actually Works: Beam Search, LM Fusion, and Confidence
- Index Freshness and Incremental Updates: How Just-Ingested Content Becomes Searchable
- Filtered Vector Search: How Agents Combine Similarity with Hard Constraints
- How Vision-Language Models Fuse Image and Text Tokens
- Hybrid Search Fusion: How to Combine Dense and Lexical Retrieval Without Breaking Ranking
- Audio Fingerprinting: How Agents Recognize a Specific Recording in Noise
- Embedding Fine-Tuning and Distillation: Teaching an Agent to See and Hear Your Domain
- BM25 and the Inverted Index: The Lexical Retriever Every Hybrid Search Treats as a Black Box
- Perceptual Image Hashing: How Agents Recognize the Same Picture After It Has Been Re-Encoded, Cropped, and Recolored
- Streaming Video Understanding: How Agents Watch an Unbounded Live Feed in Real Time
From concept to production
These guides explain how multimodal perception and retrieval actually work. Mixpeek is the platform that runs them: point it at your storage and get back relevant, timestamped results.