What is AI video analysis?
AI video analysis is the use of machine learning models to turn video into data you can query: what is on screen, what is said, what text appears, who appears, and when each of those happens. A model reads sampled frames and the audio track, and the output is labels, transcripts, descriptions or embeddings tied to timestamps. The most common reason to do it is to find a moment: you describe the moment in words, or show a clip of it, and get back the video and the second it starts.
This page covers how it works, a request you can run right now against a public footage library, how the main tools compare, and where the approach breaks.
How does AI video analysis work?
Every tool on the market runs some version of the same five steps. They differ in which models they use at each step and in what they keep afterwards.
1. Decode and segment. The video is split into frames and into segments: fixed windows (every 6 seconds, say), shots cut at visual changes, or scenes. Segments are what a search returns, so their length sets how precise an answer can be. 2. Sample frames. Most pipelines read 1 frame per second, some read every shot's keyframe. Sampling is the main cost lever, and it is also why very fast events get missed. See how video frame sampling works. 3. Extract signals. Models run per segment: a vision model for objects and actions, speech recognition for the transcript, OCR for on-screen text, face and logo detectors, an audio classifier for sounds. A vision-language model can also write a plain-language description of the segment. 4. Embed and index. Each segment gets one or more embedding vectors (a list of numbers that places similar content close together) plus its metadata: start and end time, transcript, labels. These go into a vector index. 5. Retrieve. A query is embedded the same way and compared against every segment. The closest segments come back with their timestamps, optionally filtered by metadata and grouped so one video does not fill the whole result list.
An analysis API stops at step 3 and hands you JSON for one video. A search system keeps going through step 5, which is what makes "find every clip where this happens" possible across a library.
How do I find a moment in a video with AI?
Index the library once into segments with embeddings, then search it with a sentence. The request below runs against a public sample library of 67 clips (63 from Pexels, 4 from NASA), cut into 6-second segments and indexed with Mixpeek's multimodal extractor. It needs no API key.
curl -s -X POST https://api.mixpeek.com/v1/public/retrievers/footage-intelligence-sample/execute \
-H 'Content-Type: application/json' \
-d '{"inputs":{"query":"police car lights flashing at night"},"pagination":{"method":"offset","page_number":1,"page_size":3}}'The same call in Python, printing one line per moment:
import requests
r = requests.post(
"https://api.mixpeek.com/v1/public/retrievers/footage-intelligence-sample/execute",
json={
"inputs": {"query": "police car lights flashing at night"},
"pagination": {"method": "offset", "page_number": 1, "page_size": 3},
},
timeout=30,
)
r.raise_for_status()
for m in r.json()["results"]:
print(f'{m["start_time"]:5.1f}s to {m["end_time"]:5.1f}s score {m["score"]:.4f} {m["title"]}')What it printed when we ran it on October 18, 2026:
0.0s to 7.0s score 0.1345 Close up video of police car lights 19.0s to 25.0s score 0.1118 Aerial nighttime traffic stop with police lights 31.0s to 37.0s score 0.0766 A car is driving down a road in the dark
The first two results are right: a close-up of a police light bar and an aerial shot of a night traffic stop, 19 seconds into its clip. The third is a car on a dark road with no police in it, the closest thing left in a small library. Each result also carries the segment's description, its transcript, a thumbnail and a URL that plays just that segment.
Two details matter when you build on this. The scores are raw cosine similarity between the query text and the segment, so they rank results within one query and should not be compared across queries: here a correct answer scored 0.13, and on the query "product shown on a white background" the correct top answer scored 0.065. And the retriever groups results by source video and keeps one moment per video, so the top three are three different clips.
You can run your own queries on the same library at the footage intelligence template or on its public retriever page. These are the 8 suggested queries and how they scored when each result was checked against frames pulled from the source clip:
The top result was right on all 8. The misses in places 2 and 3 are near neighbors (a horse galloping at sunset for the dog query, latte art for the pouring query) because a 67-clip library often has only one or two real answers. One query was dropped from the set: "car chase through city streets" has no correct answer in this library, and it still returned results, the aerial police stop and a night highway timelapse. A search always returns its closest matches, so when the library holds no answer you see low scores on unrelated results.
The tools below cover the four ways teams usually do this: a video-native API, a cloud analysis API, an AWS service, and building it yourself from open-source models. Prices are list prices on the vendors' own pages on October 18, 2026.
The other tools buyers ask about are covered in the best AI video analysis tools: the Gemini API for open-ended questions about one clip (about 100 tokens per second of video plus 32 for audio, roughly $0.006 per minute on Gemini 3.8 Flash), Azure Video Indexer ($0.045 to $0.15 per minute), and Amazon Bedrock Data Automation ($0.050 per minute for standard output). For the feature-by-feature version of the most common pairing, see Mixpeek vs Twelve Labs.
The build-it-yourself path is real and common. A minimal version is ffmpeg pulling one frame per second, a CLIP or SigLIP model embedding each frame, and the vectors going into FAISS, pgvector or another vector database with the video id and timestamp alongside. A text query is embedded with the same model's text encoder and ranked by cosine similarity.
That gets you visual search at no software cost. What it leaves to you: speech (add Whisper and a text index), on-screen text (add OCR), grouping frames into moments so one shot does not return 20 near-identical hits, motion and action, which per-frame models cannot see, and re-indexing everything when you change the embedding model. Those are the parts most teams end up building over months, and they are the parts a managed system provides. See why video search returns the wrong scene for the failure modes.
Fast events between sampled frames. At 1 frame per second, a 0.3-second event may never be seen. Raise the rate for that footage or use a model that reads short clips.
Text on screen. Small or fast-moving text needs OCR; an embedding of the frame rarely captures it. See why video search misses on-screen text.
Long videos. A model that reads a whole video at once gets less precise as the video grows. See why AI misses things in a long video.
Scores across queries. Similarity scores move with the query, so a fixed cutoff drops real answers on some queries and keeps wrong ones on others.
Cost of reprocessing. Changing a model means re-running the library. Keep the source files in your own storage so a re-index is a batch job over files you already hold.
Video-native APIs (Twelve Labs), general multimodal models that accept video (the Gemini API), cloud analysis services (Google Video Intelligence, Azure Video Indexer, Amazon Rekognition, Amazon Bedrock Data Automation), and search platforms that index video for retrieval (Mixpeek). You can also build your own from open-source models such as CLIP, SigLIP and Whisper. The right one depends on whether you need to understand one video or search many.
Several have free tiers: the Gemini API's Flash models, Google Video Intelligence's first 1,000 minutes a month, Amazon Rekognition's 60 free minutes a month for the first year, and Twelve Labs' free plan. Open-source models are free to run on your own hardware. The sample library on this page can be searched with no account.
The Gemini API accepts a public YouTube URL directly and can answer questions about the video. Most other tools need the file itself, which means downloading it first, and that is only appropriate for footage you have the rights to use.
Indexing usually runs at a fraction of the video's length to a few times its length, depending on how many signals you extract and how much runs in parallel. A search against an indexed library is a single query; on the public sample above it took between 1.1 and 4.5 seconds.
Analysis turns one video into data: labels, a transcript, a summary. Search runs a question across many analyzed videos and returns the moments that match. Most products start with analysis and need search as soon as the library grows past what a person can browse. For searching with a clip as the query, see how reverse video search works, and for one moment in one long recording, see how to find one specific moment in hours of footage.
Sample library, its index settings and the 8 query checks: the footage intelligence sample, built October 15, 2026 (67 clips under the Pexels License and NASA media guidelines).
Twelve Labs pricing
Google Cloud Video Intelligence pricing
Amazon Rekognition pricing
Gemini API pricing and video understanding
Azure AI Video Indexer pricing
Amazon Bedrock pricing
Mixpeek pricing
Two details matter when you build on this. The scores are raw cosine similarity between the query text and the segment, so they rank results within one query and should not be compared across queries: here a correct answer scored 0.13, and on the query "product shown on a white background" the correct top answer scored 0.065. And the retriever groups results by source video and keeps one moment per video, so the top three are three different clips.
Try it on a public footage library
You can run your own queries on the same library at the footage intelligence template or on its public retriever page. These are the 8 suggested queries and how they scored when each result was checked against frames pulled from the source clip:
| Query | Correct in top 3 | Top result |
|---|---|---|
| police car lights flashing at night | 2 of 3 | Close-up of a police light bar flashing, 0 to 7 s |
| a dog running on a beach | 2 of 3 | Dog playing with waves, 0 to 7 s |
| someone pouring coffee | 2 of 3 | Ground coffee poured into a pour-over dripper, 0 to 7 s |
| a goal celebration | 2 of 3 | Player kneeling and shouting in front of the goal, 0 to 7 s |
| product shown on a white background | 2 of 3 | Cosmetic bottles on a white surface, 0 to 7 s |
| aerial view of a coastline | 3 of 3 | Drone shot over a rocky coastline, 7 to 12.5 s |
| crowd cheering at night | 2 of 3 | Concert crowd with hands up, 0 to 7 s |
| a logo on screen | 3 of 3 | The "pexels" wordmark on a phone screen, 0 to 7 s |
The top result was right on all 8. The misses in places 2 and 3 are near neighbors (a horse galloping at sunset for the dog query, latte art for the pouring query) because a 67-clip library often has only one or two real answers. One query was dropped from the set: "car chase through city streets" has no correct answer in this library, and it still returned results, the aerial police stop and a night highway timelapse. A search always returns its closest matches, so when the library holds no answer you see low scores on unrelated results.
How do the main tools compare?
The tools below cover the four ways teams usually do this: a video-native API, a cloud analysis API, an AWS service, and building it yourself from open-source models. Prices are list prices on the vendors' own pages on October 18, 2026.
| Mixpeek | Twelve Labs | Google Video Intelligence | Amazon Rekognition Video | Open-source CLIP pipeline | |
|---|---|---|---|---|---|
| What you get back | Timestamped moments ranked across your library | Timestamped moments from its hosted index, plus summaries from Pegasus | Labels, shots, text, objects and faces with timestamps, for one video per request | Labels, faces and celebrities with timestamps, for one video per job | Frame embeddings you rank yourself |
| Search across many videos | Yes | Yes, within an index you upload to | No | Face search against a face collection only | Yes, once you add a vector database |
| Signals | Visual, speech, on-screen text, faces, audio, descriptions | Visual, audio and on-screen text through Marengo | Labels, shots, text, objects, logos, faces, speech | Labels, faces, celebrities, text, moderation | Visual only unless you add speech and OCR models |
| Where the footage lives | Your S3, GCS or Azure bucket | Uploaded to Twelve Labs | Google Cloud Storage | Amazon S3 | Wherever you run it |
| Price | Video $0.05 per minute, plans from $25 a month | Marengo 3.0 video $2.50 per hour, search $4 per 1,000 queries | First 1,000 minutes a month free, then $0.10 per minute for labels | $0.10 per minute for labels or face search | Free software; you pay for GPUs and the database |
| Self-hosted | Licensing available | No | No | No | Yes |
The other tools buyers ask about are covered in the best AI video analysis tools: the Gemini API for open-ended questions about one clip (about 100 tokens per second of video plus 32 for audio, roughly $0.006 per minute on Gemini 3.8 Flash), Azure Video Indexer ($0.045 to $0.15 per minute), and Amazon Bedrock Data Automation ($0.050 per minute for standard output). For the feature-by-feature version of the most common pairing, see Mixpeek vs Twelve Labs.
What does an open-source pipeline involve?
The build-it-yourself path is real and common. A minimal version is ffmpeg pulling one frame per second, a CLIP or SigLIP model embedding each frame, and the vectors going into FAISS, pgvector or another vector database with the video id and timestamp alongside. A text query is embedded with the same model's text encoder and ranked by cosine similarity.
That gets you visual search at no software cost. What it leaves to you: speech (add Whisper and a text index), on-screen text (add OCR), grouping frames into moments so one shot does not return 20 near-identical hits, motion and action, which per-frame models cannot see, and re-indexing everything when you change the embedding model. Those are the parts most teams end up building over months, and they are the parts a managed system provides. See why video search returns the wrong scene for the failure modes.
Where does AI video analysis go wrong?
Frequently asked questions
What AI can analyze videos?
Video-native APIs (Twelve Labs), general multimodal models that accept video (the Gemini API), cloud analysis services (Google Video Intelligence, Azure Video Indexer, Amazon Rekognition, Amazon Bedrock Data Automation), and search platforms that index video for retrieval (Mixpeek). You can also build your own from open-source models such as CLIP, SigLIP and Whisper. The right one depends on whether you need to understand one video or search many.
Is there a free AI that can analyze videos?
Several have free tiers: the Gemini API's Flash models, Google Video Intelligence's first 1,000 minutes a month, Amazon Rekognition's 60 free minutes a month for the first year, and Twelve Labs' free plan. Open-source models are free to run on your own hardware. The sample library on this page can be searched with no account.
Can AI analyze a YouTube video?
The Gemini API accepts a public YouTube URL directly and can answer questions about the video. Most other tools need the file itself, which means downloading it first, and that is only appropriate for footage you have the rights to use.
How long does AI video analysis take?
Indexing usually runs at a fraction of the video's length to a few times its length, depending on how many signals you extract and how much runs in parallel. A search against an indexed library is a single query; on the public sample above it took between 1.1 and 4.5 seconds.
What is the difference between video analysis and video search?
Analysis turns one video into data: labels, a transcript, a summary. Search runs a question across many analyzed videos and returns the moments that match. Most products start with analysis and need search as soon as the library grows past what a person can browse. For searching with a clip as the query, see how reverse video search works, and for one moment in one long recording, see how to find one specific moment in hours of footage.