Search & Discovery

7 min read

Updated 2026-10-10

# Why Does AI Miss Things When I Ask It About a Long Video?

AI misses things in a long video because it only sees sampled frames. Gemini reads one frame per second by default, and Claude's and OpenAI's APIs take images only, capped at 600 and 1,500 per request, while an hour of video at one frame per second is 3,600 frames. The fix is to find the minutes your question is about first and send only those, at a higher frame rate.

![Ethan Steininger](/people/ethan.webp)

By [Ethan Steininger](/blog/authors/ethan), Founder & CEO, Mixpeek

Video Understanding

Vision Language Models

Long Video

Frame Sampling

Video Search

[Run this on your own data. Try it as you read.Search your own archive →](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)

[One-click templateFootage Intelligence at Archive ScaleMillions of scenes you can move through instead of search. Footage, ads and editor assets land in three clean collections, and scenes are clustered one partition at a time so every run stays under the sample-size caps.Deploy it](/templates/footage-intelligence)

## Why does AI miss things when I ask it about a long video?

  
  
The model only sees some of the frames. A model that accepts video samples frames from it, and a model that accepts images only sees the frames you send, so the moment you are asking about can fall between them. Gemini samples one frame per second by default, and its documentation notes that fast action can lose detail at that rate. At one frame per second, an hour of video is 3,600 frames, more than Claude's API takes in one request (up to 600 images) or OpenAI's (up to 1,500). The fix that holds up is to find the minutes your question is about first, then send only those to the model at a higher frame rate.  
  

## How much of a long video does the model actually see?

  
  
What each API documents for video input, and what a one-hour video becomes under those limits. The last column is arithmetic on the documented numbers.  
  

API

How video goes in

Documented limit

A one-hour video

Gemini API

Video file, sampled at 1 frame per second by default; a custom fps can be set

258 tokens per frame (66 at low media resolution) plus 32 audio tokens per second; a 1M-token context holds about 1 hour at 258 tokens per frame or 3 hours at low

3,600 frames, about 1.04M tokens with audio at 258 tokens per frame

Claude API

Images only, no video input

600 images per request (100 on models with a 200k context, 20 per message in claude.ai); a frame costs ceil(width/28) x ceil(height/28) tokens

At most one frame every 6 seconds; a 1280x720 frame is 1,196 tokens, so 600 frames is about 718,000

OpenAI API

Images only in the vision guide, which lists no video input

1,500 images per request, 512 MB per request

At most one frame every 2.4 seconds

  
  
Sources: [Gemini video understanding](https://ai.google.dev/gemini-api/docs/video-understanding), [Claude vision](https://platform.claude.com/docs/en/build-with-claude/vision), [OpenAI images and vision](https://developers.openai.com/api/docs/guides/images-vision). Checked 2026-10-10.  
  

## Why do short events get missed?

  
  
At one frame per second, anything shorter than a second can land between two samples: a logo on screen for half a second, a jersey number in a fast pan, a one-shot cutaway. When you pick the frames yourself to stay under an image cap, the gap grows. Spread 600 frames across an hour and the model sees one every six seconds.  
  
Sound is a separate channel. Gemini reads the audio track with the video. An API that takes images hears nothing, so anything that was only said is gone unless you add a transcript as text.  
  

## Why do answers get vaguer as the video gets longer?

  
  
A longer input puts more frames in front of the model for the same question, and most of them are unrelated to it. Research on long text inputs found that models use information at the start and end of the input more reliably than information in the middle ([Liu et al., "Lost in the Middle", 2023](https://arxiv.org/abs/2307.03172)). A detail from minute 34 of a 60-minute video sits in the middle of the input.  
  
Cost moves the same way. Every frame is billed as input tokens on every question you ask, so asking ten questions about an hour of video pays for that hour ten times.  
  

## How do I get reliable answers about a long video?

  
  
Search first, then ask.  
  
1\. **Split the video into segments with timestamps.** Cut at scene changes for edited footage and at fixed windows for lectures, meetings or camera feeds. The [video scene segmentation guide](/guides/video-scene-segmentation) covers the trade-offs. 2. **Index each segment.** Store an embedding of its frames, its transcript, and any text shown on screen. On-screen text needs its own step; [why video search misses on-screen text](/guides/why-video-search-misses-on-screen-text) explains why. 3. **Search for the part your question is about.** Run the question against the index and keep the top few segments, with their start and end times. 4. **Send only those segments to the model, sampled densely.** A two-minute clip at several frames per second shows the model far more of that moment than the whole hour at one frame per second. 5. **Ask with the timestamps attached**, so the answer can say where in the video it found each claim and you can check it.  
  
For a question about the whole video, such as a summary, summarize each segment first and then summarize those summaries. Each call then sees a short span in full.  
  

## When is it fine to send the whole video?

  
  
When the video is short, when the question is about the overall gist, or when you need the model to follow events across the entire timeline in one pass. Gemini can take about an hour in one request at 258 tokens per frame, and about three at low media resolution. Once you ask about specific moments, or ask many questions about the same footage, searching first is more accurate and cheaper.  
  

## How do I do this with Mixpeek?

  
  
Mixpeek reads your files where they already live, in S3, GCS or any bucket, and turns what's in them into data your software can search, classify and moderate. The [video scene search recipe](/recipes/video-scene-search) does steps 1 to 3: it splits each video at scene changes, embeds each scene's frames and transcript, and returns each match as the video's ID with the scene's start and end time. Cut those spans out and send them to the model with your question.

import subprocess
from mixpeek import Mixpeek

client = Mixpeek(api\_key="YOUR\_API\_KEY", namespace="video-scenes")

# The retriever from the video scene search recipe
results = client.retrievers.execute(
    "ret\_your\_scene\_retriever",
    inputs={"query": "the speaker walks through the revenue chart"},
)

for i, doc in enumerate(results\["documents"\]\[:3\]):
    start, end = doc\["start\_time"\], doc\["end\_time"\]
    # Re-encode so the clip starts exactly at the scene boundary
    subprocess.run(
        \["ffmpeg", "-y", "-ss", str(start), "-to", str(end), "-i", "talk.mp4", f"clip\_{i}.mp4"\],
        check=True,
    )
    print(f"clip\_{i}.mp4 covers {start}s to {end}s")

# Send clip\_0.mp4 to clip\_2.mp4 to your vision model with the question and these timestamps

Mixpeek fits teams with a video library in object storage who ask many questions of the same footage. It is more than you need for asking one question about one short clip; send that clip straight to the model. Prices are on the [pricing page](/pricing). For choosing the model that reads the clips, see the [best vision language models](/curated-lists/best-vision-language-models) and [best AI video analysis tools](/curated-lists/best-ai-video-analysis-tools). For the research side of long video, see [long-context video understanding](/guides/long-context-video-understanding).  
  

## Frequently Asked Questions

  
  

### Why does AI miss things when I ask it about a long video?

  
  
It only sees sampled frames. Gemini reads one frame per second by default, and image-only APIs see just the frames you send, capped at 600 per request on Claude and 1,500 on OpenAI. Anything between those frames, or anything only spoken when no transcript is sent, is invisible to the model.  
  

### Can ChatGPT or Claude watch a whole video?

  
  
Their APIs take images, so you send frames from the video. Claude's API takes up to 600 images per request and OpenAI's up to 1,500, which across an hour of video is one frame every 6 or 2.4 seconds at best. Gemini's API takes a video file directly and samples it at one frame per second.  
  

### How many frames per second does Gemini read from a video?

  
  
One frame per second by default, at 258 tokens per frame (66 at low media resolution) plus 32 tokens per second of audio. You can set a custom frame rate, and the documentation notes that fast action can lose detail at the default.  
  

### What is the best way to ask questions about a long video?

  
  
Find the part of the video your question is about first, then send only that part to the model at a higher frame rate, with its timestamps. Splitting the video into segments, indexing each one, and searching that index does the finding.

## Run this on your own data

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this guide describes. Build starts at $25/mo for up to 1M vectors.

[Search your own archive](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)[Read Docs](/docs)

[Explore MVS](/mvs)[See pricing](/pricing)

## Related guides

[

Search & Discovery

### How to Search Inside Videos in iconik with Natural Language

iconik search matches metadata, so finding a moment by what is on screen needs an index built from the footage. Compare iconik's own AI search, the iconik and TwelveLabs connector, and syncing iconik to Mixpeek, with setup steps, costs and FAQs.

Read guide →](/guides/iconik-video-search)[

Search & Discovery

### What Is AI Video Analysis? How It Works, and How to Find a Moment in a Video

AI video analysis turns video into data you can query: what is on screen, what is said, what text appears and when each happens. How it works in five steps, a request you can run against a public footage library with no key, and how Mixpeek, Twelve Labs, Google Video Intelligence, Amazon Rekognition and an open-source CLIP pipeline compare on price and on what they return.

Read guide →](/guides/ai-video-analysis)[

Search & Discovery

### Can You Do a Reverse Video Search on Google?

Not directly: Google has no upload-a-video search. Screenshot a distinctive frame and search it with Google Lens, or use Circle to Search on Android, then open the earliest upload among the matches. Why frame search misses some videos, what Lens does with a recorded clip, and how to search a library you own.

Read guide →](/guides/can-you-do-a-reverse-video-search-on-google)