We Benchmarked Every Gemini Flash Generation on Metadata Extraction. The Winner Is Not the One You Want.

Short answer: across seven Gemini Flash generations extracting metadata from 143 real product photos, the F2 spread is 0.035. There is no meaningful winner on the headline metric. Gemini 3.7 Flash leads on recall at 0.291, which decides whether an asset is findable at all. The result that should change a decision is lower down the table: Gemini 3.5 Flash Lite names the exact product type 61% of the time against 3.7 Flash's 42%, at 1502ms versus 3063ms. If you are tagging a catalogue for search, the fifth-ranked model is probably the one you want.

What we measured, and why F2

Every model saw an identical set of 143 real e-commerce product images with an identical prompt, so the model is the only variable. Ground truth is each merchant's own product title, and predicted and reference terms are compared as sets.

F2 is the primary metric because recall and precision are not symmetric here. A term the model fails to emit makes the asset unfindable. A spurious term only adds a little noise to ranking. F1 treats those as equally bad, which does not match what breaks a search experience, so F2 weights recall four times as heavily. We report precision and F1 alongside it rather than asking you to take the weighting on trust.

One thing to hold onto while reading the table: absolute F-scores compress when the reference term set is small. A merchant title carries a handful of scoreable terms, so precision is bounded by how many tags a model emits and F2 drifts toward the ratio of reference size to prediction size. Recall and product-type hit rate carry more signal than the headline number.

Which Gemini Flash model is best for metadata extraction?

The full per-model breakdown, including when to choose and when to avoid each, is on the Gemini Flash metadata extraction comparison.

The lite models trade differently, and that is the finding

3.1 Flash Lite and 3.5 Flash Lite sit mid-table on F2 while leading two axes the full models trail. Their precision runs higher, 0.195 and 0.202 against roughly 0.165 for the full models. They name the exact product type far more often, 57% and 61% against roughly 42%. And they do it at under half the latency.

The mechanism is not mysterious. A lite model writes a shorter, cleaner tag list, so it has fewer chances to emit something wrong and its precision climbs. A terse list also lands on the obvious noun more reliably. If your pipeline needs the right product type on every asset and you are indexing at volume, that combination beats a marginally better F2.

The case for 3.7 Flash is recall. It surfaces more of the merchant's own vocabulary than anything else tested, and vocabulary coverage is what decides whether a search for the customer's own words returns the item.

Do newer generations extract better metadata?

Only marginally, and not on every axis. 3.7 Flash scores 0.241 against 2.5 Flash's 0.223 and the 3 Flash preview's 0.207. That is real movement across four generations, and it is small. The clearer generational gain is speed: 2.5 Flash averaged 4751ms per image while the newer lite generations run near 1500ms. If you are on 2.5 Flash today, the upgrade argument is latency and product-type accuracy rather than a step change in extraction quality.

The embedding side: three model IDs, one usable generation

We also ran the Gemini embedding generations on video retrieval, and the interesting result is what could not be measured. Of the three embedding model IDs, only one is scoreable for video:

• gemini-embedding-001 is text-only. Handing it a video returns 400 INVALID_ARGUMENT because it reads only text and finds none. Scoring it would mean embedding a text description instead of the clip, which is a different task.
• gemini-embedding-2-preview returns byte-identical vectors to gemini-embedding-2: cosine 1.000000, max absolute element difference 0.0, and every one of 60 per-query metric rows matching exactly. It is the same weights.
• gemini-embedding-2 is the one generation that embeds video. On 20 videos and 60 queries with zero errors: NDCG@10 0.764, MRR 0.896, mAP@10 0.702, Recall@1 0.200, at 3072 dimensions.

If you are comparing embedding providers, see the multimodal embedding models comparison for the wider field, and MVS for storing and searching the vectors this produces on object storage.

How to reproduce this

The harness, configs, dataset and raw per-model results are public at github.com/mixpeek/video-embedding-benchmark. Clone it and the numbers above are re-derivable.

Two honesty notes that belong with any benchmark. One model recorded two API failures during the run, and that is reported in its row rather than silently retried. And the F1/F2 here is not a private implementation: it is byte-identical to the scoring behind Mixpeek's evaluation API, verified across every prediction, so a number here matches what you would get scoring the same sets through the API yourself.

The product images come from a brand-protection customer's catalogue and are not published. The manifest and aggregate results are.

Why this matters for search

Metadata quality is a retrieval problem wearing a captioning costume. Every tag a model emits becomes a term someone can search; every term it misses becomes a way the asset cannot be found. That is why the axis to optimise depends on the job. Tagging for discovery favours recall. Tagging for filtering and faceting favours precision and correct typing.

Mixpeek indexes and retrieves over object storage at the token level, so the extraction model is a choice you make rather than a property of the platform. MVS takes vectors you already produce. The managed platform handles extraction and indexing if you would rather not run this comparison yourself. Either way, swapping the extraction model should not mean rewriting your application.

Related reading: our video embedding benchmark for the retrieval-side methodology, the model directory for individual model pages, and extractor docs for wiring one into a pipeline.