The $0 Image Tagger: Test-Time Compute on a Frozen Embedding Model
Summary
A frozen 1B-parameter embedding model becomes an open-vocabulary image tagger with zero training: no fine-tune, no classifier head, no curated tag list. Score every 16px patch against the model's own 25,000-word vocabulary: the global vector alone lands at 0.26 mAP (COCO-150), max-over-patches reaches 0.64 (late interaction recovered from a frozen forward pass), and re-encoding 14 crops hits 0.71 with 81% precision at 1. De-biasing subtracts each word's background score; embedding-NMS merges synonyms across languages. The finding that matters: calibration tricks (whitening, label propagation) barely moved the pipeline, while new input (patches, crops) added 37 then 7 more mAP points. Mixpeek taxonomies run tagging the same way: classification is retrieval, so you swap the label set and nothing retrains. Explore: mixpeek.com/recipes/image-tagging-pipeline
About this video
A frozen 1B-parameter embedding model becomes an open-vocabulary image tagger with zero training: no fine-tune, no classifier head, no curated tag list. Score every 16px patch against the model's own 25,000-word vocabulary: the global vector alone lands at 0.26 mAP (COCO-150), max-over-patches reaches 0.64 (late interaction recovered from a frozen forward pass), and re-encoding 14 crops hits 0.71 with 81% precision at 1. De-biasing subtracts each word's background score; embedding-NMS merges synonyms across languages. The finding that matters: calibration tricks (whitening, label propagation) barely moved the pipeline, while new input (patches, crops) added 37 then 7 more mAP points. Mixpeek taxonomies run tagging the same way: classification is retrieval, so you swap the label set and nothing retrains. Explore: mixpeek.com/recipes/image-tagging-pipeline