The Short Answer
Automatically classifying content against a taxonomy means assigning each item — an image, a video, a document, an audio clip — to one or more predefined categories, at scale, without a human labeling each one. In 2026 there are four viable methods, and the right choice depends on how stable your taxonomy is and how much labeled data you have: zero-shot classification with a vision-language or text model (no training, flexible, good enough for most catalogs), embedding-similarity classification (embed the item and each category, assign by nearest centroid — cheap and updatable), a trained classifier head (highest accuracy when you have thousands of labeled examples per class), and LLM classification (a prompt that returns categories, best for nuanced or hierarchical taxonomies). Most production systems combine them: embeddings or zero-shot for the broad pass, an LLM or trained head for the hard cases.
Taxonomy classification vs the things it's confused with
Three different problems wear similar names. Getting them straight saves a lot of wasted modeling:
| Task | Question it answers | You supply |
| Taxonomy classification | "Which of MY categories does this belong to?" | A fixed taxonomy + the content |
| Discovered taxonomy (clustering) | "What categories even exist in my data?" | Just the content — the structure is discovered |
| Zero-shot classification | "Does this match this arbitrary label?" | Content + any label string, no training |
| Entity/metadata extraction | "What specific things are named in this?" | Content — output is spans, not categories |
The four methods, and when each wins
Zero-shot (VLM/LLM against label strings). Embed or prompt the model with your category names and let it score the item against each. No training, no labeled data, and you can change the taxonomy by editing a list of strings. It is the right default for a new or evolving taxonomy, for long-tail categories with few examples, and for multimodal content (a CLIP-style model classifies an image against "kitchenware" / "outerwear" with no product tags). Weakness: accuracy on subtle or domain-specific distinctions, and cost if you score every item against hundreds of labels.
Embedding-similarity (nearest-centroid). Embed each category (from its name, a description, or a few examples) into the same space as your content, then classify by cosine similarity to the nearest category centroid. Cheap at query time, trivially updatable (add a category = add a vector), and it reuses the embeddings you already computed for search. This is the workhorse for large catalogs where the taxonomy shifts and re-training is impractical. Weakness: it inherits the modality gap and calibration quirks of the embedding space — a raw cosine score is not a probability.
Trained classifier head. With thousands of labeled examples per class, a supervised classifier (a linear head on frozen embeddings, or a fine-tuned model) beats zero-shot on accuracy and is fast at inference. The right choice for a stable, high-value taxonomy where you can afford the labeling. Weakness: every taxonomy change means relabeling and retraining, so it is the wrong tool while your categories are still moving.
LLM classification. Give a model the content (or its extracted text/captions) plus the taxonomy and ask for the categories, optionally with reasoning. Best for hierarchical taxonomies, nuanced treatment-sensitive categories (the difference between reporting on an event and glorifying it — the crux of brand suitability), and taxonomies that need a justification per label. Weakness: cost and latency at scale, and the need to constrain output to valid categories.
How do you classify non-text content — images, video, audio?
The trap is treating everything as text. A robust multimodal classifier decomposes each item into the signals the taxonomy actually depends on, then classifies their combination. For a video: visual frames/scenes (objects, setting, action), speech transcript (topic, tone), on-screen text (captions, logos), and metadata. Each channel produces category evidence; a fusion step maps the combination onto the taxonomy. Classifying a whole video from its title alone is the most common failure mode — the same architecture that powers multimodal RAG and content moderation applies here, with categorical output instead of a ranked list. For video specifically, classify at the scene level with timestamps, not once for the whole file, so a 40-minute video can carry different categories in different segments.
How do you design a taxonomy that classifies well?
Classification accuracy is capped by taxonomy quality, so this is upstream work, not an afterthought. Four rules that hold across methods: categories should be mutually distinguishable by the signals you actually have (do not split "casual" vs "smart-casual" if your images can't support it); they should be balanced enough that rare classes still have signal (merge or hierarchy-nest the long tail); the taxonomy should be versioned, because it will change and every stored label is tied to a version; and where possible, seed it from the data rather than a whiteboard — discovered taxonomies surface the categories your corpus actually contains, which classify far better than aspirational ones nobody's content matches.
How does this work on Mixpeek?
Mixpeek treats taxonomy classification as enrichment over an already-indexed, decomposed corpus, and it runs in two modes (taxonomy docs). At ingest, a taxonomy is applied as content lands, so every document carries its category labels as filterable fields from day one. On demand, a
taxonomy_enrich stage inside a retriever classifies at query time — useful for testing a taxonomy, or classifying against reference data that changes faster than you want to reprocess. Because classification runs over stored features rather than raw files, changing a threshold, adding a category, or swapping the taxonomy re-scores the catalog without re-running extraction — the query-time reclassification economics that avoid re-paying analysis every time policy shifts. The taxonomy itself can be hand-authored or promoted from discovered clusters, so the categories are grounded in your data. Rates for the underlying indexing are on pricing; the classify-at-query-time pattern and the stage config are in the taxonomy docs.