The Short Answer
Building a taxonomy from unlabeled data runs in four steps: embed every item so similarity becomes measurable, cluster the embeddings with an algorithm that does not require you to know the number of categories in advance, label each cluster from the items closest to its center, then decide which clusters are stable and distinct enough to become real categories. The fourth step is the one that gets skipped, and skipping it is why most auto-generated taxonomies are unusable.
Clustering tells you what groups exist in the data. A taxonomy is a set of categories somebody has committed to, with names people agree on and a rule for what belongs in each. Getting from the first to the second is a judgment call that the algorithm cannot make for you, and the useful systems are the ones that make that call reviewable rather than automatic.
Why You Cannot Just Write the Taxonomy First
The obvious approach is to sit down, write out the categories you expect, and classify everything against them. That works when you already know the domain well. It fails on real archives for two reasons that show up immediately.
Your categories will not match the data. Someone writes twelve categories from memory, runs the classifier, and 40% of the corpus lands in "other" while three of the twelve match nothing at all. The content had structure; it just was not the structure anyone guessed.
And the interesting groups are the ones nobody thought to name. A media library turns out to be organized by shooting location rather than by product. An support archive splits by which integration the customer was using, which no one had considered a category. Those groups are only visible from the data.
Clustering first inverts the order. You find out what is actually in there, then decide what to call it. For why this is a distinct class of question that ordinary search cannot answer at all, see what concepts exist in my data that nobody has labeled yet. This guide is the procedure.
Step 1: Embed The Content
Clustering operates on vectors, so every item needs an embedding that puts similar things near each other. The choice of embedding model decides what "similar" means, and it decides more about your final taxonomy than the clustering algorithm does.
A text embedding groups documents by topic. A CLIP-style image-text model groups pictures by what a caption would say about them. A self-supervised visual model like DINOv3 groups by visual appearance, which can mean grouping two unrelated products because both were shot on a white background. None of those is wrong. They answer different questions, and you should pick the one matching the taxonomy you want.
For mixed corpora the honest approach is to cluster each modality separately before trying to combine them. Concatenating a text vector and an image vector produces a space where distance means nothing in particular, because the two halves have different scales and different intrinsic dimensionality. Composite clustering covers the ways to combine them that do work.
Step 2: Cluster Without Choosing K
k-means demands that you specify the number of clusters up front. On unlabeled data you do not know that number, and guessing it is the whole problem you were trying to solve. k-means also forces every point into a cluster, so noise and one-off items get absorbed into whatever group is nearest, quietly polluting it.
Density-based algorithms avoid both. HDBSCAN builds a hierarchy of density levels and extracts the clusters that persist across many of them, which means the count comes out of the data. It also has an explicit noise label, so an item that belongs to nothing stays in nothing.
That noise label is worth more than it sounds. A taxonomy built with k-means has no way to say "this file is genuinely uncategorized", so every archive appears fully covered. HDBSCAN typically leaves 10% to 40% of a real corpus unclustered, and that number is information: a high noise fraction usually means the embedding is wrong for the question, not that the data lacks structure.
Two parameters do most of the work.
min_cluster_size sets the smallest group you are willing to call a category, and it should come from a product decision rather than a default. min_samples controls how conservative the density estimate is, with higher values producing more noise and tighter cores.Step 3: Label What Came Out
A cluster is a set of item IDs. Turning it into a category means giving it a name a human would recognize.
The cheapest method that works is exemplar-based. Take the items nearest the cluster centroid, since those are the most typical members, and derive the name from them. For text, the highest-TF-IDF terms across those exemplars against the rest of the corpus gives a usable label. For images or video, passing a handful of exemplar frames to a vision-language model and asking what they have in common produces something readable.
Centroid text alone is a trap on multimodal clusters. The geometric center of a group of image embeddings often decodes to something generic, because the center of "outdoor product shots" is not itself a recognizable outdoor product shot. Sampling several real members beats describing an average that no member resembles.
Label quality is also where a human review step pays for itself. Naming is fast to check and expensive to get wrong, since the name is what everyone downstream will reason about.
Step 4: Decide Which Clusters Deserve To Be Categories
Here is the step that separates a taxonomy from a clustering run. Not every cluster should become a category, and promoting all of them produces a flat list of two hundred groups that nobody can navigate.
Four properties are worth checking before promotion:
Size. A cluster with six items in a corpus of a million is a curiosity. It might still matter if those six are high-value, but the default should be to leave it as noise.
Separation. If two clusters sit close together and their exemplars are hard to tell apart, they are one category that the algorithm split. Merge them and give the merged group one name.
Stability. Re-run the clustering on a random 80% sample. Clusters that appear in both runs with substantially the same membership are real structure. Ones that appear only once are artifacts of the parameters.
Nameability. If a reviewer cannot write a one-line rule for what belongs in a cluster, downstream users will not be able to either. That cluster is not ready to be a category regardless of how clean it looks geometrically.
Stability testing is the one most often skipped and the one that catches the most damage. It costs a second clustering run.
Why Silhouette Scores Mislead Here
The standard cluster quality metrics were designed for low-dimensional data with roughly spherical, equally-sized clusters. Embeddings are none of those things.
Silhouette score compares each point's distance to its own cluster against its distance to the nearest other cluster. In 768 or 1024 dimensions, distances concentrate: everything is roughly equidistant from everything else, so the numerator and denominator both shrink and the score loses discriminating power. A silhouette of 0.15 on embedding data can describe a perfectly useful taxonomy, and chasing a higher number usually means merging clusters that should have stayed apart.
Stability under resampling is the more honest signal, and human review of the exemplars is more honest still. If you want a single number, cluster persistence from the HDBSCAN hierarchy is at least measuring something the algorithm actually optimized.
How The Approaches Compare
| Approach | Needs cluster count up front | Handles noise | Finds nested structure | Main limitation |
| k-means | Yes | No, forces every point into a cluster | No | You must already know the answer to the question you are asking |
| Agglomerative | No, but needs a cut threshold | No | Yes, produces a dendrogram | Quadratic memory, so it struggles past a few hundred thousand items |
| HDBSCAN | No | Yes, explicit noise label | Yes, via the condensed tree | Sensitive to min_cluster_size, and that choice is a product decision |
| Topic models (LDA) | Yes, number of topics | Partially | No | Text only, and bag-of-words ignores word order |
| Classify against a written taxonomy | Not applicable | Yes, via a confidence threshold | Only what you wrote | Cannot discover a category nobody thought of |
What Happens When New Content Arrives
A taxonomy built once is accurate on the day it ships and drifts from then on. New content arrives that belongs to no existing category, and existing categories slowly change meaning as their membership shifts.
Re-clustering everything on a schedule produces a different taxonomy each time, with renamed and re-split categories, which breaks anything built on the old names. The workable pattern is to freeze the promoted categories, classify new items against them, and watch two signals: the fraction of new items that fail to match any category with confidence, and the drift of each category's centroid over time. When the unmatched fraction crosses a threshold you set, that is the trigger to cluster the unmatched pool and consider promoting new categories, leaving existing ones intact.
When This Whole Approach Is Wrong
Clustering-first assumes the structure you want is the structure the embeddings expose. Sometimes it is not.
If your categories are defined by policy rather than by content, clustering will not find them. "Requires legal review" is not a visual or semantic property, and no embedding groups by it. Those categories have to be written down and classified against, or learned from labeled examples.
If the corpus is small, under a few thousand items, reading a sample and writing the taxonomy by hand is faster and better. Clustering earns its cost at scale.
And if you need the taxonomy to match an external standard, an industry schema or a regulatory list, discovery is the wrong tool entirely. You are doing classification against a fixed target, and the question is only how to handle content that fits nothing.
Doing This On Mixpeek
The steps above are the general method. On Mixpeek, clusters runs the embedding and clustering over content already indexed from object storage, so the input is whatever is in your buckets rather than an export. Cluster output can be enriched with labels and promoted into a governed taxonomy that later content is classified against. Agentic hierarchical cluster search covers using the cluster tree itself as a retrieval surface, which is useful before you have committed to any taxonomy at all.
If you are running your own vectors and only want the retrieval layer, MVS is the standalone vector store. Pricing covers both paths.
Frequently Asked Questions
How many clusters should I expect from a corpus?
There is no target number, and treating one as a goal is how taxonomies go wrong. What matters is that each promoted cluster passes the size, separation, stability and nameability checks above. A million-item archive might honestly contain twelve useful categories or two hundred, and the answer depends on the questions people will ask of it rather than on the corpus size.
Can I build a taxonomy from video or audio without transcripts?
Yes, using visual or audio embeddings directly, and the resulting categories will be grouped by appearance or sound rather than by subject matter. That is often what you want for footage libraries. If you need topical categories, you need text: transcripts from speech recognition, OCR, or descriptions from a vision-language model, and then you cluster that text.
Why does my clustering produce one giant cluster and a lot of noise?
Almost always the embedding rather than the algorithm. If most items are near each other in the vector space, the model is not separating on the property you care about. Try a model trained for the modality and the kind of similarity you want, and check a handful of nearest-neighbor pairs by hand before tuning any clustering parameter.
Should I use an LLM to generate the taxonomy directly?
An LLM is good at naming a cluster you hand it and unreliable at inventing a taxonomy that matches a corpus it has not seen. Asking for categories cold produces a plausible list that is not grounded in your data, which is the same failure as writing the taxonomy from memory. Use clustering to find the groups and the model to name them.
How is this different from classification?
Classification assigns items to categories that already exist. Discovery finds the categories. They run in sequence: discover once to build the taxonomy, then classify continuously as new content arrives. Confusing the two is why "just use a zero-shot classifier" produces a taxonomy that misses whatever the label list omitted.