Taxonomy-Aware Retrieval

The hardest part of building a search system usually isn't the search. It's the taxonomy.

You need categories for filtering, faceting, and navigation. Define them by hand and you're guessing what's in your data before you've really looked at it. Guess wrong and you're rebuilding from scratch. Add a new data type and your categories are stale on arrival.

There's a better way: let the taxonomy grow out of the data.

It goes in four moves. You embed the documents into a shared vector space. Unsupervised clustering finds the natural groups in that space. An LLM reads each cluster and names it, something like "Product Reviews," "Technical Docs," or "Legal Contracts." The clusters that hold up become nodes in a reusable classification hierarchy.

Once it exists, the taxonomy earns its keep three ways. New documents get classified the moment they land, no manual tagging. The nodes turn into search filters and facets at query time. And when the taxonomy changes, existing documents reclassify themselves.

The point is that the best taxonomies come out of the data, not off a whiteboard. You don't have to know your categories up front, because the data tells you. The ones you never saw coming get discovered. The ones that don't hold up get merged. You end up with a taxonomy that fits your actual corpus instead of your assumptions about it.