What is Mixpeek?

[Back to Blog](/blog)

[](https://twitter.com/intent/tweet?text=The%203072-Dimension%20Problem&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fthe-3072-dimension-problem)[](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fthe-3072-dimension-problem&title=The%203072-Dimension%20Problem)[](https://www.reddit.com/submit?url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fthe-3072-dimension-problem&title=The%203072-Dimension%20Problem)

This was originally in our newsletter.

Get Updates

Apr 29, 2026•6 min read

# The 3072-Dimension Problem

A 3072-dimensional embedding encodes everything about a video and distinguishes nothing. Decomposing content into named, measurable features, then placing them in a queryable hierarchy, is how multimodal search actually works at scale.

![The 3072-Dimension Problem](https://blog.mixpeek.com/content/images/2026/04/ChatGPT-Image-Apr-29--2026--10_12_20-AM.png)

Architecture

TABLE OF CONTENTS

You embedded your video library with Gemini. Or [CLIP](https://mixpeek.com/model/openai/clip-vit-large-patch14?ref=blog.mixpeek.com), or [SigLIP](https://mixpeek.com/model/google/siglip2-giant-opt-patch16-384?ref=blog.mixpeek.com), or whatever the model of the month was when the project started. You stored a few million vectors in Pinecone or Qdrant or Weaviate. You wired up [cosine similarity](https://mixpeek.com/glossary?ref=blog.mixpeek.com). You ran your first query.

The results were fine. Not great. Fine.

A search for "person holding a coffee cup" returned videos with people, and videos with cups, and a surprising number of videos with neither. Reranking helped a little. [Hybrid BM25](https://mixpeek.com/hybrid-search?ref=blog.mixpeek.com) helped a little more. But somewhere around the fourth or fifth round of tuning, you started to suspect that the problem wasn't your retriever, or your reranker, or your prompt. The problem was that a 3072-dimensional embedding is not a useful unit of work.

It encodes everything. The lighting, the camera angle, the dominant color, the demographic of the person on screen, the room they're in, the _vibe_ of the room they're in. All of it, smeared across three thousand floats. Cosine similarity treats every dimension equally. Your application does not.

Most teams hit that wall about six weeks into a [multimodal project](https://mixpeek.com/multimodal-search?ref=blog.mixpeek.com). A better vector database won't get you past it.

* * *

## The reframe

A data science lead we work with (thirty years in the industry, ran one of the first audio fingerprinting startups in the early 2000s) described our job to me recently in a way I haven't stopped thinking about:

> Reduce the number of dimensions in a video to a handful of measurable features, then place every new video into a hierarchical structure defined by those features.

He called the output a "compact fingerprint."

When I shared that framing with an engineer at AWS, he pushed back immediately. Every kind of [representation learning](https://mixpeek.com/glossary/representation-learning?ref=blog.mixpeek.com) does that, he said. Find small, interpretable representations of complex things. He was right. The idea is old. What's rare is running it as managed infrastructure, continuously, across millions of objects, with the reduced features exposed as first-class queryable surfaces.

The job of multimodal infrastructure is not to be a faster [vector database](https://mixpeek.com/vector-search?ref=blog.mixpeek.com). It's the systems layer that takes you from _I have embeddings_ to _I have a usable application._ That layer has a specific shape, and most of the work is in the shape.

![Representation learning applied to a video library. A 3072-dimensional embedding encodes everything about a video and distinguishes nothing: lighting, camera angle, dominant color, who is on screen, the room and its vibe are all smeared across the same floats at equal weight, with no name you can filter on, which is why a search for a person holding a coffee cup returns videos with people, videos with cups, and a surprising number with neither. Decomposition is where the dimension reduction actually lives: a shot detector turns a thirty-second video into eight scenes with timestamps, a face identity model turns a frame into bounding boxes and 512-dimension face embeddings, a multimodal extractor turns a scene into a Gemini embedding plus OCR output plus dominant colors, and a custom plugin does the same for a proprietary file type. Each extractor's output becomes its own collection with its own schema and embedding space, so the features are named and filterable. Reduction alone leaves a smarter vector database; the leverage is placement. A new ad is placed against a brand taxonomy at ingest, and because the taxonomy that locates it is the same retriever a user invokes to ask for ads in that brand, enrichment and search become one traversal in two directions. Six primitives compose in one direction: bucket, extractor, collection, retriever, taxonomy, cluster.](https://blog.mixpeek.com/content/images/2026/08/the-3072-dimension-problem.png)

Reduce to named, measurable features, then place them in a hierarchy. The compact fingerprint is computed once; after that, retrieval is mostly traversal.

* * *

## How decomposition works in practice

The shape, in our system, is six primitives that compose in one direction.

Bucket raw objects land Extractor decompose features Collection queryable surfaces Retriever compose pipelines Taxonomy hierarchical placement Cluster emergent structure feeds back reduction & addressing composition & structure

**Buckets** are where raw objects land. Videos, images, documents, audio. A bucket has a schema and an optional [sync connection](https://mixpeek.com/connectors?ref=blog.mixpeek.com) (S3, GCS, Drive) and its only job is to be the entry point. Nothing has happened to the content yet.

## Run this on your own content

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this post describes. Build starts at $25/mo for up to 1M vectors.

[Search your own files](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=blog_onramp_mid&utm_campaign=mvs_onramp)[Talk to us](/demo)

**Feature extractors** are the decomposition step, where the dimension reduction actually lives. A [shot detector](https://mixpeek.com/extractors?ref=blog.mixpeek.com) turns a thirty-second video into eight scenes with timestamps. A [face identity model](https://mixpeek.com/model/deepinsight/insightface-retinaface-r50?ref=blog.mixpeek.com) turns a frame into bounding boxes and 512-dim face embeddings. A [multimodal extractor](https://mixpeek.com/model/google/siglip2-giant-opt-patch16-384?ref=blog.mixpeek.com) turns a scene into a Gemini embedding _plus_ [OCR output](https://mixpeek.com/ocr?ref=blog.mixpeek.com) _plus_ dominant colors _plus_ whatever else you configured. [Custom plugins](https://mixpeek.com/marketplace?ref=blog.mixpeek.com) (Python code, optionally with model weights) let you do the same for proprietary file types or domain-specific features.

**Collections** are where the reduced features live. Each collection is the output of one extractor, which means each has its own schema, its own embedding space, its own indices. A single bucket fans out into many collections: scenes, faces, OCR text, objects, audio segments. Instead of one giant vector per video, you get many small structured records, each measurable along a dimension you chose on purpose. That's the move that makes everything else work.

**Retrievers** compose collections. A [retriever](https://mixpeek.com/retrievers?ref=blog.mixpeek.com) is a multi-stage pipeline: [feature search](https://mixpeek.com/semantic-search?ref=blog.mixpeek.com) on collection A, attribute filter on collection B, LLM filter on the merged set, reciprocal rank fusion at the end. Stages pass documents forward in a working set. If you've written a MongoDB aggregation pipeline, the mental model transfers directly. Retrievers are pipelines because no real application wants a single similarity score. It wants similarity _plus_ a metadata filter _plus_ a rerank _plus_ a join.

[**Taxonomies**](https://mixpeek.com/taxonomies?ref=blog.mixpeek.com) are the hierarchy. A taxonomy is a semantic join between two collections, where the join operation is itself a retriever pipeline. The canonical example: collection A is faces extracted from a casting database (with names attached), collection B is faces extracted from new ad creative. The taxonomy says _for every face in B, find the nearest face in A above some threshold, and enrich B with the name._ Run that at ingest time and every new ad arrives pre-labeled with its talent.

[**Clusters**](https://mixpeek.com/clusters?ref=blog.mixpeek.com) are the dual of taxonomies. Taxonomies impose structure top-down (you defined the casting database). Clusters discover structure bottom-up. Run [HDBSCAN on the embeddings](https://mixpeek.com/blog/multimodal-taxonomies?ref=blog.mixpeek.com) in a collection, send the centroids to an LLM, and you get labeled groups you didn't have to specify in advance. The output is, of course, another collection, which can feed another retriever, which can populate another taxonomy.

Six primitives. They compose in one direction: raw object, decomposed feature, queryable surface, composed pipeline, hierarchical placement, emergent structure. The operation is _reduce to measurable features and make them addressable._

* * *

## Why hierarchy matters

Reduction alone is not enough. If you stop after the extractor stage, you have a smarter vector database. Better features, sure, but still a system where every query starts from scratch.

The leverage is in the hierarchy. Once you have a taxonomy of brands, products, talent, scenes, moods (or whatever your domain calls for), every new piece of content gets _placed_ against it at ingest time. The compact fingerprint is computed once. Its location in the hierarchy is computed once. After that, retrieval is mostly traversal.

That collapses a distinction most teams treat as fundamental: enrichment versus search.

*   **Flat embedding world:** enrichment is a separate batch job. Run a labeling pipeline, write labels back to the database, hope the labels stay fresh
*   **Decomposed-and-placed world:** enrichment _is_ the same operation as search, run in reverse

The taxonomy that locates a new ad in your brand hierarchy is the same retriever pipeline a user would invoke to ask "show me ads in this brand." One traversal, two directions.

Reducing dimensions is table stakes. Making the reduction a permanent, queryable, hierarchical structure is the part that compounds.

* * *

## What this unlocks

Three things fall out of this architecture that are hard to build any other way.

### Agentic retrieval

When your features are decomposed into named collections with named extractors, you can hand them to an LLM as tools. The LLM doesn't get a single search endpoint. It gets a feature search tool, an attribute filter tool, an LLM filter tool, each with explicit input and output shapes. The agent composes [retriever stages](https://mixpeek.com/agentic-rag?ref=blog.mixpeek.com) dynamically based on the task.

> Find ads featuring this actor that performed well in Q3 and use a similar color palette to this reference image

That becomes a four-stage pipeline the agent assembles on its own. You can't do that against a flat vector store because there's nothing to compose.

### Cross-collection joins

The casting-database example is the simple form. The general form: any two collections with comparable feature spaces can be joined via a taxonomy, so you can enrich any feature with any other feature.

*   Faces with names
*   Scenes with brands
*   Audio with transcripts
*   Products with categories

The join is a retriever, the retriever is reusable, and the enriched output is itself a collection that feeds the next join.

### Clustering that closes the loop

Run a clustering job on a collection, label the centroids with an LLM, and the labels become a taxonomy you didn't have to design. Apply that taxonomy to incoming content and every new object gets placed in a category that emerged from your data. The system bootstraps its own hierarchy.

The flywheel:

1.  More content produces better clusters
2.  Better clusters produce better taxonomies
3.  Better taxonomies produce better placement of the next batch

* * *

## The conceptual frame

The point of decomposition isn't to do anything magical. It takes the operation that [representation learning](https://mixpeek.com/glossary/representation-learning?ref=blog.mixpeek.com) has always done (reduce complex things to small interpretable representations) and makes it a piece of infrastructure instead of a piece of research.

*   [Extractors](https://mixpeek.com/extractors?ref=blog.mixpeek.com) instead of one-off model training
*   Collections instead of pickled embeddings
*   [Retrievers](https://mixpeek.com/retrievers?ref=blog.mixpeek.com) instead of bespoke search code
*   [Taxonomies](https://mixpeek.com/taxonomies?ref=blog.mixpeek.com) instead of manual labeling
*   [Clusters](https://mixpeek.com/clusters?ref=blog.mixpeek.com) instead of EDA notebooks

A 3072-dimensional [embedding](https://mixpeek.com/embeddings?ref=blog.mixpeek.com) is a starting point. The interesting work is in what you do after the embedding. If your stack stops there, you'll spend the next eighteen months reinventing the rest in application code. We know because we've watched it happen.

If any of that resonates, the [docs](https://docs.mixpeek.com/?ref=blog.mixpeek.com) walk through the primitives in the order they compose, with code.

[ES

Ethan Steininger

Former lead of MongoDB's Search Team, Ethan noticed the most common problem customers faced was building indexing and search infrastructure on their S3 buckets. Mixpeek was born.



](/blog/authors/ethan)

## Run this on your own content

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this post describes. Build starts at $25/mo for up to 1M vectors.

[Search your own files](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=blog_onramp_end&utm_campaign=mvs_onramp)[Talk to us](/demo)

## Continue Learning

[

![The Multimodal Data Warehouse: Why Unstructured Data Needs Its Own Snowflake](https://blog.mixpeek.com/content/images/2026/03/feature-image-6.png)

### The Multimodal Data Warehouse: Why Unstructured Data Needs Its Own Snowflake

We are drowning in unstructured data — video, audio, images, documents, IoT — but our infrastructure still assumes everything is a row or a vector. The multimodal data warehouse is the missing layer: object decomposition, tiered storage, and multi-stage retrieval pipelines for the AI era.

Read More





](/blog/multimodal-data-warehouse)[

![Why Vector Search Alone Can't Find What's in Your Videos](https://blog.mixpeek.com/content/images/2026/04/ChatGPT-Image-Apr-26--2026--09_58_30-AM-1.png)

### Why Vector Search Alone Can't Find What's in Your Videos

Text-only RAG pipelines miss 80%% of what is in your content. A video contains faces, dialogue, on-screen text, background music, and brand logos. No single embedding captures all of that. The solution is multi-stage retrieval.

Read More





](/blog/multimodal-retrieval-beyond-vector-search)

### Author

ES

Ethan Steininger

![Mixpeek Icon](/brand/logo-dark-icon.webp)

#### Multimodal Search, On Demand

Index, retrieve, and reason over videos, images, audio, and docs, all from a single API.

[Start Building](/start)

### Share

[](https://twitter.com/intent/tweet?text=The%203072-Dimension%20Problem&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fthe-3072-dimension-problem)[](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fthe-3072-dimension-problem&title=The%203072-Dimension%20Problem)[](https://www.reddit.com/submit?url=https%3A%2F%2Fmixpeek.com%2Fblog%2Fthe-3072-dimension-problem&title=The%203072-Dimension%20Problem)