Semantic Layer for Unstructured and Multimodal Data
Traditional semantic layers define stable business concepts over structured tables. Unstructured data carries an extra problem: its useful semantics do not exist until something extracts them. A semantic layer for video, audio, images and documents has to run that extraction, keep the output governed and durable, then serve it through a contract that survives changing the encoder or the index underneath.
What a semantic layer does
In a warehouse, revenue is a column somewhere and five teams hold five definitions of it. A semantic layer keeps the one definition everybody binds to. Cube, LookML and dbt metrics all do this job: a consumer asks for revenue by region, the layer resolves that request into tables, joins and filters, and the physical plan changes underneath without breaking the dashboard.
Four properties make that a layer rather than a naming convention.
A vocabulary consumers bind to
Every dashboard, notebook and service asks for the same named concept and gets the same answer. The name is the interface.
Governance attached to definitions
Permissions, lineage and version history live on the definition rather than on each copy of the SQL that implements it.
Execution independence
The warehouse, the physical schema and the query plan can all change while the definition holds. Consumers never see the migration.
One place to review meaning
When someone disputes a number, there is a single artifact to read and a single change to make.
What changes when the data is unstructured
revenue already exists in the table. "The scene where the product sits on a shelf" exists nowhere in an hour of video until a model produces it. The first job of the layer is manufacturing the semantics, and that changes four things about how the layer has to be built.
Extraction is a write-time cost
Running a shot detector across an archive costs real GPU time. The results have to be materialized and reused, so the layer owns durable derived state instead of compiling a fresh query on every request.
One object yields many units at different granularity
A single video is scenes, shots, frames, faces, spoken words and on-screen text. Each of those is a separate thing to name, address and permission.
Each unit wants its own index
A face embedding and a transcript embedding do not share a space. Collapsing an object into one vector destroys the ability to filter on either one.
The semantics are model-dependent
Swap the encoder and every vector moves. The contract consumers bind to has to be versioned apart from the model that produced it, or every model upgrade becomes a breaking change for every caller.
The architecture
Seven layers, each owning one job. The column on the right is what an architecture without that layer ends up paying for.
The bytes and their identity, in object storage you control.
Derived data with no source to re-derive from. Every model upgrade turns into re-collecting the corpus.
Turning bytes into named units: scenes, faces, transcripts, blocks, entities, embeddings.
The semantics never come into existence, and retrieval degrades to keyword search over filenames.
Durable storage for extracted units, each with its own schema and its own embedding space, all of it filterable.
One vector per object, and no way to ask for the audio without dragging the video along with it.
The vocabulary, and the relationships between entities that hold across modalities. Person, Product, Scene, Brand mean one thing.
Every consumer invents its own labels. Two teams count the same entity differently and nobody can reconcile them.
The stable multi-stage contract a consumer binds to: filter, search, join, rerank, composed once and named.
Applications bind to indexes. Changing an index becomes a migration for every caller that touched it.
Stage order, filter pushdown, candidate budgets, and the cost of a request.
The plan is hardcoded in the caller, and cost is whatever the caller happened to guess.
Consumption. They name a capability and receive results with provenance.
Nothing. This layer is the consumer.
A semantic layer beyond SQL
A structured semantic layer compiles a named concept into SQL. There is no SQL for "the frame where the logo covers more than twenty percent of the screen". The compilation target becomes a retrieval plan, and the planning problem keeps the same shape with different operators.
| Dimension | Structured semantic layer | Multimodal semantic layer |
|---|---|---|
| Unit of meaning | A row, or an aggregate over rows | A feature extracted from part of an object: a scene, a face, a paragraph, a spoken phrase |
| Where semantics come from | The schema, written by whoever modelled the table | A model at write time. The meaning is absent until extraction runs |
| Compilation target | SQL | A retrieval plan: metadata filters, vector search over one or more spaces, cross-collection joins, reranking |
| Join key | Foreign keys and shared column values | Semantic similarity, plus entities resolved through a taxonomy |
| Dominant cost | Query execution | Extraction at write time. Queries are cheap next to it |
| What versioning protects | The meaning of a column | The meaning of a feature, and the model that produced its vectors |
| Refresh | Recompute the view | Re-extract the affected objects, then reindex the features they produced |
A semantic layer for AI agents
Give an agent raw index access and it has to choose an embedding model, a candidate count, a filter and a rerank strategy on every call. It will choose differently each time, and none of those choices can be reviewed afterwards or permissioned in advance.
A retriever collapses all of that into one named capability with a fixed contract, so the agent selects among capabilities. Access control, rate limiting and audit logging attach to the capability. A raw index scan has no surface to attach them to.
The same property is what lets a platform team upgrade an encoder. The agent asked for scene_search and keeps asking for scene_search while the vectors underneath are replaced.
Live query or materialized features
Structured semantic layers push most work down to the warehouse at query time, because SQL over a table is fast enough to compile on demand. Multimodal semantics do not behave that way. A shot detector over an hour of video takes longer than any interactive budget allows, so the layer extracts at write time and plans over materialized features at read time.
A planner that decides per predicate whether to read materialized features or push a query live into a source system is a real design space and an open one. Mixpeek materializes today. We say that here because a semantic layer that overstates its federation story sends people into an architecture that will not hold once they depend on it.
How Mixpeek implements this architecture
Mixpeek is the semantic retrieval layer for unstructured and multimodal data. Each layer above maps to a primitive with an API.
| Layer | Primitive | What it does |
|---|---|---|
| Raw objects | Buckets | Objects stay on your own S3, GCS, Azure Blob, R2 or other S3-compatible storage. |
| Decomposition and extraction | Feature extractors | Shot detection, transcription, OCR, face and logo identity, document parsing, plus custom extractors running your own code and weights. |
| Semantic features | Collections | One collection per extractor, each with its own schema and embedding space, which is what keeps a feature filterable. |
| Taxonomies and ontologies | Taxonomies, ontologies, clusters | Taxonomies behave as semantic joins across modalities. Clusters find structure nobody defined, and a cluster can be promoted into a governed taxonomy. |
| Retrievers | Retrievers | Multi-stage pipelines composed once and addressed by name. This is the contract your application binds to. |
| Query planning | Stage configuration and budgets | Stage order, filter pushdown and candidate budgets are configuration on the retriever rather than logic in the caller. |
| Storage underneath | Mixpeek Vector Store | Vectors live on your object storage with dense, sparse and BM25 search. Bring your own vectors, or let managed indexing produce them. |
Frequently asked questions
What is a semantic layer for unstructured data?
A semantic layer for unstructured data is the layer between raw objects and the applications that query them. It extracts named semantic features from video, audio, images and documents, holds a governed vocabulary over those features, and exposes stable retrieval contracts. Applications bind to the contract, so the embedding model, the index and the storage engine underneath can all change without breaking any caller.
How is a multimodal semantic layer different from a structured semantic layer like Cube?
The architecture is the same and the substrate differs. Cube and similar tools define stable business concepts over tables where the semantics already exist in a schema, and compile requests to SQL. A multimodal semantic layer has to produce the semantics first, because nothing in an hour of video says which scene contains a product until a model extracts it. That makes extraction a write-time cost, makes materialized features the normal case, and makes the compilation target a retrieval plan rather than SQL.
Is a semantic layer the same as a vector database?
No. A vector database stores and searches vectors, which is the storage job. A semantic layer sits above it: it decides what gets extracted, names the features, holds the taxonomy that connects them, and exposes the retrieval contract. You can build a semantic layer on top of any vector database, and you still have to build it. Mixpeek ships its own vector store (MVS) so vectors stay on your object storage, and the layer above it is the part applications actually bind to.
What does a semantic layer look like beyond SQL?
The compilation target changes. A structured semantic layer resolves a named concept into SQL over tables. A semantic layer for unstructured data resolves a named capability into a retrieval plan: metadata filters over extracted features, vector search across one or more embedding spaces, joins across collections through entity taxonomies, and reranking. The planning problem has the same shape, with different operators and a different cost model, since extraction dominates rather than query execution.
Why do AI agents need a semantic layer?
An agent given raw index access has to choose an embedding model, a candidate count, a filter and a rerank strategy on every call, and it will choose differently each time. None of those choices are reviewable after the fact and none of them can be permissioned. A retriever collapses that into one named capability with a fixed contract, so the agent selects among capabilities. Access control, rate limiting and audit logging attach to the capability, which a raw index scan cannot support.
Does a semantic layer for unstructured data query live sources or materialized features?
Materialized features, in almost every practical system today, because extraction is too slow to run inside an interactive query. Running a shot detector across an hour of video takes longer than any query budget allows, so features are extracted at write time and the planner works over them at read time. Pushing predicates live into arbitrary third-party systems and merging the results is an open design space. Mixpeek materializes today and does not federate live across arbitrary sources.
Can I put a semantic layer over embeddings I already have?
Yes. Existing vectors can be written into collections directly, which skips the extraction step and keeps the rest of the layer: schemas, taxonomies, retrievers and query planning. This is the common path for teams who already ran extraction and found that the missing piece was the contract above the index rather than another index.
What breaks in an architecture with no semantic layer?
Applications bind to indexes. Changing the embedding model becomes a coordinated migration across every caller, each team invents its own labels for the same entities, retrieval logic is copied into services where nobody can review it, and cost is whatever each caller guessed when it hardcoded a candidate count. Each of those failures is cheap to fix early and expensive once several services depend on the current shape.
Keep reading
The storage and lifecycle half: decompose, tier, reassemble.
The retrieval contract layer, in product terms.
How a governed vocabulary behaves as a semantic join across modalities.
Cross-modal relationships between entities.
Vectors on your own object storage, underneath the layer.
Why raw objects are the wrong unit to query directly.