The Short Answer
Materialize the expensive semantics, compute the cheap ones at query time. For unstructured data that split is forced rather than chosen: running a shot detector, a transcriber or a face model over an hour of video takes minutes of GPU time, and an interactive query budget is a few hundred milliseconds. The gap is three orders of magnitude or more, so extraction happens on write and the query plans over what is already there.
That decision has a price, and the price is staleness plus the cost of re-extraction whenever a model changes. This piece works through both, and through what a genuinely live path would require.
The budget arithmetic
Start with the numbers that decide it.
| Work | Order of magnitude | Can it run inside a query? |
| Metadata filter over indexed fields | Sub-millisecond to low milliseconds | Yes |
| Embedding a short text query | Single-digit to low tens of milliseconds | Yes |
| ANN search over a materialized index | Milliseconds to tens of milliseconds | Yes |
| Cross-encoder rerank of 50 candidates | Tens to low hundreds of milliseconds | Yes, with a budget |
| LLM listwise rerank of 20 candidates | Hundreds of milliseconds to seconds | Sometimes, for agents |
| Embedding one image | Tens of milliseconds | Yes, for the query image |
| Transcribing an hour of audio | Tens of seconds to minutes | No |
| Scene detection plus per-scene embedding over an hour of video | Minutes | No |
| Face detection and identity over an hour of video | Minutes | No |
Why this is a different question than it is in a warehouse
In a warehouse, materialization is an optimization. A view can always be recomputed on demand, so a materialized view trades storage and staleness for latency, and the fallback of running the query live is available whenever the view is missing.
Here there is no fallback. A feature nobody extracted cannot be computed inside the query at all, and a system that tries will either blow the latency budget or burn GPU the caller never asked for. So the planner has to fail loudly when a plan requires a feature that was never materialized, rather than degrading into an inline extraction. That constraint is one of the things that makes a retrieval plan a different object than a SQL statement.
That single behavior is worth checking in any system you evaluate. A layer that quietly extracts on read is one traffic spike away from a very expensive afternoon.
Staleness is what you pay
Materializing means a window where the object exists and the semantics do not. The size of that window is a real property of the system and belongs in the semantic contract, because callers build UI around it.
Three states worth distinguishing, since collapsing them is what produces confusing product behavior:
Not yet ingested. The object is not known to the layer. Nothing will find it.
Ingested, extraction pending. The object is addressable and its derived features do not exist. A search returns nothing for it, and a well-built layer can say why.
Extracted, index not yet refreshed. The features exist and the ANN index has not absorbed them. This is the shortest window and the most confusing one, because the document is readable while the query that should find it comes back empty. Index freshness and incremental updates covers how systems close it.
A UI that shows "indexed" the moment an upload completes is describing the first transition and implying the third. That is where support tickets come from.
Re-extraction is the dominant cost of change
The other half of the materialization bill arrives when a model changes.
Swapping an embedding model in a materialized architecture means re-embedding the corpus. If the swap is a different extractor rather than a different encoder, it means re-running inference over the source objects, which is the full extraction cost again. On a large archive that is a budgeted project with a schedule, not a config change.
Three things reduce the pain, and none of them eliminate it.
Keep the raw objects. Re-derivation is only possible when the source bytes are still yours, which is the argument for the object storage layer sitting under everything else. An architecture that discarded the originals after extraction has no path back.
Version the features rather than overwriting them, so old and new can serve side by side while you compare. That makes an encoder swap a rollout instead of a cutover.
Check whether you need a full re-embed at all. Some model changes can be handled by learning a mapping between spaces instead of re-running the encoder, which is worth evaluating before committing GPU months. Switching embedding models without re-embedding everything covers when that works and when it does not.
The hybrid that actually works
Almost every production system lands on the same split, and it is worth stating explicitly so you can check yours against it.
Materialized at write time. Everything that requires running a model over corpus content: segments, embeddings, transcripts, detected objects and faces, extracted document structure, classifications against a taxonomy.
Computed at query time. Query embedding, filters over already-indexed fields, fusion of ranked lists, reranking of a small candidate set, projection, and any per-request policy such as tenant scoping.
Computed at query time, on a small candidate set only. This is the interesting middle. A model too expensive to run corpus-wide is affordable over 40 results. Cross-encoders live here. So does an LLM that reads the top few segments and answers. The trick is that the expensive model never sees the corpus, only what the cheap stages already narrowed.
That third category is how a system gets query-time semantics without query-time extraction, and it is usually the answer when someone asks for "live" analysis.
What live federation would actually require
The stronger version of the question is federation: leave the data in place across several systems, push a semantic predicate into each one, and merge the results. Structured federation engines do a version of this over SQL sources.
Doing it over unstructured sources needs three things that do not currently exist together.
A pushdown target that understands the predicate. "Scenes containing this product" has to be executable by the remote system, and a video store that holds files and metadata cannot execute it. Without pushdown, federation degrades into transferring candidate objects to wherever the model runs, which is the expensive part with extra network in front of it.
A cost model across heterogeneous sources. The planner has to decide which predicate to push where, and it needs latency and selectivity estimates per source to do that. Those estimates are hard enough for one local index.
A way to reconcile results scored by different models. Two sources returning similarity scores from different embedding spaces cannot be merged by score, and merging by rank throws away the magnitude that made the scores useful.
None of this is impossible, and it is a genuinely open design space rather than a solved one. Worth being suspicious of any product claiming live semantic federation across arbitrary sources today, and worth asking which of those three problems they solved.
Where Mixpeek stands
Features are materialized at write time. Extraction runs when objects land, results are stored in collections, and every query plans over materialized state. Nothing federates live across arbitrary third-party sources.
The read-time half is the hybrid above: query embedding, filtering, fusion and reranking all happen per request, and the reranking stage can run a model that would be unaffordable corpus-wide. Raw objects stay in your own storage, which is what keeps re-derivation possible when a model changes. MVS keeps the vectors on that same object storage.
The architecture this sits inside is described in full in the semantic layer for unstructured and multimodal data, and what it looks like in production names the same materialization boundary in a shipped deployment.
Frequently Asked Questions
Should semantic features be materialized or computed at query time?
Materialized, for anything that requires running a model over corpus content. Transcription, scene detection, object and face extraction, and corpus-side embedding all cost seconds to minutes per hour of media, against an interactive query budget of a few hundred milliseconds. Query-side work stays live: embedding the query itself, filtering on indexed fields, fusing ranked lists, and reranking a small candidate set with a model that would be unaffordable across the whole corpus.
What is the cost of materializing semantic features?
Two costs. Staleness, meaning a window between an object arriving and its features becoming searchable, which callers need declared so they can build honest UI around it. And re-extraction, meaning that changing an extractor or an embedding model requires re-running inference over the corpus. Keeping the raw objects is what makes re-derivation possible at all, and versioning features rather than overwriting them turns a model swap into a rollout instead of a cutover.
Can a semantic layer query live sources without materializing?
Not usefully today, for unstructured content. Live federation needs a remote system that can execute a semantic predicate, a cost model that spans heterogeneous sources, and a way to merge results scored in different embedding spaces. Without pushdown, federation reduces to shipping candidate objects to wherever the model runs, which is the expensive part with network added. Metadata predicates can federate; "scenes containing this product" cannot.
Why can't extraction run inside a query?
Because the latency gap is roughly three orders of magnitude. Scene detection with per-scene embedding over an hour of video takes minutes of GPU time, and interactive queries budget a few hundred milliseconds. A system that falls back to inline extraction when a feature is missing will either blow every latency target or spend GPU the caller never authorized, so the correct behavior is to fail loudly when a plan requires a feature nobody materialized.
How do you close the gap between upload and searchable?
Distinguish the three states and expose them. An object can be unknown to the layer, known with extraction still pending, or extracted with the index not yet refreshed. The third window is the shortest and the most confusing, because the document is readable while the search that should find it returns nothing. Showing "indexed" at upload time describes the first transition while implying the last one, which is where most of the confusion comes from.