The Short Answer
Model unstructured data in layers, the same way a warehouse separates raw tables from staging models from serving marts. Five layers, each with one owner and one rule about what may reference it:
1. Raw objects. The bytes, immutable, addressable, in storage you control. 2. Units. Decomposition into addressable pieces: segments, pages, regions, utterances. 3. Features. Typed, model-produced attributes of a unit, each in its own space. 4. Entities. Resolved, named things that features point at, governed by a vocabulary. 5. Contracts. Named retrieval capabilities the application binds to.
The value of the split is the same as in a warehouse. Each layer can be rebuilt from the one below it, so a mistake costs one rebuild rather than a re-collection.
Where the warehouse analogy holds and where it stops
Raw, staging and marts works because staging is deterministic. Rebuild it from raw and you get the same rows, so raw is the only thing that has to be preserved.
Two of those properties survive the move to unstructured data and one does not.
Re-derivability survives, and it is the reason to keep raw objects rather than discarding them after extraction. Layer separation survives, and it is why entity resolution belongs in its own layer instead of inside an extractor.
Determinism does not survive. Re-running extraction with a newer model version produces different features from the same bytes, so a rebuild is a change rather than a no-op. That is why the layer below the change has to keep its identifiers stable, and why features carry their producing version.
The five layers
| Layer | Owns | Rebuilt from | What breaks without it |
| Raw objects | Bytes and their identity. Never mutated, only versioned | Nothing. This is the floor | Derived data with no source. Every model upgrade becomes a re-collection from the customer |
| Units | Decomposition into addressable pieces, with stable ids and offsets back into the source | Raw | Results that point at a whole two-hour file, which is not an answer |
| Features | Model-produced attributes of a unit: embeddings, transcripts, detections, classifications | Units plus a named model version | One vector per object, and no way to filter on the audio without dragging the video along |
| Entities | Resolution and naming. "These 400 face crops are the same person, and that person is P-1183" | Features plus a resolution policy | Every consumer invents its own labels, and two teams count the same entity differently |
| Contracts | Named retrieval capabilities with a declared unit, score meaning and filterable surface | Everything above | Applications bind to indexes, and changing an index becomes a migration for every caller |
The id discipline is the whole trick
Everything above depends on one unglamorous property: a unit id that survives re-extraction.
If segment ids are assigned by position in the output of the segmenter, then re-running with a better scene detector renumbers everything. Every feature that referenced segment 47 now points at different content. Every human annotation, every cached result and every stored user interaction is silently wrong, with no error anywhere.
The alternative is deriving the id from content and position in the source rather than from processing order. A segment identified by
(object_id, start_time, end_time) or by a hash over its byte range keeps its identity when the segmenter changes its mind about neighboring boundaries. Segments that genuinely changed get new ids and the old ones get tombstoned, which is a visible, diffable event.Same principle one layer up. An entity id should not be the index of a cluster, because clustering is not stable across runs. It should be a minted identifier that the resolution step maps into, so re-clustering moves members between entities rather than renaming all of them.
This is the cheapest thing on the list to get right at the start and among the most expensive to retrofit.
What a change touches, layer by layer
Reading this table before a migration saves a lot of guessing about blast radius.
| Change | Raw | Units | Features | Entities | Contracts |
| New embedding model, same unit | untouched | untouched | rebuild | re-resolve if identity used it | version bump if score semantics moved |
| New segmentation strategy | untouched | rebuild, new ids for changed spans | rebuild for changed units | re-resolve affected | usually unchanged |
| New extractor added | untouched | untouched | additive | may gain a new entity type | additive |
| Taxonomy revision | untouched | untouched | untouched | re-resolve | unchanged if names held |
| Storage engine swap | untouched | untouched | reindex | untouched | unchanged |
Idempotency, and the re-run that costs money
This is the write-side half of the materialization decision, and it is where its bill actually lands. Every pipeline gets re-run. Someone backfills, someone repairs a partial failure, someone fixes a bug and reprocesses a window.
The question that decides your bill is what a re-run does with work that is already done. A pipeline that re-extracts every object it is handed will re-pay full GPU cost on a repair that only needed to touch 2% of the corpus, and the cost lands after the job has already run.
So the dedup behavior of a re-run belongs in the design rather than in a runbook. Make it explicit which of these a re-run does: skip units that already carry a feature from this model version, recompute regardless, or recompute only where the version differs. All three are legitimate and they cost wildly different amounts. The failure is having the behavior be a default nobody read.
Production ingestion reliability covers ledgers and backfills, and what it costs to make a media library searchable has the arithmetic on the re-extraction multiplier.
Testing a layer at a time
Each layer gets a different kind of test, and running the right one at the right level is what keeps a failure diagnosable.
Raw: every object referenced by a unit still resolves. This catches lifecycle-policy deletions before they turn into unexplainable gaps.
Units: counts and coverage. An hour of video should produce a plausible number of segments with offsets covering the duration and no overlaps. Silent under-segmentation is common and looks like nothing.
Features: population, not just presence. Assert that units have the feature they should have, per model version, because partial extraction leaves the aggregate looking healthy. A check that only asks "did anything get written" passes on a run that processed one file out of a thousand.
Entities: precision and recall on a labeled sample. Resolution has an error rate, and the only way to know it is to measure it.
Contracts: for every field the read path exposes, assert the filter path reaches the same documents at the same path. This one catches the silent class, where a field is readable at
metadata.brand and indexed somewhere else, so a filter on it returns empty with a 200.The layer teams skip
Entities. Almost everyone skips entities, because the first use case works without them.
The shape is recognizable. A face identity extractor gives you "these two faces are the same person," which is entity resolution, and it is enough while people are the only entity type in your queries. Then products arrive. Then brands. Each one gets its own extractor with its own notion of sameness, each collection encodes its own labels, and there is no shared vocabulary to join them.
Retrofitting a governed vocabulary after several collections already encode their own labels is the expensive version, and the bill arrives on the day someone asks a question spanning two entity types. Building a taxonomy from unlabeled data is the path in when you get there, and taxonomies is the primitive that holds it.
Mapping onto primitives
For anyone implementing this on Mixpeek: raw objects are buckets over your own storage, units and features come from extractors writing into collections, entities are taxonomies and promoted clusters, and contracts are retrievers. The full architecture and the reasoning behind the split is in the semantic layer for unstructured and multimodal data.
Frequently Asked Questions
How do you model unstructured data from raw files to searchable features?
In five layers, each rebuildable from the one below it: raw objects held immutably in storage you control, units produced by decomposing objects into addressable pieces with stable ids, features produced by models over those units, entities resolved and named through a governed vocabulary, and contracts that expose named retrieval capabilities to applications. Versioning attaches at two of those boundaries, once on features to record the producing model and once on contracts to record the caller-visible interface.
Is this the same as raw, staging and marts in a data warehouse?
The layering idea carries over and one property does not. Re-derivability holds, which is why raw objects have to be kept rather than discarded after extraction. Layer separation holds, which is why entity resolution belongs outside the extractor that produced the features. Determinism does not hold: re-running extraction with a newer model produces different features from identical bytes, so a rebuild is a change rather than a no-op, and identifiers underneath it have to stay stable for that change to be reviewable.
Why do segment ids need to be stable across re-extraction?
Because everything above them references them. Ids assigned by position in the segmenter's output get renumbered whenever the segmenter changes, so every feature, annotation, cached result and stored interaction that referenced segment 47 now points at different content, with no error raised anywhere. Deriving ids from source position or content, so a segment is identified by its span in the original object, keeps identity across re-runs and turns genuinely changed segments into new ids with tombstones on the old ones.
What does changing the embedding model touch?
Features get rebuilt, entity resolution re-runs if identity depended on that space, and the contract needs a version bump only if the score semantics moved. Raw objects and unit ids are untouched. That containment is the point of the layering: a storage engine swap is invisible above the feature layer, while a change to segmentation is the most expensive one available because it invalidates the ids every layer above depends on.
How do you keep a pipeline re-run from re-paying full extraction cost?
Decide the dedup behavior explicitly instead of inheriting a default. A re-run can skip units that already carry a feature from the current model version, recompute unconditionally, or recompute only where the recorded version differs. All three are legitimate for different jobs and they cost very differently, and the expensive mistake is a repair that needed 2% of the corpus running full extraction over all of it, with the cost visible only after the job completes.