Data Infrastructure

11 min read

Updated 2026-08-29

# How Do You Model Unstructured Data From Raw Objects to Serving Contracts?

Five layers, each rebuildable from the one below: raw objects, units, features, entities, contracts. What each owns, why stable unit ids are the whole trick, a blast-radius table for every kind of change, how to keep a re-run from re-paying full extraction, and the layer almost everyone skips.

![Ethan Steininger](/people/ethan.webp)

By [Ethan Steininger](/blog/authors/ethan), Founder & CEO, Mixpeek

Semantic Layer

Data Modeling

Architecture

Feature Extraction

Taxonomies

Entity Resolution

Data Infrastructure

[Run this on your own data. Try it as you read.Search your own archive →](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)

[One-click templatesStart from a templateEach template deploys a working search app on your own files.Browse templates](/templates)

## The Short Answer

  
  
Model unstructured data in layers, the same way a warehouse separates raw tables from staging models from serving marts. Five layers, each with one owner and one rule about what may reference it:  
  
1\. **Raw objects.** The bytes, immutable, addressable, in storage you control. 2. **Units.** Decomposition into addressable pieces: segments, pages, regions, utterances. 3. **Features.** Typed, model-produced attributes of a unit, each in its own space. 4. **Entities.** Resolved, named things that features point at, governed by a vocabulary. 5. **Contracts.** Named retrieval capabilities the application binds to.  
  
The value of the split is the same as in a warehouse. Each layer can be rebuilt from the one below it, so a mistake costs one rebuild rather than a re-collection.  
  

## Where the warehouse analogy holds and where it stops

  
  
Raw, staging and marts works because staging is deterministic. Rebuild it from raw and you get the same rows, so raw is the only thing that has to be preserved.  
  
Two of those properties survive the move to unstructured data and one does not.  
  
Re-derivability survives, and it is the reason to keep raw objects rather than discarding them after extraction. Layer separation survives, and it is why entity resolution belongs in its own layer instead of inside an extractor.  
  
Determinism does not survive. Re-running extraction with a newer model version produces different features from the same bytes, so a rebuild is a change rather than a no-op. That is why the layer below the change has to keep its identifiers stable, and why features carry their producing version.  
  

## The five layers

  
  

Layer

Owns

Rebuilt from

What breaks without it

Raw objects

Bytes and their identity. Never mutated, only versioned

Nothing. This is the floor

Derived data with no source. Every model upgrade becomes a re-collection from the customer

Units

Decomposition into addressable pieces, with stable ids and offsets back into the source

Raw

Results that point at a whole two-hour file, which is not an answer

Features

Model-produced attributes of a unit: embeddings, transcripts, detections, classifications

Units plus a named model version

One vector per object, and no way to filter on the audio without dragging the video along

Entities

Resolution and naming. "These 400 face crops are the same person, and that person is P-1183"

Features plus a resolution policy

Every consumer invents its own labels, and two teams count the same entity differently

[Contracts](/guides/semantic-contract-for-unstructured-data)

Named retrieval capabilities with a declared unit, score meaning and filterable surface

Everything above

Applications bind to indexes, and changing an index becomes a migration for every caller

  
  
The layer boundaries are where versioning goes. A feature carries the version of the model that made it. A contract carries its own version, which changes on a different schedule from the models underneath it. Getting those two confused is how an encoder upgrade becomes a caller-visible break.  
  

## The id discipline is the whole trick

  
  
Everything above depends on one unglamorous property: a unit id that survives re-extraction.  
  
If segment ids are assigned by position in the output of the segmenter, then re-running with a better scene detector renumbers everything. Every feature that referenced segment 47 now points at different content. Every human annotation, every cached result and every stored user interaction is silently wrong, with no error anywhere.  
  
The alternative is deriving the id from content and position in the source rather than from processing order. A segment identified by `(object_id, start_time, end_time)` or by a hash over its byte range keeps its identity when the segmenter changes its mind about neighboring boundaries. Segments that genuinely changed get new ids and the old ones get tombstoned, which is a visible, diffable event.  
  
Same principle one layer up. An entity id should not be the index of a cluster, because clustering is not stable across runs. It should be a minted identifier that the resolution step maps into, so re-clustering moves members between entities rather than renaming all of them.  
  
This is the cheapest thing on the list to get right at the start and among the most expensive to retrofit.  
  

## What a change touches, layer by layer

  
  
Reading this table before a migration saves a lot of guessing about blast radius.  
  

Change

Raw

Units

Features

Entities

Contracts

New embedding model, same unit

untouched

untouched

rebuild

re-resolve if identity used it

version bump if score semantics moved

New segmentation strategy

untouched

rebuild, new ids for changed spans

rebuild for changed units

re-resolve affected

usually unchanged

New extractor added

untouched

untouched

additive

may gain a new entity type

additive

Taxonomy revision

untouched

untouched

untouched

re-resolve

unchanged if names held

Storage engine swap

untouched

untouched

reindex

untouched

unchanged

  
  
Two things stand out. A storage migration is invisible to everything above it, which is exactly what the layering is for. And a segmentation change is the most expensive one available, because it invalidates ids that everything above depends on. Decide segmentation carefully and early. [Multimodal chunking strategies](/guides/multimodal-chunking-strategies) covers how to pick it.  
  

## Idempotency, and the re-run that costs money

  
  
This is the write-side half of the [materialization decision](/guides/live-vs-materialized-semantic-features), and it is where its bill actually lands. Every pipeline gets re-run. Someone backfills, someone repairs a partial failure, someone fixes a bug and reprocesses a window.  
  
The question that decides your bill is what a re-run does with work that is already done. A pipeline that re-extracts every object it is handed will re-pay full GPU cost on a repair that only needed to touch 2% of the corpus, and the cost lands after the job has already run.  
  
So the dedup behavior of a re-run belongs in the design rather than in a runbook. Make it explicit which of these a re-run does: skip units that already carry a feature from this model version, recompute regardless, or recompute only where the version differs. All three are legitimate and they cost wildly different amounts. The failure is having the behavior be a default nobody read.  
  
[Production ingestion reliability](/guides/production-ingestion-reliability-agent-perception) covers ledgers and backfills, and [what it costs to make a media library searchable](/guides/cost-to-make-a-media-library-searchable) has the arithmetic on the re-extraction multiplier.  
  

## Testing a layer at a time

  
  
Each layer gets a different kind of test, and running the right one at the right level is what keeps a failure diagnosable.  
  
Raw: every object referenced by a unit still resolves. This catches lifecycle-policy deletions before they turn into unexplainable gaps.  
  
Units: counts and coverage. An hour of video should produce a plausible number of segments with offsets covering the duration and no overlaps. Silent under-segmentation is common and looks like nothing.  
  
Features: population, not just presence. Assert that units have the feature they should have, per model version, because partial extraction leaves the aggregate looking healthy. A check that only asks "did anything get written" passes on a run that processed one file out of a thousand.  
  
Entities: precision and recall on a labeled sample. Resolution has an error rate, and the only way to know it is to measure it.  
  
Contracts: for every field the read path exposes, assert the filter path reaches the same documents at the same path. This one catches the silent class, where a field is readable at `metadata.brand` and indexed somewhere else, so a filter on it returns empty with a 200.  
  

## The layer teams skip

  
  
Entities. Almost everyone skips entities, because the first use case works without them.  
  
The shape is recognizable. A face identity extractor gives you "these two faces are the same person," which is entity resolution, and it is enough while people are the only entity type in your queries. Then products arrive. Then brands. Each one gets its own extractor with its own notion of sameness, each collection encodes its own labels, and there is no shared vocabulary to join them.  
  
Retrofitting a governed vocabulary after several collections already encode their own labels is the expensive version, and the bill arrives on the day someone asks a question spanning two entity types. [Building a taxonomy from unlabeled data](/guides/build-a-taxonomy-from-unlabeled-data) is the path in when you get there, and [taxonomies](/taxonomies) is the primitive that holds it.  
  

## Mapping onto primitives

  
  
For anyone implementing this on Mixpeek: raw objects are buckets over your own storage, units and features come from [extractors](/extractors) writing into collections, entities are taxonomies and promoted clusters, and contracts are [retrievers](/retrievers). The full architecture and the reasoning behind the split is in [the semantic layer for unstructured and multimodal data](/semantic-layer).  
  

## Frequently Asked Questions

  
  

### How do you model unstructured data from raw files to searchable features?

  
  
In five layers, each rebuildable from the one below it: raw objects held immutably in storage you control, units produced by decomposing objects into addressable pieces with stable ids, features produced by models over those units, entities resolved and named through a governed vocabulary, and contracts that expose named retrieval capabilities to applications. Versioning attaches at two of those boundaries, once on features to record the producing model and once on contracts to record the caller-visible interface.  
  

### Is this the same as raw, staging and marts in a data warehouse?

  
  
The layering idea carries over and one property does not. Re-derivability holds, which is why raw objects have to be kept rather than discarded after extraction. Layer separation holds, which is why entity resolution belongs outside the extractor that produced the features. Determinism does not hold: re-running extraction with a newer model produces different features from identical bytes, so a rebuild is a change rather than a no-op, and identifiers underneath it have to stay stable for that change to be reviewable.  
  

### Why do segment ids need to be stable across re-extraction?

  
  
Because everything above them references them. Ids assigned by position in the segmenter's output get renumbered whenever the segmenter changes, so every feature, annotation, cached result and stored interaction that referenced segment 47 now points at different content, with no error raised anywhere. Deriving ids from source position or content, so a segment is identified by its span in the original object, keeps identity across re-runs and turns genuinely changed segments into new ids with tombstones on the old ones.  
  

### What does changing the embedding model touch?

  
  
Features get rebuilt, entity resolution re-runs if identity depended on that space, and the contract needs a version bump only if the score semantics moved. Raw objects and unit ids are untouched. That containment is the point of the layering: a storage engine swap is invisible above the feature layer, while a change to segmentation is the most expensive one available because it invalidates the ids every layer above depends on.  
  

### How do you keep a pipeline re-run from re-paying full extraction cost?

  
  
Decide the dedup behavior explicitly instead of inheriting a default. A re-run can skip units that already carry a feature from the current model version, recompute unconditionally, or recompute only where the recorded version differs. All three are legitimate for different jobs and they cost very differently, and the expensive mistake is a repair that needed 2% of the corpus running full extraction over all of it, with the cost visible only after the job completes.

## Run this on your own data

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this guide describes. Build starts at $25/mo for up to 1M vectors.

[Search your own archive](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)[Read Docs](/docs)

[Explore MVS](/mvs)[See pricing](/pricing)

## Related guides

[

Data Infrastructure

### What Does It Cost to Make a Video Library Searchable?

Making a video library searchable costs roughly $3 to $21 per hour of video to process once, depending on how many things you extract (a transcript, visual search, faces, on-screen text), plus a small storage bill for the index and the full processing cost again whenever you change models. This guide shows where each number comes from, a worked example for 1,000 hours, and how to cut each cost.

Read guide →](/guides/cost-to-make-a-media-library-searchable)[

Data Infrastructure

### Change Data Capture for a Search Index: Keeping Downstream Systems in Sync

How to keep a warehouse, cache or downstream index in sync with a search index that is constantly being written to, compared across polling by timestamp, webhooks and an ordered change feed, including the two ways cursor-based consumers silently lose data.

Read guide →](/guides/change-data-capture-for-a-search-index)[

Data Infrastructure

### How Do You Run Your Own Model Inside a Managed Search Pipeline?

The four places a managed retrieval platform can let you run your own code (ingest-time extraction, query-time inference, reranking, and enrichment), what each one demands of the platform, and why the ingest and query sides must load the identical model or your vectors and your queries end up in different spaces. Covers the packaging contract, the cold-start and GPU-allocation problems that decide whether query-time custom inference is usable, versioning against an already-indexed corpus, and how to tell a real extension point from a webhook with good marketing.

Read guide →](/guides/bring-your-own-model-custom-extractor-search-pipeline)