Architecture

11 min read

Updated 2026-08-29

# What a Semantic Layer Looks Like in Production

The semantic-layer architecture for unstructured data, mapped onto one shipped video-search deployment layer by layer: what each layer owns, what it prevented, and the three places this deployment does not prove the thesis. Uses Cutsio's RED R3D and ARRI RAW pipeline, with their published numbers.

![Ethan Steininger](/people/ethan.webp)

By [Ethan Steininger](/blog/authors/ethan), Founder & CEO, Mixpeek

Semantic Layer

Architecture

Retrieval Architecture

Video Search

Feature Extraction

Case Study

Object Storage

Entity Resolution

[Run this on your own data. Try it as you read.Search your own archive →](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)

[One-click templateFootage Intelligence at Archive ScaleMillions of scenes you can move through instead of search. Footage, ads and editor assets land in three clean collections, and scenes are clustered one partition at a time so every run stays under the sample-size caps.Deploy it](/templates/footage-intelligence)

## The Short Answer

  
  
A [semantic layer for unstructured data](/semantic-layer) is easy to describe and easy to dismiss as an abstraction someone drew on a whiteboard. So here is one that runs in production, layer by layer, with the parts that are missing named as missing.  
  
The deployment is Cutsio, a collaborative editing platform for post-production teams working with RED R3D and ARRI RAW footage. Editors there were spending about 35% of the workday looking for clips. They now spend about 5%, and a search that averaged 35 minutes returns in under 15 seconds. The full write-up with their numbers is the [Cutsio case study](/case-studies/video-editor-footage-search).  
  
What makes it worth reading as architecture rather than as a customer story is that every layer in the general model is doing an identifiable job, and you can see what each one is protecting against.  
  

## The deployment in one paragraph

  
  
Raw camera footage lands in Cutsio's Mux infrastructure. A selective sync decides which assets Mixpeek indexes. A custom extractor transcodes the two raw cinema formats and passes everything else through untouched. Three chained collections turn each video into timestamped segments with scene-level embeddings, then run face detection and identity embedding over those segments. Two named retrievers serve the editing UI: one for visual search by text or reference frame, one for finding every moment a given person appears. An editor uploading an R3D file has it searchable within minutes.  
  

## Layer 1: raw objects

  
  
Cutsio remains the source of truth for their media. Mux carries the sync hint. Mixpeek indexes and does not take custody.  
  
The mechanism is worth copying. Each asset carries a passthrough metadata flag, and `metadata_filters` on the sync only index assets whose flag matches. A `video.asset.updated` webhook re-evaluates the filter when the flag changes, and `reconcile_on_sync` unindexes assets that stop qualifying.  
  
That last clause is the part most integrations skip. An index that adds on eligibility and never removes on ineligibility drifts silently, and the drift shows up as a search result nobody can explain months later. Eligibility is a two-way door here.  
  
What breaks without this layer: derived data with no source to re-derive from. Every model upgrade turns into re-collecting the corpus from the customer.  
  

## Layer 2: decomposition and extraction

  
  
This is where the semantics come into existence, and it is the reason a semantic layer for video cannot be a thin wrapper over a warehouse.  
  
Nothing in an R3D file says which scene contains a product, or who is on screen at 04:12. A model has to produce that, and producing it costs real GPU time. Cutsio's pipeline runs a custom conversion extractor at v3.1.0 that passes standard formats through and transcodes only R3D and ARRI RAW, using REDline and the ARRI Reference Tool already present in the engine image. That removed a DaVinci Resolve install from every editor's machine.  
  
Two properties of this layer generalize:  
  
*   **Extraction is a write-time cost.** It is too slow to run inside a query. Everything downstream is planning over what was already extracted.
*   **Passthrough is a feature.** An extractor that transcodes everything would have burned GPU on footage that needed nothing done to it. The cheap path has to be the default path.
  
  
What breaks without this layer: the semantics never exist, and retrieval degrades to keyword search over filenames. Which is the thing editors were doing manually.  
  

## Layer 3: semantic features

  
  
One video yields many units at different granularity, and each unit wants its own home.  
  
The pipeline is three chained collections rather than one: format detection and transcoding, then multimodal decomposition into timestamped segments carrying scene-level embeddings for composition, subjects, motion, color palette and on-screen text, then face detection and identity embedding per segment.  
  
Chaining them is not an implementation detail. A face embedding and a scene embedding do not share a vector space, and collapsing a video into one vector destroys the ability to filter on either. Separate collections keep each feature addressable, filterable, and independently re-computable when its model changes.  
  
What breaks without this layer: one vector per object, and no way to ask for the face without dragging the whole scene along with it.  
  

## Layer 4: entity resolution, and where this deployment is honest about a gap

  
  
In the general model this layer holds taxonomies and ontologies: the governed vocabulary, and the relationships between entities that hold across modalities.  
  
Cutsio resolves one kind of entity, people, through `face_identity_extractor@v1` (SCRFD for detection, ArcFace for identity embedding). That is entity resolution and it works. It is not a declared [taxonomy](/taxonomies), and the difference matters if you are copying this.  
  
An identity extractor gives you "these two faces are the same person". A taxonomy gives you a name for that person that other collections can join against, a place to attach governance, and a definition that survives swapping the face model. Cutsio does not need the second thing yet, because people are the only entity type in their queries. A deployment that added products, brands or locations would need it on the day the second entity type arrives, and retrofitting a vocabulary after several collections already encode their own labels is the expensive version.  
  
So this layer is present in its narrow form and absent in its general form. Saying so is more useful than claiming seven for seven.  
  

## Layer 5: retrievers

  
  
Two retrievers serve the product. One takes a natural-language description or a reference frame and ranks by scene similarity. One takes a photograph and returns every moment that person appears.  
  
The editing UI binds to those two names. It does not know which embedding model produced the vectors, how many candidates each stage requests, or which collection answered. That indirection is the entire point of the layer: the platform team can replace the encoder, and the application keeps asking for the same capability.  
  
What breaks without this layer: applications bind to indexes, and changing an index becomes a migration for every caller that touched it. See [retrievers](/retrievers) for the stage model.  
  

## Layer 6: query planning

  
  
This is the layer I can say the least about from the outside, and I would rather say that than fill it in.  
  
What is visible: stage order, filter pushdown and candidate budgets are configuration on the retriever rather than logic in the editing client, and visual search returns in under a second against a library measured in thousands of hours. What is not visible from the published material is the candidate budget per stage or how the plan changes under load.  
  
The general claim still holds and is worth stating plainly: when the plan lives in the caller, cost is whatever the caller guessed, and nobody discovers the guess was wrong until the corpus is large enough to make it expensive.  
  

## Layer 7: the application

  
  
The editor types what they remember about a shot, or drops in a frame, or uploads a photo of an actor. Results come back with timestamps, which is what makes them usable rather than merely correct. Every ingestion, conversion and search is logged with lineage, so a producer can ask what was processed and when.  
  
Their founder's description of the change is the shortest version of the thesis: "Our editors used to lose half their day hunting for the right take. Now they describe what they need and the exact frame comes back in seconds."  
  

## What the numbers actually support

  
  
Four measured changes, from the published case study:  
  

Metric

Before

After

Footage search time

35 min average

under 15 sec

Editor time spent searching

35% of workday

5% of workday

Raw format handling

Manual DaVinci conversion

Native R3D and ARRI decode

Finding a person on screen

Manual scrubbing

Upload a photo

  
  
Read those as evidence for a specific claim rather than for the general one. They show that manufacturing semantics at write time and serving them through named contracts beat manual search for this workload, at this corpus size, with these two query types. They do not show that the architecture scales to a hundred collections, and no honest single deployment would.  
  

## Where this deployment does not prove the thesis

  
  
Three places, named so you can judge the rest:  
  
The taxonomy layer is narrow, as above. One entity type, resolved by an extractor rather than governed by a vocabulary.  
  
Nothing here federates live across sources. Features are materialized at write time and every query plans over materialized state, which is the practical answer today and is also the easy case. A planner that decides per predicate whether to read materialized features or push a query into a live source is a real design space and this deployment does not enter it.  
  
The model-versioning claim is untested here. The layer is supposed to let you replace an encoder without breaking callers. Cutsio has not yet had to, so the property is designed for rather than demonstrated. That is the one I would most want a second deployment to prove.  
  

## What to copy

  
  
*   Make eligibility a two-way door. Index on a flag, unindex when the flag clears, and reconcile on sync rather than trusting the webhook alone.
*   Give the cheap path priority in your extractors. Passing standard formats through untouched is what kept the GPU bill attached to the footage that actually needed work.
*   Split collections by embedding space, not by convenience. Anything you might want to filter on independently needs its own home.
*   Name the retrieval capability before the application needs a second one. Two named contracts are cheap to add on day one and expensive to introduce after five services have each written their own query.
*   Write down which layer you are missing. This deployment is missing a governed vocabulary and it is fine, because they know it and know what will force the issue.
  
  
The architecture these map onto, stated vendor-neutral and in full, is in [the semantic layer for unstructured and multimodal data](/semantic-layer). Two of the gaps named above have their own arguments: the materialization boundary is worked through in [live versus materialized semantic features](/guides/live-vs-materialized-semantic-features), and the versioning property this deployment has not had to exercise is [what the semantic contract is for](/guides/semantic-contract-for-unstructured-data).

## Run this on your own data

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this guide describes. Build starts at $25/mo for up to 1M vectors.

[Search your own archive](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)[Read Docs](/docs)

[Explore MVS](/mvs)[See pricing](/pricing)

## Related guides

[

Architecture

### What Does a Semantic Layer Compile To When the Target Is Not SQL?

A structured semantic layer compiles a named concept into SQL. Over unstructured data it compiles a named capability into a retrieval plan. The operator set, the guarantees a retrieval plan gives up, why the cost model is inverted, why the optimizer has no statistics, and why text-to-SQL does not cover this.

Read guide →](/guides/semantic-layer-beyond-sql-retrieval-plans)[

Architecture

### Should Semantic Features Be Materialized or Computed at Query Time?

The latency arithmetic that decides it, what the query-time path can actually afford, why staleness and re-extraction are the price of materializing, the hybrid split every production system lands on, and the three problems live semantic federation across arbitrary sources still has to solve.

Read guide →](/guides/live-vs-materialized-semantic-features)[

Architecture

### What Is a Semantic Contract? The Interface Between Extraction and Retrieval

A semantic contract is what an application binds to instead of an index: what one result unit is, which model produced it, what the score means, which fields are filterable and at what path, how fresh it is, and what happens when a stage fails. Includes what counts as a breaking change and a worked declaration.

Read guide →](/guides/semantic-contract-for-unstructured-data)