What a Semantic Layer Looks Like in Production

The Short Answer

A semantic layer for unstructured data is easy to describe and easy to dismiss as an abstraction someone drew on a whiteboard. So here is one that runs in production, layer by layer, with the parts that are missing named as missing.

The deployment is Cutsio, a collaborative editing platform for post-production teams working with RED R3D and ARRI RAW footage. Editors there were spending about 35% of the workday looking for clips. They now spend about 5%, and a search that averaged 35 minutes returns in under 15 seconds. The full write-up with their numbers is the Cutsio case study.

What makes it worth reading as architecture rather than as a customer story is that every layer in the general model is doing an identifiable job, and you can see what each one is protecting against.

The deployment in one paragraph

Raw camera footage lands in Cutsio's Mux infrastructure. A selective sync decides which assets Mixpeek indexes. A custom extractor transcodes the two raw cinema formats and passes everything else through untouched. Three chained collections turn each video into timestamped segments with scene-level embeddings, then run face detection and identity embedding over those segments. Two named retrievers serve the editing UI: one for visual search by text or reference frame, one for finding every moment a given person appears. An editor uploading an R3D file has it searchable within minutes.

Layer 1: raw objects

Cutsio remains the source of truth for their media. Mux carries the sync hint. Mixpeek indexes and does not take custody.

The mechanism is worth copying. Each asset carries a passthrough metadata flag, and metadata_filters on the sync only index assets whose flag matches. A video.asset.updated webhook re-evaluates the filter when the flag changes, and reconcile_on_sync unindexes assets that stop qualifying.

That last clause is the part most integrations skip. An index that adds on eligibility and never removes on ineligibility drifts silently, and the drift shows up as a search result nobody can explain months later. Eligibility is a two-way door here.

What breaks without this layer: derived data with no source to re-derive from. Every model upgrade turns into re-collecting the corpus from the customer.

Layer 2: decomposition and extraction

This is where the semantics come into existence, and it is the reason a semantic layer for video cannot be a thin wrapper over a warehouse.

Nothing in an R3D file says which scene contains a product, or who is on screen at 04:12. A model has to produce that, and producing it costs real GPU time. Cutsio's pipeline runs a custom conversion extractor at v3.1.0 that passes standard formats through and transcodes only R3D and ARRI RAW, using REDline and the ARRI Reference Tool already present in the engine image. That removed a DaVinci Resolve install from every editor's machine.

Two properties of this layer generalize:

Extraction is a write-time cost. It is too slow to run inside a query. Everything downstream is planning over what was already extracted.

Passthrough is a feature. An extractor that transcodes everything would have burned GPU on footage that needed nothing done to it. The cheap path has to be the default path.

What breaks without this layer: the semantics never exist, and retrieval degrades to keyword search over filenames. Which is the thing editors were doing manually.

Layer 3: semantic features

One video yields many units at different granularity, and each unit wants its own home.

The pipeline is three chained collections rather than one: format detection and transcoding, then multimodal decomposition into timestamped segments carrying scene-level embeddings for composition, subjects, motion, color palette and on-screen text, then face detection and identity embedding per segment.

Chaining them is not an implementation detail. A face embedding and a scene embedding do not share a vector space, and collapsing a video into one vector destroys the ability to filter on either. Separate collections keep each feature addressable, filterable, and independently re-computable when its model changes.

What breaks without this layer: one vector per object, and no way to ask for the face without dragging the whole scene along with it.

Layer 4: entity resolution, and where this deployment is honest about a gap

In the general model this layer holds taxonomies and ontologies: the governed vocabulary, and the relationships between entities that hold across modalities.

Cutsio resolves one kind of entity, people, through face_identity_extractor@v1 (SCRFD for detection, ArcFace for identity embedding). That is entity resolution and it works. It is not a declared taxonomy, and the difference matters if you are copying this.

An identity extractor gives you "these two faces are the same person". A taxonomy gives you a name for that person that other collections can join against, a place to attach governance, and a definition that survives swapping the face model. Cutsio does not need the second thing yet, because people are the only entity type in their queries. A deployment that added products, brands or locations would need it on the day the second entity type arrives, and retrofitting a vocabulary after several collections already encode their own labels is the expensive version.

So this layer is present in its narrow form and absent in its general form. Saying so is more useful than claiming seven for seven.

Layer 5: retrievers

Two retrievers serve the product. One takes a natural-language description or a reference frame and ranks by scene similarity. One takes a photograph and returns every moment that person appears.

The editing UI binds to those two names. It does not know which embedding model produced the vectors, how many candidates each stage requests, or which collection answered. That indirection is the entire point of the layer: the platform team can replace the encoder, and the application keeps asking for the same capability.

What breaks without this layer: applications bind to indexes, and changing an index becomes a migration for every caller that touched it. See retrievers for the stage model.

Layer 6: query planning

This is the layer I can say the least about from the outside, and I would rather say that than fill it in.

What is visible: stage order, filter pushdown and candidate budgets are configuration on the retriever rather than logic in the editing client, and visual search returns in under a second against a library measured in thousands of hours. What is not visible from the published material is the candidate budget per stage or how the plan changes under load.

The general claim still holds and is worth stating plainly: when the plan lives in the caller, cost is whatever the caller guessed, and nobody discovers the guess was wrong until the corpus is large enough to make it expensive.

Layer 7: the application

The editor types what they remember about a shot, or drops in a frame, or uploads a photo of an actor. Results come back with timestamps, which is what makes them usable rather than merely correct. Every ingestion, conversion and search is logged with lineage, so a producer can ask what was processed and when.

Their founder's description of the change is the shortest version of the thesis: "Our editors used to lose half their day hunting for the right take. Now they describe what they need and the exact frame comes back in seconds."

What the numbers actually support

Four measured changes, from the published case study:

Metric

Before

After

Footage search time	35 min average	under 15 sec
Editor time spent searching	35% of workday	5% of workday
Raw format handling	Manual DaVinci conversion	Native R3D and ARRI decode
Finding a person on screen	Manual scrubbing	Upload a photo

Read those as evidence for a specific claim rather than for the general one. They show that manufacturing semantics at write time and serving them through named contracts beat manual search for this workload, at this corpus size, with these two query types. They do not show that the architecture scales to a hundred collections, and no honest single deployment would.

Where this deployment does not prove the thesis

Three places, named so you can judge the rest:

The taxonomy layer is narrow, as above. One entity type, resolved by an extractor rather than governed by a vocabulary.

Nothing here federates live across sources. Features are materialized at write time and every query plans over materialized state, which is the practical answer today and is also the easy case. A planner that decides per predicate whether to read materialized features or push a query into a live source is a real design space and this deployment does not enter it.

The model-versioning claim is untested here. The layer is supposed to let you replace an encoder without breaking callers. Cutsio has not yet had to, so the property is designed for rather than demonstrated. That is the one I would most want a second deployment to prove.

What to copy

Make eligibility a two-way door. Index on a flag, unindex when the flag clears, and reconcile on sync rather than trusting the webhook alone.

Give the cheap path priority in your extractors. Passing standard formats through untouched is what kept the GPU bill attached to the footage that actually needed work.

Split collections by embedding space, not by convenience. Anything you might want to filter on independently needs its own home.

Name the retrieval capability before the application needs a second one. Two named contracts are cheap to add on day one and expensive to introduce after five services have each written their own query.

Write down which layer you are missing. This deployment is missing a governed vocabulary and it is fine, because they know it and know what will force the issue.

The architecture these map onto, stated vendor-neutral and in full, is in the semantic layer for unstructured and multimodal data.