The Short Answer
A semantic contract is the named, versioned declaration that an application binds to instead of binding to an index. It states what one unit of meaning is, which model produced it, what the returned score means, which fields can be filtered and at what path, and what is allowed to change without telling you.
In a structured semantic layer that declaration is a metric or dimension definition, and the schema underneath it already carries most of the meaning. Over video, audio, images and documents the declaration has to carry more, because the meaning did not exist until a model manufactured it, and the model can be replaced.
What the contract is protecting
Every system that searches unstructured content ends up with the same coupling problem. An application wants "find the moments this person appears." What it gets handed is a collection name, a vector field, a similarity metric, a candidate count, and a filter syntax. Those five things are implementation. When any of them changes, every caller that hardcoded them changes too.
The coupling is invisible while there is one caller. It shows up when the platform team wants to swap an encoder. The vectors move, the score distribution shifts, and three services that each picked their own similarity threshold start returning different result sets on the same query. Nobody wrote the thresholds down anywhere, because a threshold felt like a tuning knob rather than an interface.
A contract makes that a versioned change with a named owner.
The seven things a multimodal contract has to declare
| Field | What it pins down | What breaks when it stays implicit |
| Unit | What one result represents: a 4-second segment, a detected face, a document page, a spoken phrase. See how the layers below produce units | Callers assume object-level results, get segment-level, and dedupe wrong |
| Provenance | The extractor and model version that produced the feature | An encoder upgrade silently changes what the same query returns |
| Space | The embedding space, its dimensionality and its distance metric | Two features get compared in spaces that were never aligned |
| Score semantics | Whether the number is a cosine, a fused rank, or a calibrated probability, and what range is meaningful | Every caller invents a cutoff, and the cutoffs disagree |
| Filterable surface | Which fields can be filtered, sorted and faceted, at the exact path they appear on read | A filter matches nothing, returns empty, and empty reads as a legitimate answer |
| Freshness | How long after ingestion a unit becomes retrievable | A UI shows a file as indexed while the query that would find it is still empty |
| Failure mode | What happens on a stage timeout: an error, a partial result, or a silent truncation | A degraded pipeline reports success and the fallback never fires |
Score is part of the contract, and it usually is not in one
Ask any team running vector search what a score of 0.72 means and you will get a shrug followed by "it's a cosine." That answer is only half a specification. Over normalized CLIP embeddings a cosine sits in a narrow band where 0.2 is a real difference. Swap in a text encoder trained on a different objective and the same measure spreads much wider. Fuse two retrievers with reciprocal rank fusion and the output is not a similarity at all, it is a rank aggregate whose absolute value has no meaning across queries.
So a threshold hardcoded in an application is a claim about the model, made by someone who did not know they were making it. Move to a new encoder and the claim is wrong, quietly, in the direction of returning fewer results.
The contract fix is small. Publish the score type, publish the intended operating point, and if the layer can afford it, publish a calibrated score instead of a raw distance. Calibrating similarity scores covers the mechanics of getting from one to the other.
What counts as a breaking change
This is the question the contract exists to answer, and it has a specific answer for unstructured data that the SQL version does not need.
Breaking. Changing the unit. Changing the embedding model or its version, unless you can show the score distribution and the top-k overlap held. Removing a filterable field, or moving it to a different path. Narrowing what the retriever searches. Tightening a default candidate budget in a way that drops recall.
Not breaking. Adding a new filterable field. Adding a stage that improves precision while the returned shape holds. Changing the index type, the shard layout, or the storage engine. Re-extracting with the same model version after fixing a pipeline bug.
The interesting middle. Re-extracting with a newer version of the same model family. The shape is identical, the field names are identical, and the results move. Treat it as breaking until measured, because the only honest way to classify it is to run both versions against a query set and compare. Embedding portability and versioning goes into why the vectors are less transferable than the shared model name suggests.
The contract has to hold on every surface, not just on read
A contract that describes the read path and silently differs on the filter path is worse than no contract, because a filter that matches nothing returns an empty result rather than an error, and empty is indistinguishable from a legitimate answer.
This has a concrete shape. A document is returned with
metadata.brand at the top level. A caller writes a pre-filter on metadata.brand. The payload index was built on an internal path, so the filter reaches nothing and the search returns zero results with a 200. Every downstream check records that as correct exclusion.The rule worth writing into the contract: a field is addressable at the same path on every surface that exposes it. Read, filter, sort, facet, group. Anything the read path shows a caller, the filter path has to reach on the same documents at the same path.
Enforcing that needs a test rather than a convention, because the failure is silent by construction. For every field the read path exposes, assert the filter path reaches the same documents.
Writing one down
A contract does not need a new file format. It needs the fields to be somewhere a caller can read and a reviewer can diff. Here is the shape, expressed as configuration on a retriever:
name: person_moments
version: 2
unit: video_segment # not the video, not the frame
returns:
document_id: string
collection_id: string
start_time: float # seconds into the source object
end_time: float
score: {type: cosine, space: face_identity_v1, operating_point: 0.38}
provenance:
extractor: face_identity_extractor
extractor_version: 1.2.0
embedding_model: arcface-r100
embedding_dim: 512
filterable:
- metadata.production_id # same path the read path returns
- metadata.shot_type
- created_at
freshness: p95 under 4 minutes from object write
on_stage_timeout: error # never a silent partialTwo things about that file matter more than its syntax. It is versioned alongside the code that serves it, so a change to it shows up in review. And the caller never names a collection, an index or a vector field, so none of those are part of the interface.
The alternative is what most teams have today. Applications call a vector database directly, each one carrying its own copy of the retrieval logic: which collection, which filters, how many candidates, which reranker, which threshold.
The cost lands in four places. Model upgrades become coordinated migrations across every service. Retrieval logic gets copied into application code where no reviewer with retrieval context ever sees it. Cost is whatever each caller guessed when it hardcoded a candidate count, and the guess is discovered to be wrong when the corpus gets large enough to make it expensive. Access control has nothing to attach to, because a raw index scan is not a permissionable operation.
Each of those is cheap to fix while there is one consumer. What the contract resolves into, once a caller invokes it, is a retrieval plan rather than a SQL statement, and that compilation target has its own set of operators and guarantees. Retrievers are the primitive Mixpeek uses to hold the contract, and what a semantic layer looks like in production walks one deployment layer by layer, including the places where its contracts are thinner than the model suggests.
A semantic contract is a named, versioned declaration that an application binds to instead of binding to a specific index. It specifies what one result unit represents, which model and version produced the underlying features, what the returned score means, which fields are filterable and at which path, how fresh results are, and what happens when a stage fails. The storage engine, the embedding model and the query plan underneath can all change while the contract holds.
A schema describes the shape of stored data. A semantic contract describes the shape and the meaning of a retrieval result, including things a schema has no place for: which model manufactured the semantics, what the similarity score is measured in, and what recall guarantee the plan offers. Over unstructured data the meaning is model-dependent, so provenance and score semantics belong in the interface rather than in a runbook.
Changing the unit of a result, swapping the embedding model or its version without showing that the score distribution and top-k overlap held, removing or relocating a filterable field, narrowing what the retriever searches, and tightening a candidate budget in a way that drops recall. Adding a filterable field, changing the index type, and re-extracting with the same model version after a pipeline fix are all non-breaking.
Because a threshold hardcoded in an application is a claim about the model that produced the vectors. Cosine distributions differ by encoder and by training objective, and a fused rank from two retrievers is not a similarity at all. When the score type and its intended operating point are undeclared, every caller picks its own cutoff, the cutoffs disagree, and an encoder change breaks all of them in the direction of returning fewer results.
Yes, and it should. Customer metadata keeps the customer's path. Anything the platform derives about that data goes somewhere clearly internal. The contract names which paths are filterable, and the same path has to work on read, filter, sort and facet, otherwise a filter can match nothing and return an empty result that looks like a legitimate answer.
What binding to an index costs instead
The alternative is what most teams have today. Applications call a vector database directly, each one carrying its own copy of the retrieval logic: which collection, which filters, how many candidates, which reranker, which threshold.
The cost lands in four places. Model upgrades become coordinated migrations across every service. Retrieval logic gets copied into application code where no reviewer with retrieval context ever sees it. Cost is whatever each caller guessed when it hardcoded a candidate count, and the guess is discovered to be wrong when the corpus gets large enough to make it expensive. Access control has nothing to attach to, because a raw index scan is not a permissionable operation.
Each of those is cheap to fix while there is one consumer. What the contract resolves into, once a caller invokes it, is a retrieval plan rather than a SQL statement, and that compilation target has its own set of operators and guarantees. Retrievers are the primitive Mixpeek uses to hold the contract, and what a semantic layer looks like in production walks one deployment layer by layer, including the places where its contracts are thinner than the model suggests.
Frequently Asked Questions
What is a semantic contract in a semantic layer?
A semantic contract is a named, versioned declaration that an application binds to instead of binding to a specific index. It specifies what one result unit represents, which model and version produced the underlying features, what the returned score means, which fields are filterable and at which path, how fresh results are, and what happens when a stage fails. The storage engine, the embedding model and the query plan underneath can all change while the contract holds.
How is a semantic contract different from a database schema?
A schema describes the shape of stored data. A semantic contract describes the shape and the meaning of a retrieval result, including things a schema has no place for: which model manufactured the semantics, what the similarity score is measured in, and what recall guarantee the plan offers. Over unstructured data the meaning is model-dependent, so provenance and score semantics belong in the interface rather than in a runbook.
What counts as a breaking change to a retrieval contract?
Changing the unit of a result, swapping the embedding model or its version without showing that the score distribution and top-k overlap held, removing or relocating a filterable field, narrowing what the retriever searches, and tightening a candidate budget in a way that drops recall. Adding a filterable field, changing the index type, and re-extracting with the same model version after a pipeline fix are all non-breaking.
Why does the similarity score need to be in the contract?
Because a threshold hardcoded in an application is a claim about the model that produced the vectors. Cosine distributions differ by encoder and by training objective, and a fused rank from two retrievers is not a similarity at all. When the score type and its intended operating point are undeclared, every caller picks its own cutoff, the cutoffs disagree, and an encoder change breaks all of them in the direction of returning fewer results.
Can a contract cover fields the customer supplied?
Yes, and it should. Customer metadata keeps the customer's path. Anything the platform derives about that data goes somewhere clearly internal. The contract names which paths are filterable, and the same path has to work on read, filter, sort and facet, otherwise a filter can match nothing and return an empty result that looks like a legitimate answer.