What Does a Semantic Layer Compile To When the Target Is Not SQL?

The Short Answer

A structured semantic layer compiles a named concept into SQL. A semantic layer for unstructured data compiles a named capability into a retrieval plan: metadata filters over extracted features, vector search across one or more embedding spaces, joins that resolve entities across collections, fusion, reranking, and a projection of the fields the caller asked for.

The planning problem has the same shape as query planning over tables. The operators differ, the guarantees are weaker, and the cost model points the other way, because extraction at write time dominates everything that happens at read time.

The operator set

A retrieval plan is built from a small vocabulary. Most systems implement some subset of it, usually without naming the operators, which is why plans end up hardcoded in application code.

Filter. A predicate over stored fields. Structurally identical to a SQL WHERE, with one difference that matters: whether it runs before or after the vector search changes both the result set and the cost, and neither position is always right. Filtered vector search covers the three placements.

Search. Approximate nearest neighbor over one embedding space, returning a candidate set of size k with no guarantee that it contains the true top k. This is the operator with no relational equivalent.

Lexical match. BM25 or a learned sparse retriever over the same units. A different vocabulary for the same corpus, which is why fusion exists.

Fuse. Combine ranked lists from several retrievers into one. Reciprocal rank fusion and score normalization are the usual choices, and they are not interchangeable. Hybrid search fusion is the detail.

Join. Connect units in different collections. Sometimes on a shared identifier, which behaves like a foreign key. Sometimes through a resolved entity in a taxonomy, which behaves like nothing in SQL at all.

Rerank. Score a small candidate set with a model too expensive to run over the corpus. Cross-encoders and listwise LLM rerankers both live here.

Project. Return the fields the caller asked for. Under-appreciated, because dragging full payloads through an agent's context window is a real cost. Payload projection covers it.

What SQL guarantees that a retrieval plan cannot

This is the honest part of the comparison, and skipping it is how a semantic layer gets oversold.

Property

SQL over tables

Retrieval plan over features

Result completeness	Exact. The plan returns every row matching the predicate	Approximate. ANN search returns a candidate set with a recall the plan cannot measure at query time
Ordering	Deterministic under an ORDER BY	Scored, and the score depends on the model version that produced the vectors
Join correctness	Defined by key equality	Defined by similarity or by an entity resolution that has its own error rate
Optimizer inputs	Table statistics, histograms, cardinality estimates	Almost none. Selectivity of a metadata predicate over an ANN traversal is not something the index tells you
Plan equivalence	Two plans returning different rows is a bug	Two plans returning different rows is normal and often correct

The last row is the one that changes how you build. A rewrite that returns the same rows faster is safe in SQL. In retrieval, a rewrite that reorders stages changes the candidate set, so plan changes need evaluation against a query set rather than a correctness proof. Evaluating multimodal retrieval is where that discipline lives.

The join is where the analogy earns its keep

Relational joins match on equality. A person in one table is the same person in another because the ids agree, and nothing in the query has to reason about it.

Nothing in a video says which face belongs to which customer record. Getting from "this face embedding" to "this named person, who also appears in these documents and this transcript" needs a resolution step with a threshold and an error rate. Once resolved, that entity becomes a join key that behaves almost like a foreign key: other collections can reference it, filters can use it, and governance can attach to it.

That is what a taxonomy is doing in this architecture. It is the mechanism that turns a similarity into an identifier stable enough to join on. Without it, every cross-collection query is a similarity search that happens to be pointed at two indexes, and the results cannot be reconciled with anything.

The cost model is inverted

In a warehouse, query execution is the expensive part. Materialized views exist to move work off the read path, and the decision to build one is a trade between storage and query latency.

Over unstructured data, the extraction is the expensive part by orders of magnitude. Running scene detection, transcription and face identity across an hour of video is GPU minutes. The query that reads the results is milliseconds. So materialization is not an optimization here, it is the only way the read path can exist at all. Live versus materialized semantic features works through the arithmetic.

Two consequences follow for the planner.

The first is that the planner's job is mostly to avoid re-extraction. A plan that would need a feature nobody extracted has to fail loudly rather than fall back to computing it inline, because inline extraction blows any interactive latency budget and burns GPU the caller did not ask for.

The second is that the read-side knobs are about candidate budgets. Every stage after the first one costs in proportion to how many candidates the stage before it handed over, and reranking is the stage where that multiplier hurts. Query planning for multimodal retrieval covers how to size those budgets by measuring first-stage recall instead of guessing k.

The optimizer has no statistics

A relational optimizer knows roughly how many rows match country = 'FR' because the table keeps a histogram. A vector index knows nothing equivalent. It cannot tell you how many of its neighbors will survive a metadata predicate, which is exactly the number you need to decide whether to filter before or after the search.

Systems solve this three ways, and all three are worth knowing about because they show up as configuration rather than as an optimizer.

Sample the corpus offline to estimate predicate selectivity, and store the estimate as metadata on the collection. Cheap, stale, usually good enough.

Run the filter first when the predicate is known to be highly selective, accepting that a small filtered subset makes the graph traversal degenerate into something closer to a scan.

Over-fetch and filter afterwards, accepting that a selective predicate can leave you with fewer than k results and a caller who has no way to tell that from a genuinely empty corpus.

That third failure is the silent one. A post-filter that returns 3 results where the caller asked for 20 looks exactly like a query with 3 real answers.

Where it behaves exactly like SQL

Predicate pushdown is the same idea and pays off for the same reason. Naming a stable capability so callers stop writing their own version is the same idea. Reviewing one definition instead of thirty copies of a query is the same idea, and it is most of why the structured semantic layer won its argument in the first place.

Carry over the discipline and the vocabulary. Leave the correctness guarantees behind, because pretending they survived is how teams end up surprised when two supposedly equivalent plans disagree on a result set.

A worked compilation

A capability named product_moments resolves to a plan like this:

1. filter   collection=video_segments,
            metadata.campaign_id = $campaign,          # customer's own path
            created_at >= $since
2. search   space=scene_clip_v2, query=$text, k=400
3. join     video_segments.entity_id -> taxonomy(product), threshold=0.42
4. filter   product.category in $categories
5. rerank   cross-encoder over top 400 -> top 40
6. project  document_id, start_time, end_time, product.name, score

The declaration a caller reads is the semantic contract, and it holds none of the above. The caller sends a text query, a campaign id and a category list. It does not know that step 2 uses CLIP-family embeddings at 768 dimensions, that step 3 is a taxonomy lookup with a threshold, or that step 5 exists. Replace the encoder in step 2 and the caller's code does not change.

Why text-to-SQL does not cover this

The reflex when someone says "semantic layer beyond SQL" is to reach for a translation layer: let a model write the query. That works when the semantics are already in the schema and the model's job is syntax.

Here the model would have to invent the semantics. There is no column that says which scene shows the product. The extraction that would create that column has not run, and asking a language model to write a query against a feature nobody extracted produces a syntactically valid plan that returns nothing.

The order matters. Decide what to extract, extract it, name it, then let a model select among named capabilities. How agents choose retrieval capabilities is the next piece of that.

Frequently Asked Questions

What does a semantic layer for unstructured data compile to?

A retrieval plan rather than a SQL statement. The plan is composed of metadata filters over extracted features, approximate nearest neighbor search across one or more embedding spaces, lexical matching, joins that resolve entities through a taxonomy, fusion of ranked lists, reranking with a model too expensive to run corpus-wide, and a projection of the requested fields. The named capability is what the caller binds to, and the plan behind it can be rewritten without changing the caller.

Why can't a semantic layer for video just generate SQL?

Because the columns a query would need do not exist until a model creates them. Nothing in an hour of footage states which scene contains a product or who is speaking at 04:12. A text-to-SQL layer assumes the semantics already live in the schema and only the syntax is missing. Over unstructured data the semantics are manufactured at write time by extraction, so the layer has to decide what to extract before any query can be planned.

What guarantees does a retrieval plan give up compared to SQL?

Exact result completeness, deterministic ordering, and equality-based join correctness. Approximate nearest neighbor search returns a candidate set with a recall the query cannot measure, ranking depends on the model version that produced the vectors, and cross-collection joins go through an entity resolution step that carries its own error rate. Two plans that return different result sets can both be correct, which means plan changes are validated by evaluation against a query set rather than by proof.

How does query cost differ between a warehouse and a semantic layer over unstructured data?

The dominant cost moves from read time to write time. Query execution is what a warehouse optimizes, so materialized views are a trade between storage and latency. Extraction over media costs GPU minutes per hour of video while the query that reads the results costs milliseconds, so features are materialized as a precondition for the read path existing. Read-side tuning is then mostly about candidate budgets, since every stage costs in proportion to what the previous stage handed it.

How does a join work when there are no foreign keys?

Through entity resolution. A face embedding, a logo crop and a mention in a transcript are connected by resolving each to the same entity in a taxonomy, using a similarity threshold. Once resolved, the entity id behaves like a foreign key: other collections reference it, filters use it, and governance attaches to it. The resolution step has an error rate, which is the price of joining across modalities that share no key.