Inverting the semantic layer

In short

Every stack already has a semantic layer: the schema, the embeddings, the prompt templates. It describes the shape of your data, and leaves business meaning to be guessed at from that shape, one query at a time. Reversing that direction adds no new components. It changes which questions are answerable at all.

The short version

• Grounding gives a system something to run into, so its failures become loud instead of silent, the trade worth making for anything that gets reviewed.
• Context is a structure problem. The meaning of a field lives in its position, so a bigger prompt cannot supply it.
• A schema stores facts and a knowledge graph arranges them. Only a written-down domain model states what they mean.
• Inverting the layer makes answers verifiable by path, collapses the search space to identified entities, and gives every agent one shared definition.
• Retrieval stays the right tool for questions whose answers live in documents; structure answers the ones that live across records.

01 It was never the query language

An analyst asks two agents the same question: which employers do our customers work for, and in which cities?

SYSTEM A

"They work for Halvorsen Marine and Pyle & Dunn, based in Slough and Reading."

SYSTEM B

ERROR · property city not found on Employer. city is recorded on Customer. Retrying against the right entity.

Neither system can answer the question, because nothing in the estate records which city an employer sits in. System A answered anyway. It reads perfectly, carries no warning, and flags nothing for review. Had the cities been attached to the wrong entity entirely, the sentence would have looked identical.

System B didn't know more than System A. But it knew what was missing.

This is a useful place to refresh our memories on part one of this blog series. The short version: some questions have answers that exist across records rather than in any one of them. Acme is owned by Kestrel, Kestrel by Alder, Alder by Acme, every filing correct, and the beneficial ownership requirement still cannot be satisfied because the chain never reaches a person. Similarity search cannot rank the record that matters, because what makes it matter is not written in its text. Handing the model the whole subgraph doesn't save it either, because a cycle is a computation to perform rather than a thing to notice in prose.

The fair objection was that none of this is hard. A recursive query finds the loop in milliseconds and any competent engineer is able to write it. True, but that's beside the point, because a query is an answer to a question you already knew how to ask. The analyst holding the alert knows what to look for and cannot express it in SQL. The engineer who can write the SQL doesn't know that a loop is what no beneficial owner reduces to.

Which leaves two things to work out: (1) what it would take for an answer to be trustworthy rather than merely fluent, and (2) where the knowledge that connects those two people is supposed to live. There's also one popular shortcut to rule out on the way.

02 Confidence isn't evidence

For a decision that gets audited, a score is not enough. The analyst needs the path.

Text2SQL gives you a query and a result, both readable, both downstream of the choice that mattered. Semantic search gives you a ranked list, scored by embeddings that report numbers rather than reasons. Absent ground truth, the standard way to evaluate that list is to ask a language model whether the question, the context, and the answer look like they belong together.

A language model, guessing whether a language model retrieved the right thing. That is a vibe check with extra steps.

For "what's the balance?", a vibe check is fine. For a determination that goes into a regulatory filing and gets defended to an examiner two years later, the analyst needs to see which entities and which relationships produced the answer, and a bare score shows none of it.

This is also why grounding matters more than it first appears. It is what let System B stop. A system that knows what exists, and what connects to what, has something to run into; without it, there is nothing to catch.

Grounding makes a system's failures loud instead of silent; it does not make the system correct. For anything that gets reviewed, that is the trade worth making.

03 Structure is not quantity

A structure problem doesn't yield to a bigger prompt.

The natural move when a model lacks context is to give it more. More schema, more examples, more documentation, a bigger window. That treats context as a volume problem. It isn't, and the smallest possible example makes the point.

Take a field called address. In customer_profiles it holds a residence: personally identifiable information (PII), tightly regulated, down to what it can be joined against and who can see it. In branch_locations the identical field, same type, same string format, holds a public street address that the bank prints on its own website.

A static rule says address = PII and flags every column of that name across the estate, most of them wrongly, until analysts are requesting exemptions faster than governance can grant them. A bigger prompt doesn't resolve it either, because the information that distinguishes the two cases was never in the field. What resolves it is the values, the neighboring columns, and the table's own meaning, taken together.

The classification is a property of the field's position in a structure. The field alone does not carry it.

Structural questions are the same phenomenon at scale. The missing thing was relationships all along.

04 What actually answers them

The analyst's concepts have to exist somewhere the machine can use them.

The analyst can't write the query, and the schema can't tell a model what the query means. Those are the same missing thing, which is a written-down model of the domain: entity types and the relations between them, with data mapped into it, so that questions get asked and answered in Customer, Account, owns and isManagedBy rather than in tables, fields and keys.

It's worth being precise about what that is, because three things routinely get called by each other's names: the schema, the knowledge graph, and the ontology.

Which is why the middle layer is the trap. Arranging the same fact as nodes and edges makes the cycle physically present in the data, and that feels like progress, but nothing in the arrangement states that owns runs in a direction, composes along a path, or terminates only at a person.

Now note what this reverses. A semantic layer already exists in every stack. It is the schema, the embeddings, the prompt templates. It describes the shape of the data and hopes business meaning can be inferred from that shape per query, by a model that was never told what the business means. The alternative points the arrow the other way: describe the business's shape, with a human validating, and map the data into that. Same components, opposite direction.

Three things follow from the inverse layer:

CONSEQUENCE 1 · VERIFIABILITY

The path is readable, so a disposition can be defended. When an agent traverses (Customer)-[:isAnalyzedBy]-(RiskAnalyst), the entities and relations behind the answer are visible and the path can be read directly. That is the difference between an answer you would have to audit to check and one that arrives with its own audit trail.

CONSEQUENCE 2 · A COLLAPSED SEARCH SPACE

The search covers identified entities, not a ranked corpus. Once a question resolves to specific entity types and relations, the system searches the entities that are actually relevant rather than an entire embedded corpus ranked by cosine similarity and truncated at k. The window stops being the thing that decides whether the answer is correct.

CONSEQUENCE 3 · REUSE

Knowledge and logic decouple, so the definition stops being private. Many agents share one domain model instead of each keeping its own drifting theory of what a customer is. This one sounds like housekeeping until two agents answer the same question differently in front of the same user.

05 Where this breaks

Structure isn't free, isn't always right, and doesn't eliminate failure.

CAVEAT 1

Not every question is structural. The most important caveat, and the easiest one to lose sight of once the graph starts working. When an answer genuinely lives in a document, as in exploratory research or open-ended questions over prose, semantic search is the correct engineering choice.

CAVEAT 2

Ontology design is hard. It has defeated a couple of decades of enterprise data initiatives. Getting entity types and relations right takes real domain expertise, and the failure mode is nasty: a wrong ontology can be worse than none, because it is confidently wrong in a structured way. Automated inference alone doesn't get there. A human domain expert stays in the loop.

CAVEAT 3

Agentic systems are slow. A multi-agent loop that interprets the question, selects concepts, builds the query, executes it, and summarizes the result costs real seconds. That's a property of agentic architectures generally rather than of graphs; the underlying graph query returns near-instantly. It's a live problem for everyone working in this space, us included, and one we'll write about separately.

06 Why this matters more in an enterprise

Undocumented business knowledge doesn't compound. It gets rebuilt, per query, forever.

Knowing that a chain of ownership pointers is what the regulation means by tracing, and that a loop in it means the trace can never terminate, is ordinary domain expertise. Someone on the floor has held it for years. What's costly is where it sits, which is nowhere the company owns. It lives with the people who hold it, and gets re-derived from scratch on every query, by every agent.

That is specific to enterprises. A consumer product has no decades-old agreement about what a claim is or which entity types matter. An enterprise is largely made of that agreement, spread across fragmented systems, and it is the actual asset. While it stays unwritten, it cannot compound, which rarely shows up as an incident and steadily shows up as cost.

Which reframes the expensive mistake. Reaching for embeddings is fine; embeddings are good, and for half the questions in the building they are the right tool. The expensive mistake is spending two quarters building a retrieval pipeline for questions that were never retrieval questions, and finding out at the end.

An ontology is where that expertise accumulates instead of dispersing, in the vocabulary the business already uses, refined by the people who know the domain as it changes. It gives everyone else a way to ask for their judgment rather than replacing it.

The bet we're making at zaimler is that this can be inferred rather than hand-modeled, with a human validating rather than authoring. The prize is an ontology that builds itself and keeps itself current, in the vocabulary the business already speaks. That is the hard, interesting part, and it is what we're building.

Go back to the two answers at the top. The one that read perfectly was a well-built pipeline doing exactly what it was designed to do, over a stack that had no way to know the question was unanswerable; no model failed. The other one had simply been told what the business means. That is the whole of it, across both posts: the expertise is already in the building, and what it has never had is somewhere to live that the next person, or the next agent, can read.