Data governance
7 min

Semantic Layer vs Text-to-SQL: What a Refusal Is Worth

Key takeaways

  • Text-to-SQL turns a question into a query against the schema and runs it. A semantic layer answers only from definitions written in advance and declines the rest. That refusal is the product.
  • dbt's 2026 benchmark puts numbers on it: on the full question set the dbt Semantic Layer scored 72.7% against 64.5% for text-to-SQL, and every point the layer lost was a refusal.
  • A refusal is only as good as the definition behind it. The benchmark scores answers against a fixed key and never tests whether the definition inside the scope was the one the business agreed.
  • Scope is whatever someone modelled. On dbt's page an LLM widened it with three models and no code; in an enterprise the scope should widen only when a named owner approves the addition.
  • The arrangement that holds up for an agent is one layer with definitions resolved before generation and join paths drawn from the concept model, so the refusal becomes a boundary the agent cannot see past.
In this article
Share

A finance director at an insurer asks the assistant for loss ratio by product line. One version writes SQL against the warehouse and returns a number at once. The other routes the question through a semantic layer and returns an error, because nobody modelled product line on loss ratio. The semantic layer vs text-to-SQL question is which of those two answers belongs in front of an auditor.

Text-to-SQL turns a question into a query against the schema and runs it. A semantic layer routes the question through definitions written in advance and refuses anything outside them. The first answers everything, some of it wrongly. The second answers what was modelled and says so when it cannot. The refusal is what a semantic layer sells. This article is about what it is worth.

A refusal is worth exactly as much as the definition behind it. A semantic layer that declines a question outside its scope protects the board deck. It protects nothing if the definition it answered with was never agreed, and "in scope" is only as wide as the concepts someone modelled.

What is the difference between a semantic layer and text-to-SQL?

The difference is what sits between the question and the warehouse. In our post What Is a Semantic Layer? we set out the four things a layer holds: concepts, metrics, mappings, and entitlements. Text-to-SQL holds a schema and a model's guess about what it means.

Two paths from a question to a number

QuestionText-to-SQLSemantic layer
What does the model receive?The schema, or as much of it as fits in contextNamed metrics, dimensions, and the joins between them
What happens outside the modelled scope?A query is written anywayThe request is refused
How does a wrong answer look?A plausible numberAn error message
Who can check it?Whoever can read the generated SQLAnyone, against the shared definition
What does it cost up front?Nothing beyond the schemaThe modelling

What does dbt's 2026 benchmark show?

The measured evidence comes from dbt Labs, which sells a semantic layer: "Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update" (Jason Ganz and Benoit Perigaud, dbt Developer Blog, 2026-04-07). The test is the ACME Insurance benchmark by Juan Sequeda and colleagues at data.world. In their words, "11 questions, each run 20 times, across multiple LLMs" (dbt Developer Blog). The page states its own caveat: "we loaded the entire schema as context, which isn't practical for larger datasets."

Our post What Is a Semantic Layer? quotes the in-scope and out-of-scope pairs for Sonnet 4.6. Here is the rest.

The full question set, 2023 against 2026

RunText-to-SQLdbt Semantic Layer
2023, all questions32.7%60.5%
2026, all questions, either model64.5%72.7%
2026, GPT-5.3 Codex, in scope51.2%100.0%
2026, GPT-5.3 Codex, out of scope100.0%0.0%

Text-to-SQL "nearly doubled, from 32.7% to 64.5% on the full question set". The layer gained less, because its ceiling was set by scope. "The trade-off is coverage: the Semantic Layer can only answer questions that fall within the scope of what's been modeled" (dbt Developer Blog). Out of scope, one model answered every question correctly through text-to-SQL and none through the layer.

What is a refusal worth?

dbt's sentence on the difference is the one every page on this query quotes: "With text-to-SQL, failure looks like a plausible but incorrect answer. With the Semantic Layer, failure looks like an error message" (dbt Developer Blog). The error message has a value and a price.

What the refusal buys, and what it costs

  • The value: a visible gap. A refused question gets escalated. A wrong number gets published. For a regulatory return the second costs more by any accounting.
  • The value: a fixed answer. Within scope, "both models now return correct results 100% of the time" (dbt Developer Blog). The definition did the work.
  • The price: coverage. With either model the layer scored 72.7% on the full set and 0.0% outside its scope (dbt Developer Blog). Every point it lost was a refusal.
  • The price: the modelling. The scope was widened with "3 additional models", after which the layer answered "every question in the benchmark" (dbt Developer Blog). Cheap for eleven questions. At enterprise size the modelling is the whole job.

A catalog vendor's page on the self-serve decision (published 2026-08-19, updated 2026-09-18) lands in the same place. It calls a semantic layer "governed and accurate on the questions it was built to answer, capped at those questions". Its failure-mode row reads "Silent gap; question falls outside the model, no answer". Both pages treat the refusal as the end of the analysis. This one starts there.

What does a refusal rest on?

A refusal is a claim with two halves: this question is outside the definitions I hold, and the definitions I hold are right. The benchmark tests the first half. Each answer is scored against a fixed key. Nothing in the method tests whether the layer's definition of "customer" was the one the business agreed, who agreed it, or whether it has expired.

Four things a refusal assumes

AssumptionWhat the benchmark checksWhat an enterprise has to check
The definitions inside the scope are correctAgainst a fixed answer keyAgainst a named owner's approval, with a date on it
The scope covers the questions that matterEleven questions chosen in advanceWhatever the board, the regulator, and the agent ask next
The join paths are the right onesThe modelled joins returned the key's numberThe modelled joins are the only ones the business permits
The refusal reaches someone who can actAn error message in a test harnessAn owner, a route to widen the scope, and a record of the decision

Our post Ground Truth: SME-Approved vs Inferred Metadata draws the line between a definition a person approved and one a system inferred. A refusal from an inferred definition is a guess declining to guess further. Who gets to approve one when the consumer is a model is the subject of our post What Is Data Governance for AI?.

Who decides what is in scope?

On dbt's page the scope was widened by a model. "We prompted an LLM to create as few dbt models as possible to answer all 11 questions via the Semantic Layer. It created 3 new models, joining a few tables together and creating the corresponding semantic models, without us writing any code" (dbt Developer Blog). Accuracy rose to 98.2% for Sonnet 4.6 and 100.0% for GPT-5.3 Codex. The page describes what the LLM created and what the scores became. Any review of the three models sits outside the write-up.

That is the whole question for an enterprise. A layer's scope grows when someone adds a model. If that someone is an LLM and nobody confirms the join it chose, the scope was widened by inference. Every answer inside the new scope rests on a guess. Inferred context is a guess with good grammar, and a confident refusal from a guessed scope is the same guess in a uniform.

Widening the scope, two ways

StepGenerated and appliedProposed and confirmed
A question falls outside scopeThe agent refusesThe agent refuses
Someone adds the missing modelAn LLM writes the join and the semantic modelAI proposes the concept, the join, and the mapping, with a score
The addition becomes authoritativeOn mergeWhen the named owner approves it, with a version and a date
The next refusal meansOutside what a model wroteOutside what the business agreed

The catalog vendor's page says the right thing here and stops: "Governance questions need an owner before they need a tool." Its own table answers "Who owns 'the truth'" with a team name. An owner is a person. dbt's benchmark describes its layer as "a structured ontology (metrics, dimensions, entities, and the relationships between them)". We would add one item to that list: an owner per definition. Our post Semantic Layer vs Ontology: Which One Says What a Customer Is covers holding a concept that way.

Should you use text-to-SQL, a semantic layer, or both?

dbt's conclusion is "both, for different things" (dbt Developer Blog). The semantic layer takes anything that reaches a board, an auditor, or a KPI. Text-to-SQL takes exploration, with "as much schema context as possible". Sound advice for a metrics layer. For an agent the better arrangement is one layer, with the order fixed.

Three ways to put a model in front of a warehouse

PatternWhat the model getsWhat it can still get wrong
Raw text-to-SQLThe schemaThe concept, the join, and the formula
Through a metrics layerMetrics, dimensions, and declared joinsThe concept. A customer is a join key, so which records count is inferred
Through a concept modelWhat a customer is, the relationships it may traverse, then the metric logicOnly what the owner got wrong, and that has a name on it

In the third pattern the agent resolves what a term means before it composes a query. Then it generates, inside the join paths the model permits. The class of failure where a model invents a plausible join is unavailable, because the join was never offered. Our agent-ready semantic layer page calls this constrained query construction. It is the refusal rebuilt as a boundary the agent cannot see past. Our post A Semantic Layer for AI Agents: What the Agent Needs Before It Asks for Revenue lists what the agent needs, in order. Our post Semantic Layer and MCP: What the Agent Actually Receives shows what travels with each answer.

Where does TopQuadrant fit?

TQ Data Foundation is the layer that makes the refusal worth having. Every metric carries its business definition, its owner, its calculation, and its mapping to physical columns, on one governed asset. Definitions come before generation: the agent resolves what a term means before composing a query. Agents receive concepts, permitted relationships, and metric logic over API, SPARQL, and MCP. The join paths come from the ontology, so an invented join is unavailable. AI proposes the models and the mappings; your experts confirm them, so the scope widens only when a named person says so. Every definition is validated with SHACL, the W3C Shapes Constraint Language, which TopQuadrant helped author, and the standards are open. We maintain 300 reusable standards-based models. Your catalog stays, and it syncs. Each answer keeps a record: question, resolution path, versions, and result.

A refusal from a metrics layer tells you where the modelling stopped. A refusal from TQ Data Foundation tells you where the agreed meaning stops, and who to ask to extend it. The first is an error message. The second is governance.

Give your AI the context it's been missing

See how the TQ Data Foundation turns your enterprise knowledge into trusted, Al-ready context.