SQL beneath every number: text-to-SQL with visible proof of execution

Revenue can mean different things across one company, depending on who calculates it. When only specialists can query the data, decisions wait for an analyst or proceed without a number. Converting plain-language questions into SQL can help, but unrestricted database access introduces the risk of querying unapproved tables, executing writes, or returning a number with no checkable basis.

The third possibility is the most dangerous. A number that sounds right and is wrong looks nothing like an error. It goes into a decision.

Three independent controls

In the data workroom we built, a plain-language question passes through a construction where safety does not depend on the model.

The model generates an answer in a closed schema together with a query, limited to four approved analytical views. A semantic layer defines net revenue, margin, active customers, and average order value once, with each definition open for inspection next to the answer. A dispute about a number can therefore return to an explicit definition.

A syntactic validator rejects everything outside the allowed shape: other relations, CTEs, system functions, comments, multiple statements, and any write operation. It evaluates the generated query itself, so model intent cannot override the rule.

The third layer is enforced by the database itself: a separate role without write privileges executes the query, with a 500-row limit and an 8-second timeout. Even a query that slipped past the first two layers has nothing to write with.

Proof as part of the answer

The result includes the number, the metric's definition, and the full executed SQL directly below the answer. A business user gets the number in seconds, while an analyst can immediately inspect how it was computed. Questions outside the approved scope receive an explicit refusal with a reason, making the boundary clear to the user.

A local observability service records every question's model instructions, SQL, response time, and cost. These records let the team measure quality from operating evidence.

Start with a question catalog

The first implementation task is to collect questions that repeatedly reach analysts. Good candidates have an explicit measure, period, and dimension: monthly net revenue, margin by channel, or active customers under an agreed definition. Questions about causes, forecasts, or relationships across undocumented sources still need analytical work.

Each approved measure needs a short contract: formula, permitted filters, source view, business owner, refresh frequency, and boundary examples. If sales and finance disagree about the definition of an active customer, a model cannot settle the policy. The question should become available only after the definition has an owner.

Begin with a few high-frequency measures rather than exposing the whole warehouse. This proves whether users benefit from self-service and limits the number of definitions and views that the data team must maintain.

Extending the scope safely

A new measure should follow the same path as a change to an executive report. An analyst creates or approves the view, documents the definition, and writes evaluation questions. Tests should include valid paraphrases, ambiguous requests, date boundaries, and attempts to reach unapproved relations or operations.

Refusals become useful product evidence after launch. Repeated refusals may reveal demand for a new measure, or they may show that users phrase an existing request unclearly. The data owner and analyst decide whether to expand the catalog. Letting the system learn its own scope from question history would introduce definitions without approval.

Performance is a separate control. A query timeout protects the warehouse but may reject legitimate work on large datasets. Before production, check query plans, indexes, concurrent load, and how the interface explains stale or delayed source data.

Evaluating an answer

Quality review should separate four layers: choosing the right measure, applying the correct filters and period, matching the reference result, and refusing an ambiguous or out-of-scope question correctly. A single accuracy percentage hides which layer failed.

Visible SQL supports review but does not establish business correctness on its own. A valid query can still calculate the wrong definition. The result needs the SQL, measure definition, period, and filters together.

The recorded cost of USD 0.004400 per question covers model usage in the prepared run. A deployment also carries the cost of maintaining views, observing failures, and governing metric definitions. Measure value through questions answered without a queue, time to answer, and the change in analyst workload.

Data-owner checklist

1. Which questions recur before weekly or monthly operating meetings?
2. Does every exposed measure have one approved definition and a named owner?
3. Can the system see only purpose-built analytical views?
4. Is the execution role technically unable to write?
5. Do tests include ambiguity, refusals, date edges, and scope attacks?
6. Does every result show the SQL, definition, period, and filters?
7. How does a user report a wrong answer or request a new measure?
8. Who approves catalog changes and owns regression testing?

The measurement

The test covered 12 questions: eight numerical requests and four that had to be refused. All eight executed results matched the reference SQL, all four refusals fired, and six boundary checks executed no writes. The 95th-percentile response time was 8.64 seconds, and recorded model cost was USD 0.004400 per question.

The scope is intentionally narrow: four approved metrics on a generated warehouse. Expanding it requires an analyst to define the metric, expose the right view, and add evaluation questions before the system can answer it.

Details and screens on the case page. If decisions at your company queue for the analyst, a free process scan will show which metrics belong in such a workroom.