How to measure whether text to SQL solution is working
How do you measure text to SQL solution?
Measure a text to SQL solution on three layers: execution accuracy against a golden question set, trust signals such as refusal rate and repeat usage, and the business outcome the system was funded for. Report one headline number, verified answer rate, with the other layers underneath it.
Measure a text to SQL solution on three layers. Correctness is execution accuracy against a golden question set with known answers. Trust is refusal rate, repeat usage and how often people check the generated SQL. Outcome is the decision or report the system was funded to speed up. Report one headline number and keep the layers beneath it.
This article defines each metric precisely enough to instrument, gives realistic targets, describes the rig that produces the numbers automatically, and names the four measurements that look impressive and tell you nothing.
The headline number: verified answer rate
Verified answer rate is the share of questions in your golden set where the system produced a query whose result matches the known-correct result, within the permissions of the user asking. It is a single number with a stated denominator, and it is the only figure worth putting in a steering pack.
It works because it is honest about everything underneath. A system that hallucinates a join fails it. A system that ignores row-level security and returns rows the user may not see fails it. A system that refuses a question it should have answered fails it too, which keeps the number from being gamed by making the system timid.
Public benchmarks measure the same quantity in a research setting. The Spider benchmark evaluates text-to-SQL systems on databases they have never seen, which is a harder test of generalisation than most internal deployments face, and its results are a useful reminder that clean schemas flatter a model. Your number will differ from any published one because your schema, your metrics and your users are your own.
The metrics that matter, and their targets
| Metric | What it tells you | How to measure | Realistic target |
|---|---|---|---|
| Verified answer rate | Whether answers are correct | Golden set of 150 to 300 questions, result comparison | 85 to 95 per cent |
| Refusal rate | Whether it declines instead of guessing | Share of questions returning a clear refusal or clarification | 5 to 15 per cent |
| Unsafe query rate | Whether validation is working | Queries blocked for writes, missing filters or cost | Zero reaching execution |
| Correction rate | Whether analysts trust the SQL | Share of answers where a user edits the query | Falling month on month |
| Repeat usage | Whether it replaced an old habit | Users asking in week four who also asked in week one | Above 50 per cent |
| Time to answer | Whether it beats the queue | Question to result, compared with the analyst backlog | Under 30 seconds |
| Cost per answered question | Whether it scales | Model spend plus warehouse compute, divided by answers | Stable or falling |
| Analyst hours returned | Whether it paid for itself | Ad-hoc request tickets before and after | Measured quarterly |
Layer one: correctness you can audit
Build the golden set before the system
A golden question set is a list of real business questions, each with the correct answer agreed by the person who owns that number. It should be written by your analysts and finance team, not the vendor, and it should include the awkward ones: questions with ambiguous time ranges, questions that span two domains, questions a junior user is not entitled to ask. Two hundred questions is enough to be meaningful. The golden question set entry explains the format.
Compare results, not query text
Two correct SQL queries can look nothing alike, so string comparison undercounts. Compare the result set: the rows, the numbers, the ordering where ordering matters. Where a question has a legitimately ambiguous answer, record both acceptable results rather than picking one and calling the other a failure.
Grade refusals separately
Split outcomes four ways: correct, wrong, refused-correctly and refused-wrongly. A system that answers 80 per cent correctly, refuses 15 per cent it genuinely could not resolve and gets 5 per cent wrong is far more useful than one that answers everything at 88 per cent, because the second one hands you five wrong numbers with no warning. The arithmetic behind headline percentages is unpicked in what 95 per cent accuracy really means.
Layer two: whether people trust it
Correctness is necessary and not sufficient. A system can be right and ignored. The signal to watch is repeat usage by individual, not total query volume, because volume rises during the novelty period and then collapses if trust never formed.
Correction rate is the most informative trust metric. Show the generated SQL next to every answer and count how often an analyst edits it before using the result. A high rate early is healthy, because it means people are reading; a rate that does not fall by month three means the semantic layer is wrong rather than the model.
Track the questions people ask that the system cannot answer, and treat that list as a product backlog rather than a defect log. Most of those questions point at a missing metric definition, and each one you close widens the system's reach permanently. The route from repeated questions to a saved view is covered in natural-language reporting.
Layer three: the outcome it was funded for
Every deployment has a business reason, and it is almost never "accuracy". It is usually one of three: analysts spending less time on ad-hoc requests, decisions being made in the meeting rather than a week later, or a customer-facing feature that improves retention. Pick yours before launch and instrument it, because it will be the question at the first budget review.
The measurable proxy for the first is the ad-hoc request queue: count tickets and median age for a month before launch, then repeat at ninety days. For the second, ask the meeting owner. For the third, use your existing product metrics. The business case framing is in the ROI of text to SQL solution.
The rig that produces these numbers
- A version-controlled golden set, stored beside the semantic layer so a metric change and its test move together.
- An automated harness that runs the whole set against a test warehouse and writes results to a table.
- Release gates: no deploy if verified answer rate falls more than two points, no deploy if unsafe query rate is above zero.
- Triggers beyond releases: run on every model version change, schema migration and semantic layer edit, plus monthly.
- Per-role runs, executing a subset as a restricted user to prove permissions still hold.
- Tracing on every production question, capturing the query, rows returned, latency and token cost.
- A weekly review of refusals and corrections with a named owner who can update definitions.
- A quarterly outcome report comparing the funded business metric against its pre-launch baseline.
The tracing part matters more than teams expect: without per-question cost and latency you cannot tell a quality problem from a capacity problem. LLM observability covers the instrumentation, and evals as a practice covers the discipline around it.
Set the reporting cadence alongside the rig. Correctness belongs in a weekly engineering review, trust in a monthly product review, and outcome in a quarterly business review. Reporting all three at the same meeting is how a good system gets judged on the wrong layer, usually by an executive who hears 88 per cent and assumes twelve wrong numbers went out.
Four metrics that mislead
Total questions asked rewards novelty and tells you nothing about value. User satisfaction scores on an analytics tool measure the interface, not the numbers. SQL validity rate, meaning the query parsed and ran, is close to useless, because a syntactically perfect query against the wrong table runs beautifully. And a vendor's benchmark score on a public dataset says nothing about your schema, however impressive it looks in a deck.
The subtler trap is measuring only the questions the system answered. If half the traffic is refusals and you report accuracy on the rest, your number is a coverage statistic wearing a quality badge.
When measurement is not your problem
If the system has fewer than twenty users and one data domain, a formal rig is overhead. Run twenty questions by hand after every change and spend the saved effort on the semantic layer instead. Measurement earns its cost when several teams depend on the answers and nobody can personally check them all.
Measurement is also the wrong focus when the underlying numbers are disputed. If finance and sales cannot agree what a qualified lead is, no evaluation suite will resolve it, and a high verified answer rate against a definition half the business rejects is a false comfort. Settle the definition first, then measure.
What we instrument on an engagement
Every natural language data querying build, from $12,500 or ₹8 lakh to $38,500 or ₹25.6 lakh, includes the golden set, the harness and the release gates as deliverables rather than extras, because a system with no way to prove it works is not finished. A ten-day Sprint Zero at $3,250 or ₹2 lakh produces the first version of the question set before any code is written.
After launch, the $750 or ₹40,000 AI add-on to a Care Plan covers eval runs, prompt regression and cost monitoring on an ongoing basis, on top of plans that start at $1,000 or ₹68,000 a month. All figures are on the pricing page, and the safety controls the evaluation checks are described in building a safe natural-language analyst.
Related reading
The semantic layer post explains why most measurement failures are definition failures, and Eazy Insights AI shows what governed answers look like as a product.
A text to SQL solution is working when the people who used to queue for numbers stop queueing, and you can prove the numbers were right.
Frequently asked questions
What is the single best metric for a text to SQL solution?
▾
Verified answer rate: the share of questions in a golden set where the generated query returns the known-correct result under the asker's own permissions. It captures hallucinated joins, permission leaks and unhelpful refusals in one number, and it is meaningful only when reported with its denominator and question mix.
What is a realistic accuracy target?
▾
Between 85 and 95 per cent verified answer rate on a golden set of 150 to 300 real business questions, with a refusal rate of 5 to 15 per cent and no unsafe queries reaching execution. Higher published figures usually reflect simpler questions or a cleaner schema than a production warehouse has.
How often should the evaluation suite be run?
▾
On every release, every model version change, every schema migration and every semantic layer edit, plus a scheduled monthly run. Model providers deprecate versions on their own timetable, so a purely annual review will miss changes. Gate deployment on the result rather than reviewing it afterwards.