Text-to-SQL accuracy: what "95%" really means
What does 95% text-to-SQL accuracy really mean?
A text-to-SQL accuracy number only means something against a defined set of your users' questions, run on your schema, with correctness judged on the returned result and reported by question type. Here is how to measure it and what moves it.
Vendors quote text-to-SQL accuracy the way phone makers quote battery life. The number is only meaningful against a defined set of questions your users actually ask, run against your schema, with correctness judged on the result rather than the SQL text, and reported by question type so a strong average cannot hide a weak category. This article explains how we measure natural-language data querying, what a semantic layer is and why it does most of the work, how security is enforced, and which questions to ask any vendor who quotes a number.
Why the headline number is usually meaningless
Public benchmarks like Spider and BIRD measure generic schemas with generic questions. Your schema has a `customers` table with three definitions of "active", a `revenue` figure that means different things to finance and sales, and time periods that follow your fiscal calendar. A model that scores 90% on a benchmark can score 60% on your questions, and the failures cluster: joins across three tables, time windows, and any metric with a business definition. The BIRD benchmark authors themselves note the gap between benchmark and enterprise performance.
How we measure it: the golden set
Step 1: collect real questions
Gather one to three hundred questions from the people who ask them: sales, operations, finance, leadership. Not invented ones. Pull them from Slack requests, ticket queues and the data team's inbox. Each is a real information need with a real phrasing.
Step 2: verify answers
For each question, an analyst writes the correct query and records the correct result set. This is the expensive part and the reason the golden set is valuable: it is your ground truth.
Step 3: judge on results, not SQL
Two different queries can both be right. Correctness is judged on returned rows and aggregates, with tolerance rules for ordering and rounding, not on whether the generated SQL matches the analyst's text.
Step 4: report by category
| Question type | Why it fails differently | Typical fix |
|---|---|---|
| Simple lookups (one table, one filter) | Column naming, synonyms | Semantic layer vocabulary |
| Joins across tables | Wrong join path or fan-out | Canonical joins defined once |
| Time windows (last quarter, fiscal YTD, same period last year) | Calendar assumptions | Named periods in the semantic layer |
| Business metrics (churn, ARR, active user) | Definition ambiguity | Metric definitions with owners |
| Aggregations with filters and groupings | Precedence and null handling | Validated templates plus tests |
The semantic layer does most of the work
A semantic layer sits between the question and the database. It defines metrics ("active customer" means X), canonical joins (orders join customers on this key, never that one), named time periods (fiscal Q3 is these dates) and the vocabulary your teams use. Natural-language questions are mapped onto that layer, so the model composes from defined parts rather than guessing joins. This is where accuracy moves from 60% to 90%+, and it is why we build the layer with your analysts before touching the model. Tools like dbt's semantic layer formalise the idea; a lightweight custom layer works for smaller schemas.
Safety: read-only, validated, row-level secured
- Every query runs read-only through a role that cannot write, always.
- Generated SQL is validated before execution: allowed tables, no unbounded scans, cost guards, timeouts.
- Row- and column-level security are enforced at execution time by the requesting user's role, never by the prompt.
- The system shows the SQL it ran and cites the tables, so trust is earned by inspection.
Security is not a feature to add later; it is the reason a business can let non-analysts near the warehouse. The OWASP guidance on prompt injection is relevant here: user text must never be able to change what the query is allowed to touch.
What moves the number
In order of impact: the semantic layer; a golden set large enough to expose failure categories; validation before execution; hybrid retrieval of the right tables and definitions for each question; and a feedback loop where analysts correct answers and those corrections become new golden examples. Model choice matters less than people expect once the layer exists; we route between two models by question complexity to control cost.
MongoDB and other non-SQL stores
The same approach generates validated aggregation pipelines over a semantic model of your collections. The safety rules are identical: read-only, validated stages, role-scoped filters. See Natural Language Data Querying for the stores we support.
What to ask a vendor
- Which question set does your accuracy number come from, and was it measured on my schema?
- Do you report by question type?
- What happens to a query that fails validation?
- Can I see the SQL for any answer?
- How are row-level permissions enforced, and by whom?
- How do corrections feed back into the system?
A system that cannot show its work should not be trusted with your data. If you want the number for your own schema, a ProofRun builds the golden set and reports accuracy by category in three weeks.
A worked example: sales questions on a CRM warehouse
A B2B company's sales leaders asked questions like "which enterprise deals slipped from Q2 to Q3 and who owns them?" The first prototype, a model over the raw schema, answered 58% of a 140-question golden set correctly, failing mostly on joins between deals, accounts and owners and on fiscal quarter boundaries. Building a semantic layer with defined metrics ("slipped deal"), canonical joins and named fiscal periods raised results to the high eighties; validation and a feedback loop from the analyst team added the rest over a month. The system now shows the SQL for every answer in Slack, runs read-only under the requester's role, and the data team's ad-hoc queue dropped to the questions that genuinely need a person.
Implementation timeline
| Phase | Weeks | Work |
|---|---|---|
| Schema and metric audit | 1 | Interviews with analysts; list of metrics, joins and periods |
| Semantic layer | 2–3 | Definitions written and reviewed by metric owners |
| Golden set | 2–3 | 100–300 real questions with verified answers |
| Build and evaluate | 4–6 | Query generation, validation, RLS, evals to threshold by category |
| Rollout | 7–8 | Slack or in-product, one team at a time, feedback loop live |
What good looks like after launch
Business users get correct, governed answers with a chart in seconds. Accuracy is reported by category every release. Corrections from analysts become new golden examples. And nobody has write access through the system, ever. The full scope is on the Natural Language Data Querying page, related engineering on Data & Analytics Applications, and the starting price is on the pricing page; the LLM application practices behind it, evals, routing and observability, apply here too.
Team and timeline
The team is a data engineer who builds the semantic layer with your analysts, an AI engineer for query generation and evaluation, and an integration engineer for Slack, Teams or your product, over six to eight weeks. Your analysts' time in weeks one to three is the critical input: they define the metrics, verify the golden answers and, after launch, correct the misses that become new examples. Without that involvement the number will not reach the threshold, whichever vendor builds it.
Before you start: a checklist
- List the metrics your teams use and who owns each definition
- Grant read-only access to the warehouse or database
- Collect 100–300 real questions with verified answers
- Define roles and the row-level restrictions each carries
- Decide where answers appear: Slack, Teams, in-product
- Agree the accuracy threshold by question category
- Set up the analyst feedback loop before launch
Why we route between two models
Simple lookups and single-table filters are answered accurately by small, fast models once the semantic layer exists; multi-join and time-window questions benefit from a larger model's planning. Classifying the question first and routing accordingly keeps latency low for the common case and accuracy high for the hard case, and it makes cost per question predictable. The routing decision is itself evaluated on the golden set, so a change in model provider or pricing is a managed migration rather than a surprise. The same discipline, evaluation, routing, observability, applies to every LLM application we build.
Related reading
Related reading: Why basic RAG fails in production for the document-side equivalent, Why AI copilots inside SaaS beat chatbots for embedding reporting in a product, and How to choose an AI development company for evaluating accuracy claims.
The practical takeaway: do not buy a number, buy a method. A golden set from your questions, a semantic layer from your analysts, read-only execution under your roles, and accuracy reported by category on every release. With those in place the number takes care of itself, and it will be your number, measured on your data.
Frequently asked questions
Can text-to-SQL write to the database?
▾
It should never be allowed to. Production systems run read-only through a role that cannot write, with validation before execution.
Do we need a warehouse first?
▾
A warehouse helps but is not required; a well-modelled Postgres, MySQL or MongoDB with a semantic layer works.
How many golden questions are enough?
▾
One to three hundred real questions covering every category your teams ask; fewer than fifty hides failure types.