Golden question set

A golden question set is the test suite for a Text-to-SQL or conversational analytics system. Each entry holds a question as a user would phrase it, the SQL or result the data team has confirmed as correct, and notes on acceptable variations. Sets typically start at fifty to a hundred questions drawn from real analyst requests, Slack threads and dashboard filters, weighted towards the questions people actually ask rather than clever edge cases.

The system is run against the set and scored: did the query execute, did it return the right numbers, did it use the approved metric definition? The score is tracked over time, so a change to the prompt, the model or the semantic layer that quietly breaks ten questions is caught before users see it.

It is not a demo script and not a one-off acceptance test. A demo shows the questions that work; a golden set includes the ones that used to fail, the ambiguous ones and the ones that must refuse. It is the analytics equivalent of the golden dataset used for LLM evals generally.

• Data lead: it turns "the AI got it wrong again" into a reproducible failing test your team can fix and keep fixed.
• CTO / Head of Engineering: it is the only honest basis for an accuracy claim, and the gate that lets you swap models or providers without fear.
• CFO: ask to see the set and the pass rate before approving a rollout; a vendor who cannot show one is asking you to trust a demo.
• Product manager: the set doubles as a specification of what the feature promises to answer, which keeps scope honest.

Golden question sets exist because analytics AI fails quietly. A query that runs and returns a plausible table looks like success even when it has double-counted a join. Without a fixed set of questions with known answers, nobody can say whether this week's version is better or worse than last week's, and every model upgrade is a gamble. The trade-off is maintenance: questions go stale as the business changes, and someone has to verify answers by hand. That effort is small next to the cost of a wrong number reaching a board or a regulator.

• Regression-testing a SaaS product's in-app analytics before each release so a prompt tweak does not break "MRR by plan".
• Certifying that an NBFC's natural-language MIS answers portfolio questions using the same definitions as the regulatory return.
• Benchmarking three candidate models on a retailer's own hundred questions before choosing one, rather than trusting public leaderboards.
• Checking that a hospital reporting assistant refuses patient-level questions while answering aggregate ones correctly.
• Proving to a logistics client that on-time and cost-per-drop questions pass before the tool is opened to hub managers.

Technique / practiceAn evaluation technique, the analytics-specific form of Eazyware's evals-over-demos stance. Every natural language data querying build ships with one, and accuracy is reported against it rather than asserted.

Eazyware service that covers it: Natural Language Data Querying. Starting prices are on the pricing page.