Evals over demos: our engineering stance
What does evals over demos mean in practice, and why does Eazyware insist on it?
We ship evaluation suites with every AI feature because a demo proves it can work once and evals prove it keeps working. Every prompt, model and retrieval change we make is run against a graded set of real cases before release, and the client keeps the suite, so the system can be changed safely long after we leave.
Evals over demos is the first line of our engineering principles, and it is the one that most changes how a project feels. A demo is a single successful run in front of an audience. An evaluation suite is a few hundred real cases with expected outcomes, scored automatically, on every change. We build the suite before we build the feature, and we treat a feature without one as unfinished. This article explains what that means day to day, what it costs, what it does not solve, and why we think it is the fairest deal we can offer a buyer.
Why a demo is not evidence
Language models are probabilistic. The same prompt on the same input can produce a different answer tomorrow, and a slightly different input can produce a very different answer today. A demo selects the input, the moment and the presenter. It proves the feature can work; it says nothing about how often, on whose data, or what happens after the provider ships a model update. We have watched teams buy on a demo, discover the failure modes in month three, and then have no way to tell whether a fix made things better or just different. That last problem is the one evals exist to solve.
The pattern is common enough that we wrote about it separately in AI proof of concept vs demo. The short version: a proof of concept has a pass mark agreed in advance; a demo has an audience.
What an evaluation suite contains
| Component | What it is | Who provides it |
|---|---|---|
| Golden set | Real inputs with expected outputs or grading rubrics, sampled from production or historical data | Client supplies data; we curate and label with the client's domain expert |
| Graders | Exact-match checks, schema checks, retrieval hit checks and model-graded rubrics for open-ended answers | We build; the client reviews grader agreement with human judgement |
| Slices | Subsets by intent, language, document type, customer segment or difficulty | Agreed jointly so a failing slice is never hidden by an average |
| Thresholds | Pass marks per slice that gate a release | Set with the client after the first baseline run |
| Runner | A script or CI job that runs the suite on every prompt, model, retrieval or code change | We build; it lives in the client's repository |
| Report | Per-slice scores, regressions against the last run, cost and latency per case | Generated automatically; read by both teams |
How eval driven AI changes the build
Week one is data, not prompts
Before any model is called, we sit with the person who does the job today and collect cases: the routine ones, the awkward ones and the ones that went wrong last quarter. For a document-extraction feature that is a few hundred documents with the correct fields already known. For a support agent it is real conversations with the outcome the team would have wanted. This work is unglamorous and it is the most valuable week of the project, because every later decision is measured against it.
Every change is a measured change
Once the suite exists, a prompt edit is a pull request with a score attached. Swapping a model is a run with a per-slice comparison. Changing chunk size in retrieval is a run. Nobody argues about whether the new version "feels better"; the report says which slices moved and in which direction. This is what lets a junior engineer safely touch a prompt a senior engineer wrote, and it is what lets the client's own team touch it after handover.
Shadow mode reads from the same suite
When the feature reaches production, it runs in shadow mode first: it produces outputs alongside the humans, nobody acts on them, and disagreements are reviewed. Cases the humans handled differently become new golden cases. The suite grows from real traffic rather than from our imagination, which is the point. We describe the mechanics in shadow mode, the right way to launch AI agents.
What evals do not do
We do not want to oversell the practice. Evals do not prove a system is safe against inputs nobody thought of; they catch regressions on the cases you have. They do not replace monitoring in production, where the input distribution drifts. Model-graded rubrics have their own error rate and have to be checked against human judgement periodically. And a suite that is never updated becomes a comfort blanket. Our standard is that the suite is a living asset, reviewed with the client monthly under a Care Plan, with new failures from production added and stale cases retired.
Why we hold this standard against ourselves
Our programs are fixed price and fixed date, described on the pricing page. A fixed price is only honest if there is an objective definition of done. The evaluation threshold is that definition. When a client asks whether the feature is finished, the answer is a number per slice against an agreed pass mark, not an opinion. It protects the client from a vendor declaring victory, and it protects us from an open-ended argument about quality. It also makes our own AI-assisted development safe: when a coding agent drafts a prompt change, the suite decides whether it ships.
A demo is an anecdote. An eval is a measurement. We charge for measurements.
A worked example
A lender asked us to extract fields from KYC documents that arrived as photographs, scans and PDFs of varying quality. The first week produced no extraction at all; it produced a labelled set of documents across every type, language and quality level the operations team actually saw, with slices for each. The baseline run on the first prompt looked acceptable on average and poor on one document type in one language, which the average had hidden. Two weeks of iteration were spent on that slice specifically. When a provider later released a new model version, the suite was rerun, one slice regressed, and the routing for that slice stayed on the older model until the newer one caught up. The operations team now runs the suite themselves. The anonymised outline is in the KYC document intelligence case study.
Team and timeline
Building the first suite takes one engineer and one client domain expert about a week inside any of our programs, and it is the first deliverable of a ProofRun AI POC Sprint, which is three weeks at $6,250 to $10,500 (from ₹4,00,000). In a Launch 6 MVP the suite is extended every week and the release gate is enforced from week two. After handover, monthly suite review sits inside a Care Plan from $1,000 a month. The client's domain expert is the critical role: without someone who can say what the right answer is, there is no golden set, and we will say so before we start. Our AI/ML development team owns graders and runners; the client owns the data and the pass marks.
Before you start: a checklist
- Name the domain expert who can label a few hundred cases and defend the labels
- Confirm you can export real historical inputs, anonymised where needed, for the golden set
- Decide which slices matter enough to have their own pass mark
- Agree that no release ships below threshold, and who can grant an exception
- Put the suite in your repository from day one, not in the vendor's
- Plan a monthly review to add production failures and retire stale cases
- Budget for human checks on model-graded rubrics every quarter
Questions clients ask
- Can we see evals before we sign? Yes. We will show the suite and report from a comparable anonymised build, and the first thing a Sprint Zero produces is the outline of yours.
- What if we have no historical data? We build a smaller seed set with your expert, launch in shadow mode sooner, and let production disagreements fill the gaps. The threshold is lower at first and rises as the set grows.
- Do evals cover cost and latency? Every run records tokens and time per case, so a change that improves accuracy while doubling cost is visible before it ships.
- Who decides the pass mark? You do, with our recommendation. It is usually set at or slightly above the measured human baseline for the same task.
Glossary
- Golden set: real inputs paired with expected outputs or a rubric, used to score a system
- Grader: code or a model that compares an output to the expected result and returns a score
- Slice: a subset of the golden set with its own score, so a weak area cannot hide in an average
- Regression: a case that passed on the previous run and fails on this one
- Shadow mode: running a system alongside humans without acting on its output, to collect disagreements
- Release gate: the rule that a change ships only if every slice meets its threshold
Related reading
Start with evals: the practice that separates AI demos from AI products and prompt versioning and evaluation, then read how the rest of our stance fits together on the about page. For a primary source on building graded test sets, OpenAI's evaluation guidance is a sound starting point.
If a vendor cannot show you the suite, you are buying a demo; ask for the numbers per slice and the script that produced them.
Frequently asked questions
How many cases does an evaluation suite need?
▾
Enough that every slice you care about has a meaningful count, usually a few hundred in total to start. It grows from production disagreements during shadow mode, so the first version does not need to be complete, only honest.
Who owns the evaluation suite after the project?
▾
You do. It lives in your repository with the runner and graders, alongside the code, prompts and documentation, as described in who owns the code, prompts and models.
Does an evaluation suite slow delivery down?
▾
It moves a week of effort to the front and removes far more than that from the end, because nobody spends weeks arguing about whether a change helped. Fixed dates depend on it.