I've sat through a lot of demos in the past two years and I've narrowed my contribution down to one question, usually asked about four minutes in.
"How do you know it works?"
Roughly half the time this ends the demo. Not because the answer is bad — because there isn't one. The team has been iterating for months on a system whose quality they assess by trying it, looking at the output, and going "yeah, that's better." Vibes. Very sophisticated vibes, held by clever people with good instincts, but vibes, and vibes don't survive a version bump.
Here's why it matters more than it used to. With conventional software, a regression announces itself. Something throws, a test goes red, a page 500s. With a language model in the loop, a regression is a slightly worse answer, and slightly worse answers look exactly like answers. You can degrade for six weeks and only find out when a customer complains about something that had been quietly broken since the prompt tweak on the 3rd.
Start with the failures you already have
The instinct is to build a big representative test set, and that instinct will keep you busy for a month and produce something bland. Don't start there.
Start with a folder. Every time the system does something wrong — a bad answer, a hallucinated figure, a refusal that shouldn't have happened, an escalation that should have been handled — the person who spotted it drops the input, the output, and one line about why it's wrong into that folder. That's it. No process, no ticket type, no ceremony, because ceremony is how this dies.
Within a fortnight you'll have thirty real failures. Those thirty are worth more than three hundred synthetic examples, because they're the actual distribution of ways your users break your system, which is never the distribution you'd have imagined. Everything grows from there.
Somewhere around a hundred and fifty examples you have a genuine evaluation set: mostly real failures, a decent number of cases it handles correctly (so you can catch the fix that breaks something else), and a handful of deliberately nasty edge cases. That set is now the most valuable artefact in the project. More valuable than the prompt, which you will rewrite forty times.
Get the ground truth argument out of the way
For each case, someone has to say what the right answer would have been. This is where projects discover that they don't agree.
Two of your experts will look at the same output and one will call it acceptable and the other won't. That disagreement is not an annoyance to be smoothed over; it's the specification of your product surfacing for the first time. Have the argument. Write down the resolution. If you can't resolve it, you've found a place where your product doesn't know what it's for, and no amount of model work will paper over that.
Where the output is open-ended, don't try to grade it out of ten. Nobody is consistent at that, including the same person on a different afternoon. Ask narrower binary questions: did it use only the supplied sources? Did it get the number right? Did it refuse when it should have? Four crisp yes/no checks tell you far more than one holistic score, and they tell you what broke rather than just that something did.
On using a model to grade a model
You'll want to automate this, and you can, within limits. Using a strong model as a judge works decently for constrained questions of the kind above. It works poorly for "is this good," which is unsurprising.
But calibrate it before you trust it. Take fifty cases you've graded by hand, run the judge over them, and see how often it agrees with you. If it's agreeing 90% of the time on your judgements, you have a useful instrument. If it's at 70%, you have a random number generator with a nice interface, and any improvement it reports is inside its own noise. I've seen teams celebrate a four-point gain from a judge whose agreement rate meant four points was indistinguishable from nothing.
Also, and this catches people: don't have the same model family grade its own output on style. It likes its own writing. So would you.
Make it a gate, not a report
The last step is the one that changes behaviour. The eval has to run automatically on every change to the prompt, the retrieval, the model version, the temperature, the chunking — every knob — and it has to block the change if the numbers drop. Not email a dashboard. Block.
The moment that's in place, a whole class of argument disappears from the team. "Does this prompt change help?" stops being a matter of opinion and seniority and becomes a number that took eight minutes to produce. That shift is worth more than the eval itself, honestly. It's the difference between a team that improves and a team that oscillates.
None of this is exotic. It's the testing discipline that ordinary software worked out decades ago, applied to a component that fails softly instead of loudly. The reason it gets skipped is that it isn't fun, it produces no demo, and it makes visible how often the system is wrong — which is uncomfortable in month two and priceless in month nine.
If you can't answer the question, you don't have a product yet. You have a very persuasive prototype, and those are much easier to build than they used to be.
