The sentence I hear most often when something has gone wrong with a production AI system is: "It was fine in March."
It probably was. That's the problem. Ordinary software fails in a way that sets off a pager — an exception, a timeout, a queue backing up. A system with a model in it degrades instead. The answers get a bit vaguer, the classifications drift a couple of points, the extraction starts missing one field on documents from one supplier. Every dashboard stays green, because every dashboard is measuring whether the thing responded, and it did. It responded beautifully. It was just increasingly wrong.
Then in June somebody escalates a complaint, you go and look, and you find the decline started in April.
Three ways it rots
Worth separating these, because they need different responses.
The inputs change. You built a support classifier on last year's tickets; since then the company launched two products, the help centre was rewritten, and a chunk of traffic moved to mobile where people type shorter and worse. Nothing about the model changed. The world it was fitted to did.
The right answer changes. This one's nastier because the inputs look identical. A fraud pattern evolves specifically to look normal. A policy is updated, so a document that was compliant in January isn't in May, and the model is confidently applying the old rule with the old confidence.
And — by a distance the most common in practice — somebody upstream changed something. A form got a new optional field. An API started returning nulls where it used to return empty strings. A supplier switched their invoice template. A PDF library was upgraded and now the tables extract in a different column order. None of this is an AI problem at all, but it presents as one, and teams lose weeks investigating the model when the culprit was a release note nobody read.
There's also the reflexive case, which I'd file under "know that it exists." If your model's outputs shape the data you later learn from — it ranks what people see, so people click what it ranked — then it's slowly training on its own opinions. That takes longer to hurt you and is much harder to unpick.
What to watch
Not accuracy, mostly, because you rarely have the ground truth in time. Watch the things that move before accuracy does.
Input distribution. Length, language, source system, document type, the share of requests hitting each category. You don't need anything statistically elegant here; a weekly comparison against a fixed reference window catches almost everything worth catching. When the shape of what's arriving changes, something changed upstream, and you want to know that in week one rather than month three.
Output distribution. If a classifier that has always put 12% of tickets in "billing" is putting 19% there this week, either your customers have had an unusual week or you have a problem. Both are worth a look.
Fallback and refusal rates. How often does it decline, hedge, return an empty result, or trip a guardrail? This is a superb early sensor because it moves first and it's trivial to log.
Human override rate. If there's a person in the loop — an agent who can edit the draft, a reviewer who can reject the extraction — the rate at which they change the machine's answer is the single most valuable number you have. It's free continuous evaluation performed by domain experts, and I'm consistently amazed how many teams collect it and never plot it. Plot it. Break it down by segment. It will tell you where the system is failing long before any aggregate metric does.
Cost per finished task. Not per request. When the model starts struggling, retries rise, escalations rise, and the bill drifts up before quality visibly drops. Cost is a quality signal in disguise.
Making the alarm real
Two things separate teams who catch this from teams who don't.
The first is that someone owns it. A named person or rota looks at the numbers weekly, and can stop a rollout. Not a dashboard that exists — dashboards that exist are wallpaper. Somebody whose job includes noticing.
The second is a small set of held-back cases that get run and graded properly on a schedule, monthly or so, by a human. A hundred examples, drawn from real recent traffic, marked by someone who knows the answer. It's a couple of hours of work and it's the only thing that gives you a true accuracy number rather than a proxy. Every team that's told me they don't have time for this has later spent considerably longer reconstructing when a regression began.
Then, when you do change something — new model version, new prompt, new retrieval — run it in shadow first. Both paths execute, only the old one is user-visible, and you compare on live traffic for a week. Practically every unpleasant surprise I've seen would have been caught by a week of shadow running, and shadow running costs money rather than reputation, which is the better currency to spend.
The uncomfortable bit
Most organisations budget for building an AI system and treat running it as overhead. It isn't. A system with a model in it needs the same standing attention as any other piece of infrastructure that touches customers, and slightly more, because its failures are polite.
The good news is that none of the above is hard. It's a handful of counters, one weekly meeting, a monthly grading session, and the discipline to shadow-run changes. The teams doing it aren't clever. They're just the ones who've already had the June conversation once and would rather not have it again.
