I get called in when things have stalled. It's a decent chunk of what I do, and after enough of these I've stopped bothering with a diagnostic phase, because the answer is nearly always the same and I can usually confirm it in an afternoon.
The model is fine. The team is good. Nobody funded the six weeks of unglamorous data work at the start, because six weeks of data work has never once made a board paper look exciting, and now they're nine months in, spending money at a rate that requires a story, and the story has to be about the model because that's the part anyone can be persuaded to care about.
Let me be specific about what actually goes wrong, because "data quality" as a phrase is so worn out that it no longer transmits anything.
"We have loads of data"
I hear this in the first meeting almost every time, and I've learned to treat it as a warning rather than a reassurance.
Having loads of data means you have loads of rows. It doesn't tell you whether those rows describe the thing you want to predict, whether they were collected consistently, whether the process that generated them is the same process running today, or whether half the useful signal was thrown away by a form redesign in 2021 that nobody thought to mention.
A manufacturer once told me they had eight years of sensor readings. True. What they didn't have was eight years of comparable sensor readings, because the line had been re-instrumented twice and the units changed the second time. Roughly a third of the archive was measuring something subtly different from the rest, and no field in the schema recorded that. It took two engineers eleven days to work this out, mostly by finding a retired maintenance supervisor who remembered.
That's the shape of it. The problem is almost never volume. It's that the meaning of the data changed over time and nothing wrote it down.
Nobody agrees what the label means
This is the one that quietly kills classification projects.
You want a model to flag "high-risk" claims, or "urgent" tickets, or "defective" parts. So you get historical examples labelled by the people who do the job. Fine. Then you take a hundred of those examples, hand them to three experienced people independently, and ask them to label them again from scratch.
Do this. Please do this before you build anything. In my experience, on genuinely hard categories, three experts agree on maybe 60 to 75% of cases, and the disagreements are not random — they're structural. One person thinks "urgent" means the customer is angry, another thinks it means there's a contractual clock running. Both have been labelling tickets for years, and the training set is a blend of two incompatible definitions.
No model can resolve that. It'll learn the blend, produce mush, and you'll blame the architecture. The fix is a couple of days in a room with the people who know, arguing until the definition is written down in a way that survives contact with an edge case. This is the least technical work in the whole project and it has the highest leverage of anything you'll do.
The plumbing nobody owns
Then there's access, which stalls more pilots than any modelling problem I've seen.
The data lives in four systems. One of them is a supplier's, and the contract doesn't clearly permit this use. One of them is exportable only through a report designed for humans, which means a CSV where the header row is a merged cell and the totals are inline. One of them has a nightly job that everybody assumes is fine because it hasn't alerted, and which has been silently dropping records with non-ASCII names since the last upgrade. And the person who understands the fourth one left in March.
None of this is interesting and all of it is real. When a project timeline says "week 1–2: data acquisition," what it actually means is "weeks 1–9, contingent on a legal review nobody has requested yet."
What I'd do instead
Spend the first six weeks on the data and say so out loud, in the plan, with the money attached, before anyone gets excited. Frame it as what it is: the part that determines whether the rest works.
In that time, do four things. Pull a real sample and look at it with your own eyes — actual rows, on a screen, not summary statistics, because summary statistics hide exactly the corruption you're hunting for. Reconstruct the provenance of each field: who enters it, when, under what pressure, and what changed. Run the labelling agreement exercise. And build the smallest end-to-end pipeline you can, moving real data from source to output with nothing clever in the middle, so the access problems surface in week two instead of month five.
If the data doesn't survive that, you've learned it for the price of six weeks rather than nine months, and you can go and fix the collection process, which was probably the real project anyway.
There's a version of this article that ends on something inspiring. I don't have one. The work is dull, it's most of the job, and the teams who do it ship while the teams who skip it hold a lot of steering group meetings about model selection.
