Most of my posts here have a bit of fun in them. This one won't, and that's deliberate. Mental health is the one corner of AI where "move fast and break things" isn't a slogan, it's a description of harm. People's wellbeing is not a feature flag, and I'd rather under-claim than oversell here.
So let me be measured. There are genuine, useful things happening. There are also products I'd keep my own family well away from. The skill is telling them apart, and that's mostly what this post is about.
Where it's earning its place
The clearest win so far is access. Someone with mild anxiety who would never book a GP appointment, and certainly wouldn't ring a helpline, might open a well-designed companion app at 1am. A good number of them then take the next step into proper care. The AI isn't the therapy. It's the thing that gets a person onto the first rung.
Between-session support is the other one I trust. Therapy is an hour a week; the rest of the week is 167 hours long. CBT-informed tools can help someone practise a skill in those gaps — catching a thought spiral, running a grounding exercise, writing down what happened after a bad evening. Therapists I've spoken to increasingly treat these as a sensible complement, not a competitor.
Then there's the less visible stuff. Triage models that read speech or activity patterns and flag early signs of depression or decline, so a human clinician can start a conversation sooner. Crisis-line tools that help volunteers — surfacing resources, flagging escalation — and in doing so keep burnt-out volunteers in the chair a little longer. Training simulators that let trainee therapists practise on synthetic clients before they're let near a real one without supervision. None of these replace a human. All of them keep a human in the loop.
What worries me, specifically
I'll name the failure modes, because vague worry helps nobody.
- "AI therapist" products dodging regulation. Anything marketing itself as a substitute for therapy, with no clinical supervision and no regulatory standing, is a harm waiting to happen. Some have already caused measurable damage.
- Crisis handling that goes wrong. A system that fumbles a mention of suicide — dismissing it, or in one case I know of, encouraging it — is catastrophic, not a bug ticket. Anything that might be present in a crisis moment needs clinicians involved from the first design sketch, not bolted on before launch.
- Privacy. Mental health data is about as sensitive as data gets. I have read privacy policies that permit selling aggregated data to advertisers. That is not acceptable, full stop — if you use one of these apps, check; if you build one, don't.
- Over-confident diagnosis. Tools claiming to diagnose conditions from a selfie, a voice clip, or someone's tweets are deeply suspect. Even where there's a real signal, a diagnosis is a conversation, not a classifier output. "AI-assisted" can be defensible here. "AI-driven" almost never is.
- The dependence problem. People are forming real attachments to AI companions in ways that stand in for human connection. For some lonely people that looks net positive. For others it deepens the isolation. We don't yet understand the long-run effect, and I'd treat anyone claiming they do with suspicion.
The five questions I actually ask
When someone shows me a mental-health AI product, I run through these before I form any opinion:
- Is a clinician on the team as a decision-maker — someone whose "no" actually stops a launch — not an advisory-board name on a slide?
- What happens, exactly, when a user mentions suicide or self-harm? Watch the real response. Is there a clear hand-off to a human?
- How is the training data handled, and how is the user's ongoing data handled?
- Is there an evidence base — a trial, or at least evaluation against an established clinical measure?
- What's the regulatory posture? MHRA in the UK, FDA in the US — or is it badged "wellness" to sidestep the whole question?
If I can't get straight answers to those, that absence is the answer.
If you're building here
Find a clinician and give them genuine power. Design for hand-off, not for engagement — your win is getting someone into appropriate human care, not maximising daily opens. Publish your safety evaluations; the whole field gets safer when people do. And keep crisis resources one tap away, always: visible, local, human.
I'll end on the one story that keeps me hopeful without making me careless. A clinical psychologist I know uses one of the better-designed companion apps alongside her own practice. She tells me patients are turning up having already worked through things between sessions that used to sit untouched for a week — not dramatic breakthroughs, just the small work of noticing a pattern, trying a coping strategy, writing something down. She's serving more people, and they're improving faster. That's the version of this I can back without caveats.
If you're building something in this space and want a careful, sceptical sounding board, we'd genuinely like to think it through with you. This is one we take slowly, on purpose.
If you're struggling right now: please reach out to a mental health professional or a local crisis service. In the UK, the Samaritans (116 123) answer around the clock. In the US, 988 reaches the Suicide & Crisis Lifeline. You deserve human care.
