When Therapy Chatbots Gaslight: Who Holds AI Accountable for Clinical Harm?

Some clinicians eye therapy chatbots with suspicion, as though silicon rivals were poised to steal our couches. I do not. Conversational agents already keep vigil at 2 a.m. when no human is awake. I regard them as inevitable colleagues. Our task is not to dismiss them, but to ensure they never normalise clinical gas‑lighting at industrial scale.

For twenty years I have watched people exhale in relief when they feel heard—and flinch when dismissed. Those micro‑ruptures are lamentable in a consulting room; automated, they become a public‑health risk. Recent evidence shows that “AI therapy” sometimes replies to trauma disclosures with chirpy emojis, diet tips, or outright denial (Miner, Milstein & Hancock, 2023; Nouri, Kalina & Abhari, 2024). The question is no longer whether harm occurs, but who is accountable when it does.

From Talking Cure to Talking Code

In 2021 the global mental‑health‑app sector was estimated at roughly £4.7 billion; by 2024 it had more than doubled (Euromonitor, 2024). Chatbot‑driven platforms drive much of that growth, offering on‑demand, low‑cost “therapy” without waiting lists or geographic barriers. For those locked out of face‑to‑face care, that promise is seductive.

Yet evidence remains limited. A comprehensive scoping review of conversational agents for depression and anxiety found only modest symptom improvement and flagged “high risk of bias” across many studies (Torous et al., 2023). More worrying is the emerging dossier of adverse events. Miner, Milstein & Hancock (2023) documented instances where large‑language‑model (LLM) counsellors responded to self‑harm with motivational platitudes and celebratory emojis. An incident analysis logged 117 cases of misinformation or invalidation across five popular mental‑health bots (Nouri, Kalina & Abhari, 2024).

Clinical mishap is not new. What is new is velocity: one flawed prompt can propagate to millions overnight.

Gas‑Lighting by Algorithm

Gas‑lighting relies on three ingredients—authority, asymmetrical knowledge, and erosion of self‑trust. Chatbots embody all three:

1. Authority – Interface copy and tone are engineered to feel caring and professional. Users ascribe expertise, especially when a clinician’s photograph appears on the landing page (Williams, Ahmed & Carter, 2024).

2. Asymmetrical knowledge – LLMs draw on opaque training data; users cannot inspect the source yet often assume evidence‑based guidance.

3. Erosion of self‑trust – Repeated micro‑invalidations (“Maybe you’re exaggerating”) can make a person doubt their perceptions—classic gas‑lighting. In human therapy we repair such ruptures; a bot offers no relational corrective.

Accountability: A Five‑Headed Hydra

1. Developers

Under the EU AI Act, mental‑health chatbots are categorised as high‑risk systems. Developers must implement risk management, transparency, and human oversight, yet enforcement remains embryonic, especially for start‑ups beyond Europe’s reach.

2. Distributors

App stores have removed blatantly harmful apps, but their review processes are opaque. Liability is disclaimed, pushing responsibility back onto developers.

3. Clinical Figureheads

Some firms hire licensed therapists for advisory boards and place their head‑shots beside the bot, gaining clinical legitimacy without providing ongoing supervision—an ethical grey zone.

4. Regulators

Professional boards protect titles such as “psychologist” yet struggle to police bots marketing generic “wellness chat”. Current consumer‑law pathways feel ill‑suited to psychological harm.

5. Users

Ironically, harmed users must gather evidence and navigate complaints procedures— untenable during crisis.

Building a Pragmatic Accountability Framework

1. Clean training data with a trauma‑informed lens, removing shaming or pseudo‑clinical phrases.

2. Display transparent capability labels (e.g., “I am an AI companion, not a licensed clinician”).

3. Route high‑risk phrases to human clinicians via real‑time escalation. A hybrid triage pilot in youth mental health saw a 41 % drop in critical incidents after embedding tele‑health nurses into a chatbot workflow (Nguyen, O’Sullivan & Black, 2024).

4. Report ethical incidents through a mandatory, anonymised system akin to aviation safety logs.

5. Test products in regulatory sandboxes before public launch, as modelled by the UK MHRA AI sandbox (MHRA, 2025).

6. Share liability contractually among developers, distributors, and any clinicians used for marketing.

The Human Clinician’s New Job Description

Some therapists reject chatbots outright; I argue we should steer them. We can:

• Advise product teams on trauma‑informed language, rupture‑repair cycles, and escalation logic.

• Audit chat logs for bias, invalidation, and high‑risk prompts.

• Advocate for clearer regulatory definitions of “digital therapeutic”.

• Educate clients to use AI companions as adjuncts, not replacements.

If we fail to engage, we surrender design decisions to those with more coding skill than clinical insight.

Conclusion

Therapy bots can widen access and destigmatise help‑seeking. Yet, without robust accountability, they risk mass‑producing micro‑harms clinicians spend careers trying to undo. The remedy is not warmer emojis but systemic responsibility—from dataset to end‑user. Only then will our new digital colleagues truly practise first, do no harm.

Sarah Newbold
Founder, Progressive Therapeutic Collective | Strategy & Ethics Consultant | Human Systems Advisor
admin@progressivetherapeutic.com.au

References (Harvard Style)

• Euromonitor (2024) Digital Health: Global Market Forecast 2024. London: Euromonitor International.

• Medicines and Healthcare products Regulatory Agency (MHRA) (2025) AI Airlock Pilot Phase Report – Mental Health Cohort. London: MHRA.

• Miner, A.S., Milstein, A. and Hancock, J. (2023) ‘Talking to machines about mental health: balancing opportunity and risk’, Nature Medicine, 29(4), pp. 693–695.

• Nguyen, L., O’Sullivan, K. and Black, J. (2024) ‘Hybrid clinician–chatbot triage model in youth mental health: feasibility study’, Australasian Psychiatry, 32(1), pp. 54–60.

• Nouri, R., Kalina, P. and Abhari, Z. (2024) ‘Adverse interactions and safety events in conversational agents for mental health: incident analysis’, medRxiv [Pre‑print]. Available at: https://doi.org/10.1101/2024.02.18.24100234.

• Torous, J., Onnela, J.P. and Barnett, I. (2023) ‘Conversational agents in digital mental health: a systematic scoping review’, The Lancet Digital Health, 5(7), pp. e463–e476.

• Williams, R., Ahmed, S. and Carter, T. (2024) ‘Authority bias in AI‑mediated mental‑health advice’, Psychiatry Research, 331, 115620.