Move fast and break people? Ethics, companion apps, and the case of Character.ai

Riffing off move fast and break things — the internal motto coined by Meta’s Mark Zuckerberg — this paper examines the ethical dimensions of human relationships with AI companions, focusing on Character.ai, a platform where users interact with AI-generated characters ranging from fictional figures to representations of real people. Drawing on an assessment of the platform’s design, and the first civil lawsuit brought against Character.ai in the USA in 2024 following the suicide of a teenage user, the paper identifies unresolved ethical issues including risks from difficulty in separating AI-based roleplay from real life, unconstrained AI models performing edgy characters, reality detachment, and confusion by dishonest anthropomorphism and emulated empathy. All have implications for safety measures for vulnerable users. While acknowledging the potential benefits of AI companions, the paper argues for the urgent need for ethical frameworks that balance innovation with user safety, and proposes actionable recommendations for design and governance.

AI & Society, 40, 6365–6377