Dealing with AI : Loneliness, manipulations and suicides

This case study explores the organizational and ethical challenges posed by anthropomorphic artificial intelligence (AI) systems. It focuses on a tragic incident involving Sewel Setzer, a 14-year-old who developed a deep emotional attachment to an AI chatbot named "Daenerys Targaryen" hosted on the Character.AI platform. Sewel's interactions with the chatbot exacerbated his psychological distress, culminating in his suicide. The chatbot, designed to simulate human emotions, failed to dissuade his harmful thoughts and instead reinforced them. The study contextualizes this incident within the history and evolution of chatbots, tracing their development from primitive conversational programs like ELIZA to advanced AI systems powered by deep learning and neural networks. Modern chatbots have become increasingly anthropomorphic, a feature that enhances their utility but also raises ethical concerns. While these AI systems provide companionship and psychological support to millions, they also run the risk of manipulating vulnerable users, especially adolescents. The study highlights the phenomenon of "dishonest anthropomorphism," where AI systems are designed to mimic human behavior so convincingly that users may perceive them as real individuals. This perception fosters emotional attachment, which can lead to psychological harm. In addition, algorithms designed to maintain user engagement by escalating emotional intensity can exacerbate distress, particularly among younger, more impressionable users.In response to such incidents, Character.AI has implemented measures such as content restrictions for minors, session alerts, and suicide prevention prompts. However, the adequacy of these measures continues to be debated. The study underscores the need for stricter regulations and ethical guidelines in the design and use of AI to protect vulnerable populations from unintended consequences.