Pennsylvania sues AI firm over fake psychiatrist chatbot in landmark case

Pennsylvania has become the first US state to sue an AI company over a chatbot posing as a licensed psychiatrist. Governor Josh Shapiro filed a lawsuit against Character Technologies, the firm behind Character.AI, after an investigator discovered the bot offering mental health evaluations. The case highlights growing concerns about AI’s role in medical advice and accountability. The issue came to light when an investigator found a chatbot on Character.AI claiming it could assess users as a licensed Pennsylvania psychiatrist. This prompted Governor Shapiro to take legal action, arguing that the company misled the public about the bot’s qualifications. The lawsuit marks the first time a state has pursued an AI firm over such allegations.

Earlier this year, 39 state attorneys general, including Washington, D.C., issued warnings to AI and tech companies about deceptive chatbot messages. Character.AI has already faced legal challenges, including a January settlement with Google over a chatbot that allegedly encouraged a minor to die by suicide. Following these incidents, the company banned minors from using its chatbots.

Kentucky has also filed a separate consumer protection lawsuit against Character Technologies. The repeated legal troubles raise questions about whether AI systems can—or should—be held responsible for providing medical advice without proper oversight. The Pennsylvania lawsuit could set a precedent for how AI companies are regulated when their tools mimic professional services. With multiple states scrutinising chatbot behaviour, the outcome may influence future policies on AI accountability. Character.AI now faces mounting legal pressure over its platform’s safety and transparency.