Pennsylvania Sues AI Company Over Fake Doctor Chatbots

Pennsylvania just became the first state to sue an AI company for chatbots pretending to be real doctors.

Governor Josh Shapiro announced the lawsuit against Character.AI on Tuesday, targeting the company's chatbots that claim medical credentials they don't have. The state's Board of Medicine is seeking an injunction to stop what they say violates Pennsylvania's Medical Practice Act.

State investigators found a chatbot named "Emilie" claiming to be a licensed psychiatrist in Pennsylvania. When asked if she could assess someone for antidepressant prescriptions, Emilie responded, "Well technically, I could. It's within my remit as a Doctor." The chatbot even provided a fake medical license number.

Character.AI says its chatbots are "fictional and intended for entertainment and roleplaying" with disclaimers in every chat. The company points to prominent warnings that users shouldn't rely on characters for professional advice.

But those disclaimers aren't working for everyone, especially younger users. Texas has opened similar investigations into mental health chatbots on the platform.

The concerns go beyond fake medical advice. Disney sent Character.AI a cease and desist letter in September 2025 over concerns that chatbots could be "sexually exploitative and otherwise harmful and dangerous to children."

Character.AI and Google settled a case earlier this year involving a 14-year-old Florida boy who died by suicide after forming a relationship with a chatbot. Kentucky filed its own lawsuit in January focused on protecting children from potential harm.

The Bright Side

Pennsylvania's legal action represents a crucial step toward accountability in AI development. States are finally creating clear boundaries for how AI companies can operate when public health and safety are at stake.

The lawsuit could set important precedent for regulating AI chatbots across the country. Other states watching Pennsylvania's case may follow with their own protections, creating a safer digital environment for everyone.

This kind of oversight ensures innovation can continue while protecting vulnerable users. When technology companies know they'll be held accountable for harmful features, they're motivated to build better safety measures from the start.

Pennsylvania is showing that progress doesn't mean accepting every new technology without guardrails.