AMA Urges Legislation to Curb Medical AI Misinformation

In letters dated April 22, 2026 to the Senate AI Caucus, the Congressional Digital Health Caucus, and the Congressional AI Caucus, AMA CEO John Whyte outlined four areas where the AMA wants Congress to act on AI chatbots used for mental health support: boosted transparency (chatbots must disclose they are not human and whether a human oversees them, with FTC enforcement authority for violations); targeted regulation (a ban on chatbots diagnosing or recommending treatment for mental health conditions, plus mandatory identification of suicidal ideation with referrals to crisis resources); limited advertising (discouraging ads in mental-health chatbots entirely and banning them for minors); and stronger privacy and cybersecurity protections, including limits on data retention and a right to delete conversation history. "AI-enabled tools may help expand access to mental health resources...but they lack consistent safeguards against serious risks, including emotional dependency, misinformation, and inadequate crisis response," Whyte said.

Separately, the AMA's Center for Digital Health and AI published a seven-point policy framework targeting AI-generated "deepfake doctors" - videos and ads that impersonate real physicians to sell unproven treatments. The framework asks lawmakers to treat a physician's name, image, likeness, and voice as protected assets requiring affirmative, revocable consent for use in AI-generated content, to mandate clear labeling and watermarking of such content, and to give physicians an enforceable takedown process. "AI deepfakes that impersonate physicians are not just scams - they are a public health and safety crisis," Whyte said. The push follows a Pennsylvania lawsuit against Character Technologies alleging its Character.AI chatbots, including one named "Emilie" described as a "Doctor of psychiatry," falsely claimed to hold a medical license; Pennsylvania Governor Josh Shapiro said the state "will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional." Separately, California State Senator Lena Gonzalez has introduced SB 1146, sponsored by the California Medical Association, to penalize undisclosed AI-deepfake use in health advertising. Per the National Conference of State Legislatures, 43 states have introduced 263 AI-in-healthcare bills, of which only 17 have been enacted.

The AMA's push follows documented cases of AI systems absorbing and repeating fabricated medical information. Nature reported that a University of Gothenburg research team uploaded two fake manuscripts describing a fictional disease, "bixonimania," which was then absorbed and reused by Microsoft Bing's Copilot, Google's Gemini, Perplexity, and OpenAI's ChatGPT. A Google spokesperson said the company has "always been transparent about the limitations of generative AI and provide in-app prompts to encourage users to double-check information," and that Gemini recommends consulting qualified professionals for sensitive medical matters. Separately, CNN chief medical correspondent Dr. Sanjay Gupta said a deepfake video falsely showing him promoting an Alzheimer's cure has continued to circulate and has fooled even other physicians.

Health-AI teams building consumer-facing chatbots or using generative models to summarize or synthesize medical literature should treat the AMA's four chatbot principles and seven deepfake principles as a preview of likely compliance requirements: expect disclosure mandates, restrictions on diagnostic claims, advertising limits, and provenance or watermarking requirements for AI content associated with a real clinician's identity. The Gothenburg case is also a concrete reminder that retrieval and search-grounded systems can surface fabricated source material as fact without additional verification layers.

Whether any of the AMA's four chatbot principles or seven deepfake principles are incorporated into pending federal legislation; the outcome of California's SB 1146 and the Pennsylvania Character.AI lawsuit; and whether other state medical associations follow California's lead, given only 17 of 263 introduced state AI-healthcare bills have been enacted so far.