AI Chatbots Deliver Misleading Health Advice Half the Time

A coordinated set of studies led by teams at the University of Oxford, the University of Tuebingen, and collaborators published results showing consumer AI chatbots generate problematic medical advice about 50% of the time, with 20% of responses categorized as highly problematic. The researchers evaluated five popular systems, `ChatGPT`, `Gemini`, `Grok`, `Meta AI`, and `DeepSeek`, using 50 diverse health questions each for a total of 250 responses. Two clinical experts independently rated each reply; only 2 of 250 prompts were refused outright, and none of the systems consistently produced reliable reference lists.

The study combined controlled scenario testing and user simulations to mimic real-world interactions. Key methodological points practitioners should note:

• •The models were assessed on diagnostic identification, urgency triage, and recommended next steps, not on narrow knowledge tests.
• •Evaluations flagged three failure modes: factual errors, inappropriate confidence (hallucinated citations), and under- or over-triage of urgent conditions.
• •Performance varied by domain: structured fields like vaccines and oncology showed better outcomes, yet still produced problematic answers roughly 25% of the time.
• •In user-simulation arms, participants using LLMs did not make better clinical decisions than those using standard web searches or personal judgment, largely because users omitted critical symptom details.

The systems demonstrate two interacting weaknesses for medical use. First, model outputs are brittle to prompt and context; small wording changes produced opposite recommendations in acute scenarios. Second, models project high confidence and produce fabricated citations, which undermines user ability to identify unreliable advice. For example, two similar user reports of a subarachnoid hemorrhage yielded opposite triage advice, with one user advised to seek immediate emergency care and the other told to rest at home.

This work connects controlled benchmark performance, where LLMs sometimes match clinicians, with operational realism, where they underperform. The findings align with prior BMJ Open and Nature Medicine analyses that highlighted generative model limitations in safety-critical domains. For product teams, this is a reminder that aggregate accuracy metrics on standardized tests do not translate directly into safe, deployable medical assistants.

Practical implications

• •Clinical deployment demands layered safety: explicit refusal policies for high-risk queries, uncertainty calibration, and mandatory structured symptom collection before advice.
• •Auditing must include scenario-based user studies, not only token-level or benchmark evaluations.
• •Regulatory and compliance teams should treat consumer chat interfaces as potential medical devices when their outputs materially influence care decisions.

Expect vendors to respond with improved refusal behaviors, better citation veracity, and focused fine-tuning for triage tasks. Researchers and product teams should prioritize tools that 1) elicit missing clinical context, and 2) surface uncertainty and provenance. Policymakers will likely accelerate guidance for consumer-facing medical AI, and clinicians should assume that current general-purpose chatbots are unreliable for unattended medical decision-making.