AI Chatbots Deliver Problematic Health Advice Half the Time

A team of researchers published a systematic evaluation in BMJ Open showing that generative chatbots frequently give unsafe or misleading medical advice. The study posed 250 clinical and health questions across five domains to five widely used models and found roughly 50% of answers were flagged as problematic, with about 20% rated highly problematic.

The study tested `ChatGPT`, `Gemini`, `Grok`, `Meta AI`, and `DeepSeek` on 50 questions spanning cancer, vaccines, stem cells, nutrition, and athletic performance. Two domain experts independently rated every response for clinical risk, factual accuracy, and citation quality. Key quantitative findings: the median reference completeness score was 40%, only 2 responses were outright refused, and open-ended prompts produced 32% highly problematic answers versus 7% for closed prompts. Model-level performance clustered, with Grok flagged worst at 58% problematic responses, `ChatGPT` at 52%, and `Meta AI` at 50%.

• •The evaluation captured both factual errors and unsafe recommendations, including plausibly authoritative-sounding but unsupported claims.
• •Reference lists were often incomplete or fabricated, reducing the ability to triage or verify advice.
• •Domains with well-structured evidence, such as vaccines and cancer, produced fewer errors but still carried substantial risk.

The findings reinforce an emerging pattern: high fluency and persuasive language do not guarantee clinical accuracy. This study complements prior work that found LLMs can outperform clinicians on style or empathy metrics while still producing medically incorrect outputs. For deployed health features, the combination of confident hallucinations and weak citation fidelity raises regulatory, ethical, and clinical safety issues. The disparity between closed and open prompts highlights a usability risk: typical patient queries are open and ambiguous, increasing the chance of unsafe guidance.

If your product routes users to chat-based health help or uses LLMs for triage, assume substantial residual risk without explicit guardrails. Reference quality cannot be relied on for verification, and refusal behavior is rare. Design controls should include clear disclaimers, structured follow-up questions to narrow scope, retrieval-augmented verification pipelines, and human-in-the-loop thresholds for any recommendation with clinical consequence.

Expect follow-up studies that replicate across more models and languages, and regulatory attention focused on consumer-facing medical claims. Practitioners should track improvements in citation grounding, retrieval-augmented generation adoption, and whether vendors expose uncertainty estimates or calibrated refusal policies.

These chat interfaces are conversational and persuasive but not yet dependable clinical advisors. Treat them as information aids requiring verification, not as substitutes for professional medical judgment.