AI Outperforms Doctors in Hospital Diagnostics

A Harvard-led study published April 30, 2026 in Science found that o1 preview (OpenAI's step-by-step reasoning model) outperformed physicians in a diagnostic obstacle course using written clinical cases and real-world emergency room data, according to Harvard Magazine and STAT News. The Atlantic's reporting on the study frames deployment pressure: the article's author, a practicing pathologist, reports receiving an email from medical-center administrators announcing an "AI-powered clinical reasoning tool" was now available to clinicians. Co-senior author Adam Rodman (Harvard Medical School, Beth Israel Deaconess) and co-senior author Arjun Manrai presented the findings at a press conference ahead of publication. The Atlantic quotes Rodman saying, "I get a little bit queasy about how some of these results might be used." Manrai told reporters the results "do not mean that AI replaces doctors, despite what some companies [selling AI-based healthcare] are likely to say," per Harvard Magazine. At least one generative-AI product has received FDA approval, according to The Atlantic.

The study tested o1 preview, OpenAI's first model capable of step-by-step reasoning, on 76 real emergency room cases from Beth Israel Deaconess Medical Center in Boston across three stages: initial triage, first physician contact, and admission, per Harvard Magazine. Two blinded physician reviewers evaluated assessments without knowing whether they came from the AI or expert attending physicians; the model matched or exceeded expert performance at each stage. The AI performed especially well at initial triage and on complex diagnostic challenges, including real cases published in The New England Journal of Medicine, per Harvard Magazine. ChatGPT-4 was used as a separate comparison baseline; the reasoning model (o1 preview) outperformed ChatGPT-4 on management reasoning tasks, per Harvard Magazine. All evaluations used text-based inputs only - the study did not assess imaging, EKG, or physiological signal interpretation.

STAT News notes Rodman's concern that experiments based on "simulated and historical cases" will be misconstrued as proof of AI safety and efficacy in treating real patients. Harvard Magazine reports that Rodman envisions two high-value use cases: passive triage assistance that scans electronic health records to flag potential diagnostic errors before they happen, and AI-assisted second opinions, building on an Elsevier 2025 study that found 20% of clinicians were already consulting large language models for second opinions. The study team is conducting parallel evaluations on images, EKGs, and other signal types, per Harvard Magazine, with rapidly improving results.

When controlled evaluations show reasoning-model assistants outperforming clinicians on written diagnostic tasks, institutional procurement and pilot deployments typically accelerate. Observed patterns in similar transitions indicate that robust prospective validation, governance, and EHR integration work often lag behind rollouts. For practitioners, that gap typically manifests as increased demands on data pipelines, integration effort, and new monitoring and alerting requirements.

Indicators an observer should follow include:

• •uptake by electronic-health-record vendors and major hospital systems;
• •number and scope of FDA clearances or approvals for clinical-use generative-AI tools;
• •results from prospective, randomized clinical trials testing patient-centered outcomes; and
• •reports of real-world safety incidents or clinician-reported harms or workflow disruptions.