Study Evaluates VE-LLMs for ECG Image Interpretation

An exploratory evaluation published in the Journal of Medical Internet Research examined whether generalist vision-enabled large language models (VE-LLMs) can interpret electrocardiograms presented as images. The team evaluated six widely used models, including versions of OpenAI's ChatGPT, Google's Gemini, Microsoft Copilot, and Anthropic's Claude, on a retrospective set of 70 deidentified 12-lead ECG images collected during routine care in a cardiology ward at University Medical Center Gottingen, Germany. Model inference was run in July and August 2025, and expert consensus served as the reference standard. The models were prompted to perform a structured, lead-by-lead visual analysis rather than rely on pattern-matching shortcuts.

Across diagnostic categories, the models showed moderate overall accuracy but mostly low sensitivity, meaning they frequently missed true findings even when their specificity was high. For several conditions, individual models recorded sensitivity at or near zero while still posting high accuracy, a pattern that can occur when a finding is uncommon and a model mostly predicts "normal." In detecting first-degree atrioventricular block, for instance, several models registered zero sensitivity despite high specificity. Agreement with expert ECG interpretation was limited overall, and results were inconsistent from one model and condition to the next.

Performance was uneven on practical measures too. Response times varied widely between models; according to the study, one ChatGPT version took a median of roughly 4.6 minutes to begin answering, while Gemini responded in a median of about 36 seconds. That spread, combined with variable accuracy, underscores how differently these general-purpose systems behave on the same task.

The authors conclude that current generalist VE-LLMs deliver performance that is inconsistent across models and diagnostic categories and is insufficient to support clinical deployment for ECG interpretation. The finding fits a broader pattern in medical AI: general-purpose multimodal models can describe images fluently but tend to underperform tools purpose-built and validated for a specific clinical task. A separate comparative study reported that a dedicated ECG AI substantially outperformed general LLMs at detecting myocardial infarction from ECG images. For now, the results argue for keeping specialized tools and expert over-read central to ECG diagnosis, and treating LLM outputs as exploratory rather than diagnostic.