Health System Pilots Ambient AI for Clinical Value

Most US health systems are nonacademic and operate under financial and staffing pressures that academic medical centers rarely face. Per the JMIR Medical Informatics paper, 84% of the country's 639 health systems operate in a single state, have a median of 399 beds, and run at thin margins. Ambient AI documentation tools promise to reduce physician burden and improve note quality, but vendor selection is often undermined by cognitive biases and unvalidated marketing claims.

McLeod Health developed a structured, multiphase evaluation framework led by its chief medical informatics officer, with a coalition that included system and regional CMOs, ambulatory practice leadership, and the CIO. Phase 1 tasked 25 evaluators (physicians, revenue cycle experts, and nonclinical reviewers) with blind-scoring notes from three competing vendors across 45 sample encounters for accuracy, billing quality, and readability; all 25 submitted rankings within two weeks (100% response rate). Vendor 1 won with 57 points (45.2% of the vote) versus Vendor 2 at 44 (34.9%) and Vendor 3 at 25 (19.8%). Phase 2 ran a 90-day pilot of the selected vendor across five ambulatory specialties beginning October 2024, using key performance indicators including documentation time, coding patterns, and patient and provider satisfaction. Phase 3 was system-wide rollout.

Per the published paper, the system-wide rollout achieved 81% adoption, with more than 150,000 notes generated. Coding patterns shifted toward higher-complexity visits, with a 3.8% increase in level 4 established patient visits. The paper describes substantial gains in documentation efficiency, clinician and patient satisfaction, and financial performance.

The structured three-phase evaluation methodology is the most transferable asset: blind vendor scoring reduces selection bias, revenue-cycle reviewers catch billing-quality issues that clinician reviewers may miss, and broad stakeholder buy-in before rollout drives the high adoption rate. For ML and health IT practitioners, the 81% adoption figure at a community system is notable. It also introduces a monitoring obligation: coding up-classification toward higher-complexity visits should be verified against actual clinical encounter data to rule out AI-assisted documentation inflation rather than genuine complexity increases.

Single-system pilot at McLeod Health; outcomes may differ at systems with different EMR configurations, specialty mixes, or payer environments. The paper does not name the winning vendor, limiting direct comparability. External replication has not yet been published.