LLMs Expose Accessibility Testing Coverage Gap

Automated accessibility tools and LLM-assisted auditing have made measurable progress at catching surface-level issues, but they do not reproduce the lived experience of people who use assistive technology. The gap between a "passing" automated scan and actual usability with a screen reader is real, measurable, and costly.

Automated scanners such as axe-core, Lighthouse, WAVE, and SilkTide excel at deterministic checks: they reliably flag heading hierarchy violations, color contrast failures, and many missing alt attributes. LLMs raise the ceiling by helping parse noisy scan output, grouping issues, and suggesting fixes, but they suffer from hallucinations and context-misperception. Key classes of failures automated tools and LLMs miss or invent include:

• •Keyboard and focus-order problems, where tab sequence and programmatic focus diverge from visual order
• •Dynamic ARIA semantics and state changes that only matter during interactive flows
• •Screen reader reading order and label context that depend on surrounding content and user intent
• •Context-dependent labeling issues and ambiguous affordances that require task-based judgment

Treat automation and LLM assistance as triage, not validation. A pragmatic workflow looks like this:

• •Run automated scans to catch and prioritize deterministic defects
• •Use LLMs to summarize results, synthesize action lists, and generate targeted test cases
• •Perform targeted manual testing with assistive tech (NVDA, VoiceOver, TalkBack), keyboard-only navigation, and task-based scenarios
• •Include actual users with disabilities for high-risk flows and interactive components

The industry has long relied on rule-based scanners; LLMs promised contextual reasoning but currently amplify both detection and noise. That raises two organizational risks: misplaced confidence when leadership treats a passing report as proof of accessibility, and wasted engineering effort chasing false positives. The result is legal, product, and UX exposure despite more sophisticated tooling.

Vendors building better screen-reader simulation, richer runtime instrumentation, and hybrid LLM-human pipelines could narrow the gap. For now, embed manual assistive-technology testing into release gates and combine automated triage with human validation.