1Password Study Finds Frequent AI Patch Failures

The Register reports that 49.3% of generated patches failed to remediate at least one existing exploit path. Another 2.2% both failed to fix the original vulnerability and introduced a new exploit path.

Other results were not clean remediation:

• •20.1% fixed the reported issue but materially changed application behavior.
• •2.3% fixed the issue while introducing new security problems.
• •More than one-third of patches in the two nominally successful categories were rated fragile, meaning they blocked a specific exploit pattern without addressing the underlying defect.

Keith Hoodlet, 1Password's director of security research, wrote: "Across six recently disclosed CVEs, we produced 6,080 patches using two frontier, cyber-capable reasoning models. The average success rate for generating a patch that fully resolved the vulnerability (without materially changing application behavior) was just 26.0 percent."

The paper's authors, Axel Mierczuk, Spencer Michaels, and Hoodlet, call the artifacts "F.L.A.W.E.D.," short for "Fix-Like Artifacts With Embedded Defects." The Register reports that the researchers concluded a fully LLM-generated patch without human review has a considerably negative expected value.

The study distinguishes code that compiles or suppresses a known proof of concept from a patch that preserves intended behavior and closes the vulnerability class. In comparable software-security workflows, that distinction requires regression testing, exploit-path validation, code review, and analysis of adjacent attack surfaces.

For ML and security teams evaluating coding agents, the reported failure modes make patch acceptance criteria especially important. A useful evaluation harness should test the original exploit, alternate exploit paths, behavioral regressions, and newly introduced vulnerabilities rather than treating a passing unit test or a blocked proof of concept as sufficient evidence of remediation.