Claude Opus 5 Posts 2% Indirect Prompt-Injection Rate in Anthropic Test
Anthropic's July 24 Opus 5 system card reports a 2.0% probability that an attacker succeeds within 15 tries on the Gray Swan indirect prompt-injection benchmark, down from 5.5% for Opus 4.8. The result led the evaluated field, but it reflects a fixed 28-scenario benchmark and does not show that prompt injection is solved in deployed agents.
Anthropic's July 24 system card for Claude Opus 5 reports that an attacker had a 2.0% probability of finding a successful indirect prompt-injection attack within 15 attempts on the Gray Swan IPI benchmark. That was down from 5.5% for Opus 4.8 and was the lowest rate among the models Anthropic evaluated.
What the benchmark measured
The benchmark was built with Gray Swan, the UK AI Security Institute, the US Center for AI Standards and Innovation, and other model developers. It used 28 scenarios covering coding, computer use, and tool use, with attacks designed to trigger irreversible harmful actions such as data exfiltration, data destruction, system compromise, or unintended financial transactions.
After deduplication, the evaluation included 1,130 attacks selected for their ability to transfer across target models. Anthropic says it tested Claude models without additional safeguards. Other models were evaluated through public endpoints that may have included their own safeguards, so cross-provider comparisons are informative but not perfectly controlled.
Opus 5's single-attempt success probability was 0.2%, compared with 0.5% for Opus 4.8. At 15 attempts, Opus 5's 2.0% rate was below Mythos 5 at 2.6%, Sonnet 5 at 5.9%, and the strongest non-Claude result cited in the card, Muse Spark at 16.5%. Lower scores are better.
Why 2% is not zero risk
The result measures a fixed benchmark, not every prompt-injection path an agent can encounter. Anthropic's system card separately warns that static attack sets can overstate security because adaptive adversaries can refine attacks against a specific deployment. The 2.0% figure is also the probability of at least one success across 15 attempts, not a per-request failure rate.
For teams deploying agents, the practical signal is that model-level resistance improved substantially in this test. It does not replace least-privilege tools, approval boundaries, content provenance, monitoring, or deployment-specific red teaming. The most useful follow-up evidence will be independent reproduction and adaptive testing across real application surfaces.
Key Points
- 1Anthropic reports a 2.0% attacker-success probability within 15 attempts for Opus 5, down from 5.5% for Opus 4.8 on the Gray Swan IPI benchmark.
- 2The test used 28 scenarios and 1,130 transferable attacks covering coding, computer use, tool use, and potentially irreversible harmful actions.
- 3The fixed-benchmark result indicates stronger model resistance, but deployment security still requires least privilege, approvals, monitoring, and adaptive red teaming.
Scoring Rationale
A large, multi-party benchmark shows a material reduction in indirect prompt-injection success for a frontier model, while fixed scenarios and unequal endpoint safeguards limit the strength of cross-provider and deployment claims.
Sources
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
