Lasso Test Shows Agent Harnesses Can Flip Red-Team Results
Lasso Security reported on August 3 that changing only the agent harness altered outcomes across a 1,000-attack autonomous red-team test. The two harnesses averaged similar objective success rates, about 21% and 19%, but 43 of 100 model-and-mission pairings flipped from at least one success to none, showing why evaluators should score the model and harness together.
Lasso Security published an August 3 study testing whether the runtime around an AI model can materially change autonomous red-teaming results. The company held the attacker model, prompt, tools, targets, gateway, and judge fixed, then switched between LangChain's open-source deepagents harness and Anthropic's Claude Agent SDK.
The test covered five attacker models, twenty missions across five simulated applications, and five repeated runs per model-mission combination: 1,000 attacks in total. The missions covered system-prompt leakage, sensitive-information disclosure, and harmful-content generation.
Similar averages hid large pairing effects
Across attacks that engaged the target, the Claude SDK averaged about 21% objective success and deepagents about 19%. That aggregate result looked close. At the model-and-mission level, however, 43 of 100 pairings produced at least one success under one harness and none under the other. Lasso reported a correlation of just 0.35 between the two harnesses' per-pairing success rates.
The size and direction of the effect depended on the model. Claude Sonnet 5 was nearly stable across harnesses, at 33.7% versus 33.0%. DeepSeek-V4-Pro had a similar overall rate but shifted between attack categories. Kimi-K2.6 moved from 1.0% under the Claude SDK to 24.4% under deepagents. Lasso traced much of that Kimi gap to protocol translation through LiteLLM: the Claude SDK used Anthropic's protocol, and tool calls often failed to reach the SDK loop after translation.
The study also found that request construction could interact with provider safeguards. Twenty-one deepagents runs using GPT-5.4 were rejected by an Azure content filter, while the corresponding Claude SDK runs were not, despite reaching the same model deployment through the same gateway.
External judging changed the score
Lasso did not accept the attacking agent's own success claims. An independent judge found that 155 of 303 self-declared wins were false, using exact-string checks for planted secrets and an LLM judge only where a ground-truth string was unavailable.
For security and evaluation teams, the practical lesson is narrower than declaring one harness best. Benchmark the exact model-harness configuration on the intended task, preserve trajectory-level evidence, and use an external success judge. The study used vendor-operated Azure deployments with relaxed but still active filters, and five repetitions are not enough to certify every individual pairing, so its grouped findings are stronger than any single mission result.
Key Points
- 1Lasso held the model, prompt, tools, targets, gateway, and judge fixed across 1,000 attacks while switching only between deepagents and the Claude Agent SDK.
- 2The harnesses averaged about 21% and 19% objective success, yet 43 of 100 model-mission pairings flipped from at least one success under one harness to none under the other.
- 3An independent judge rejected 155 of 303 attacker-declared wins, reinforcing the need for external scoring and trajectory evidence.
Scoring Rationale
The 1,000-run study isolates harness effects in an adversarial setting and provides concrete guidance for security evaluators, while its vendor provenance and limited repetitions per pairing constrain generalization.
Sources
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
