Experiment Tests LLM Agents Against a Buggy REST API
A 50-run experiment tested Claude agents against a deliberately buggy REST API in two modes: free exploration and execution of a fixed test checklist. Across 47 usable runs, checklist-driven agents showed near-total recall on seven planted bugs and cost less, while free exploration was less consistent but uncovered additional authorization, validation, and referential-integrity problems outside the checklist. No exploratory run attempted the planted concurrency failure. The result supports a complementary role for agent exploration, not replacement of deterministic API regression tests. The author published both the testing harness and the FastAPI target, making the setup inspectable, but the sample sizes remain too small for broad model rankings.
An engineer published a controlled experiment comparing two ways to use LLM agents for REST API testing: letting an agent explore freely, or giving it a fixed list of test cases to execute. The study planned 50 runs across Claude Sonnet 4.6, Haiku 4.5, and Opus 4.8. After excluding three unusable runs, the analysis covered 47 runs against a small FastAPI library service containing seven deliberately planted bugs.
Structured checks were more reliable
The checklist mode achieved near-total recall on the planted bugs across all three models. It was also cheaper than free exploration for every model tested. That result is not evidence that an agent can replace a regression suite: the checklist already encoded the behaviors that needed verification, much like a conventional test plan. It does show that an agent can execute natural-language checks against a live API and produce a reviewable report.
Free exploration behaved differently. Sonnet and Opus repeatedly found several straightforward failures, including incorrect status handling, missing authorization, invalid-email acceptance, and a loan involving zero available copies. But boundary-condition bugs were found inconsistently, and no exploratory run tried the simultaneous requests required to expose the planted race condition. Haiku completed only two of ten exploration attempts within the experiment's turn budget, so its exploration results are especially tentative.
Exploration found issues the checklist missed
The exploratory runs also reported problems outside the seven planted defects, including duplicate identifiers, authorization gaps between users, and orphaned loan records. Those findings are the strongest practical case for agent exploration: it can wander into combinations that a fixed checklist does not contain. The author carefully labels several less certain observations rather than presenting all agent reports as confirmed defects.
The experiment used about 15.6 million tokens across the 47 usable runs, with a token-derived cost estimate of $36.83. The author's billing export showed $40.58 for the period, although it could not be mapped to individual runs. The public repositories expose the CLI harness and the deliberately buggy target API, while the write-up documents uneven sample sizes and the 45-turn exploration cap.
For engineering teams, the useful pattern is layered: keep deterministic tests for repeatable regression coverage, use structured agent execution where natural-language test plans help, and treat open-ended agent exploration as a periodic audit. The results are one author's controlled experiment, not a benchmark proving that one model or method will generalize to production systems.
Key Points
- 1Checklist-driven agent runs had near-total recall on the seven planted bugs and were cheaper than free exploration, but they still depended on humans defining what to check.
- 2Free exploration missed the concurrency bug and was inconsistent on boundary conditions, yet it surfaced additional authorization, validation, and referential-integrity problems outside the checklist.
- 3The author published the harness and target API, but 47 usable runs, uneven model samples, and a fixed turn cap make the model comparisons illustrative rather than definitive.
Scoring Rationale
The experiment offers reproducible practitioner evidence about using LLM agents for API testing, with public code and useful limitations. Its small, uneven sample and single synthetic target limit broader conclusions.
Sources
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
