SETA Releases 4,567 Verifiable Environments for Training Terminal Agents
A 22-author team submitted the first arXiv version of SETA: Scaling Environments for Terminal Agents on July 12, 2026. The project releases 4,567 executable training environments, with 3,255 synthesized tasks and 1,312 evolved tasks, plus code and dataset artifacts. SETA-Synth converts grounded technical material into containerized tasks, while SETA-Evol adjusts context and difficulty using model performance. The authors report that a Qwen3-8B model trained on a selected 560-environment subset averaged 10.7 percent on Terminal-Bench 2.0 and the base model averaged 3.1 percent. The widely cited 12 percent result is the best run, not the average. This is an unreviewed v1 preprint with author-run evaluations, not independent replication.
A 22-author team submitted the first arXiv version of SETA: Scaling Environments for Terminal Agents on July 12, 2026. The release packages a framework, code, a trained model, and 4,567 executable terminal environments for reinforcement-learning research. The paper extends an earlier January project rather than introducing the SETA name for the first time.
What happened
SETA-Env contains 3,255 tasks created through SETA-Synth and 1,312 tasks created through SETA-Evol. The published tasks use a Harbor-style structure with instructions, a container definition, a reference solution, and executable tests. The authors draw grounded task seeds from sources such as accepted technical question-and-answer pairs, notebooks, and shell-command examples instead of asking a model to invent every task without context.
SETA-Synth generates the instruction, environment, solution, and tests, then runs two basic checks: an empty or no-op attempt should fail, and the reference solution should pass. SETA-Evol starts from an existing verified environment and changes its context or difficulty. The paper uses model performance to decide whether an environment should become easier, harder, or more diverse, aiming to keep training tasks near a useful learning boundary.
| Layer | What it checks | Why it matters |
|---|---|---|
| Grounded seed | The task begins from existing technical material | Reduces arbitrary synthetic prompts |
| No-op and oracle tests | A blank attempt fails and the reference solution passes | Establishes minimum executable validity |
| Trajectory review | Real agent failures are inspected for flawed instructions or tests | Detects assumptions shared by generators and verifiers |
| Performance-aware evolution | Difficulty changes with observed model success | Avoids pools dominated by trivial or impossible tasks |
Technical context
The paper reports that Qwen3-8B trained with GRPO on a selected 560-environment subset reached a repeated-run Terminal-Bench 2.0 mean of 10.7 percent, with a 1.3-point standard deviation. The base model averaged 3.1 percent with a 0.6-point standard deviation. The authors' 12 percent headline is the best reported trained-model run, so it should not be presented as the expected average.
Under the same CAMEL terminal harness, the authors report DeepSeek-V4-Flash pass-at-1 moving from 40 percent to 43 percent and pass-at-5 moving from 54 percent to 58 percent. These are author-reported, harness-specific results. The preprint does not provide independent replication, and it tests only two model families. The environments cover terminal interaction, not graphical or multimodal agent work.
For practitioners
Verification, not task generation, is the real scaling bottleneck for terminal-agent training. A generated solution can pass generated tests while both encode the same hidden assumption. The most reusable SETA pattern is therefore the extra inspection of real failure trajectories: when every capable agent fails, the problem may be the environment rather than the model.
Teams building agent evaluations can adapt that pattern without adopting the entire training stack. Version instructions separately from tests, require negative and oracle checks, inspect clusters of universal failures, report means and variation instead of only the best run, and sandbox terminal execution. The public Apache-2.0 repositories make the mechanics inspectable, but artifact availability does not validate every benchmark conclusion.
What to watch
The next useful evidence would be independent reproduction across more model families, comparisons at matched compute budgets, and tests showing whether performance transfers beyond the CAMEL harness. The reported absolute Qwen3-8B pass rate also remains low despite the relative improvement, which is a reminder that more training environments do not by themselves solve reliable terminal agency.
Key Points
- 1SETA packages 4,567 executable terminal environments with public code, dataset artifacts, reference solutions, containers, and automated verification tests.
- 2The reported 12 percent Qwen3-8B result is a best run; the repeated-run mean was 10.7 percent, according to the paper.
- 3The strongest practitioner lesson is to inspect universal agent failures for flawed instructions and tests before blaming the trained model.
Scoring Rationale
The open artifacts and verification design are directly useful to agent-training and evaluation teams, while the unreviewed preprint and narrow model evaluation limit confidence in broad performance claims.
Sources
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
