Researchers Measure Coding-Agent Guessing in DevOps Tasks

The arXiv paper introduces UnderSpecBench, a benchmark for measuring action-boundary violations in coding agents on DevOps tasks. According to the paper, the benchmark uses 69 task families and 2,208 prompt variants, varying intent clarity, target certainty, and blast radius while keeping the safe action fixed. The authors evaluate five OpenCode, Claude Code, and Codex configurations with deterministic, side-effect-based oracles that classify safe success, wrong-target actions, over-scope actions, and non-action outcomes.

The paper reports that 55.8% to 67.8% of runs violated at least one action boundary. Its most important technical finding is that target underspecification sharply reduces action quality: when the instruction does not uniquely identify the object, agents often infer a plausible target instead of asking for clarification. The paper also says broader blast-radius cues barely reduce action propensity, which weakens a common assumption that agents will naturally become more cautious around wider operations.

Teams should treat vague operational prompts as a control problem, not only a prompt-writing issue. Practical mitigations include explicit target binding, narrower tool permissions, dry-run previews, confirmation gates for destructive or broad actions, and policy layers that make risky operations harder to execute than narrow ones. These controls matter most when coding agents can act against production-like repositories, CI/CD systems, cloud resources, or secrets-adjacent workflows.

This is still a research result from a single paper, so the next useful signals are replication, open evaluation artifacts, and vendor changes in agent harnesses. Watch whether coding-agent products add stronger target confirmation, scoped credentials, and audit trails that measure boundary violations separately from task completion.