Agents' Last Exam launches economically focused agent benchmark

ALE, led by Berkeley RDI in partnership with the RDI Foundation, Snorkel AI, and more than 300 industry experts, maps evaluation tasks to the U.S. O*NET/SOC occupational taxonomy across 55 professional sub-industries grouped into 13 clusters, with a catalog of more than 1,500 tasks (147 released publicly on GitHub) toward a longer-term goal of 5,000. Rather than abstract question-answering, agents complete real professional software workflows: 3D modeling in Siemens NX, motion and VFX work in Adobe After Effects, scene setup in Unreal Engine, mold-flow simulation in Moldex3D, architectural modeling in Rhino, and brain-imaging segmentation in FSLeyes, graded against verifiable, artifact-based outcomes rather than human judgment. Academic partners span MIT, Harvard, Stanford, and Oxford, alongside industry contributors including Goldman Sachs, JPMorgan, Morgan Stanley, Adobe, and Oracle, per the project's own site.

VentureBeat reports that OpenAI's GPT-5.5, run through the Codex harness, took the top spot on ALE's leaderboard with a 24 percent pass rate, edging out Claude Fable 5 in third place at 22 percent, an upset given that Fable 5 leads GPT-5.5 on benchmarks like SWE-Bench Pro and Humanity's Last Exam. That reversal is itself a finding: it shows that relative model rankings are not stable across benchmark design, and a model optimized for one class of tasks, such as terminal-based coding, does not automatically transfer its edge to long-horizon, GUI-heavy professional workflows.

Teams selecting agents for economically valuable deployments, not coding-assistant use cases alone, should treat generic leaderboard position as a weak proxy and instead test candidate models directly against the workflow category they plan to automate. ALE's open ale_run toolkit and reference harnesses (the official Claude Code CLI and an in-tree OpenClaw harness) offer a template for that kind of outcome-verified, sandboxed evaluation, and its rolling six-month task refresh is designed to limit the answer-leakage and benchmark-memorization problems documented in the companion audit.

Track whether additional frontier labs publish their own ALE runs, whether the public task pool grows toward the 5,000-task goal without leaking into training data, and whether ALE's deterministic graders hold up against the same categories of exploit, shared agent/evaluator environments, weak string matching, unsanitized LLM judges, that the RDI team used to break eight other benchmarks.