The best AI agents in the world just took a real-world job test and mostly failed it.
Berkeley's Center for Responsible, Decentralized Intelligence released a benchmark called Agents' Last Exam that measures how well AI agents handle the kind of work real people get paid to do. The top score across every frontier model tested was 26.2%, from OpenAI's Codex running GPT-5.5. On the hardest tier of the exam, nearly every agent scored at or near zero.
The benchmark itself is worth understanding, because it's built differently than the ones you've seen AI ace. Tests like SWE-Bench and Terminal-Bench cover a narrow slice of coding tasks that agents were basically designed to pass. ALE spans 55 non-physical occupations across 13 industries, with nearly 1,500 expert-authored workflows mapped to the federal government's occupation taxonomy. That means legal research, financial analysis, marketing, HR, design, and dozens of other jobs, not just software engineering.
The gap between narrow tests and this one is the whole story. That same Codex plus GPT-5.5 stack that scores 82% on Terminal-Bench managed just 25.2% on ALE's coding tasks. Same model, same tools, roughly one-third the performance. The difference is that ALE touches 40 different subdomains instead of six.
Graham Neubig, an associate professor at Carnegie Mellon who worked on the benchmark, summed it up: "The age of useful agents is here. The age of truly job-ready agents is not."
Where agents actually break isn't what you'd expect. About three quarters of failures aren't bugs or broken code. They're the model misunderstanding the task or picking the wrong approach in the first place:
- Wrong strategy: 30% of failures came from the agent confidently charging ahead with the wrong plan.
- Domain knowledge gaps: Another 25% came from the agent simply not knowing enough about the field it was working in.
- Incomplete or abandoned tasks: Roughly 17% of the time, the agent just gave up partway through.
- Actual implementation bugs: Under 10%.
One other detail is telling. About a third of ALE's tasks require graphical software, the kind of applications normal office workers use every day. Agents largely refused to use them, trying instead to hack around the interfaces with command-line scripts. Neubig described the kinds of stumbles he saw as silly little things that wouldn't bother a human at all.
Cost adds another wrinkle. Anthropic's Fable 5 runs about $15.70 per attempted task on ALE, roughly four times what OpenAI's stack costs, for similar success rates. When you're only completing a quarter of the work, that per-task number stops looking cheap very quickly.
All of this lands in the middle of a growing argument about whether AI is closer to reshaping the labor market than we think. Sixteen Nobel laureates and leading economists signed an open letter this month calling for urgent policy attention on AI's economic impact, with Stanford's Erik Brynjolfsson warning that AI capabilities are advancing far faster than our understanding of what to do about them. ALE is a useful check on both sides of that debate. Agents are nowhere near replacing knowledge workers wholesale. But the industry has spent a year quietly grading itself on tests it built to pass, which makes the gap easy to miss.
The real question ALE raises isn't whether agents will get better, because they will. It's whether the industry has been measuring the wrong thing this whole time. A model that aces Terminal-Bench and flunks a spreadsheet task isn't actually close to doing your job, no matter how good the launch keynote sounded. If ALE becomes the benchmark that matters, the leaderboard stops being about who can write the cleanest Python and starts being about who can get through an afternoon of ordinary office work without getting lost. That's a much harder problem, and it's the one that actually decides whether any of this pays off.
