AI’s rapid gains in coding stall when tasks demand human judgment, METR finds
AI models are racing ahead in coding and math, doubling their performance every four months on tasks where success is easy to measure. Yet when METR, an AI evaluation nonprofit, tested those same models on open-ended work—like designing research or scoring philosophical arguments—they fell short. The gap exposes a critical limit: today’s AI excels at clear-cut problems but still lacks the judgment needed for complex, real-world decisions.
Editor, Lazyfounder

TABLE OF CONTENTS
KEY HIGHLIGHTS
- AI models double their performance on verifiable tasks like coding every four months, METR’s latest evaluation shows.
- Open-ended tasks, such as designing AI evaluations or scoring philosophical arguments, remain beyond current AI capabilities.
- METR’s new report includes the first open-ended tasks in its time-horizon plot, revealing only borderline progress.
- An earlier METR study suggested AI could speed up research at Anthropic by 2x, but the new data tempers expectations.
- Evaluating judgment-heavy tasks is costly and subjective, limiting how many METR can test.
- OpenAI’s models perform worse on real-world tasks assigned by employees than on METR’s verifiable benchmarks.
Coding and math keep getting faster—but open-ended work lags
METR’s updated time-horizon plot tracks how quickly AI models complete tasks humans can verify, such as writing code or solving math problems. The latest data confirms a steady trend: every four months, models double the length or complexity of tasks they can finish. MIT Technology Review reports the same models show little progress on work that requires judgment, like designing new AI evaluations or scoring philosophical arguments.
New tasks test AI’s limits—and reveal stubborn gaps
For the first time, METR included tasks that don’t have simple pass-or-fail answers. It asked models to design AI evaluations METR might use in its own research. None met the nonprofit’s standards. Opus 5.5, one of the newest models, scored just above the minimum bar—a step up from a year ago, when every model failed outright.
A second task required models to grade philosophical arguments. Here, too, they trailed human performance. Even METR’s own team disagrees on whether this kind of task matters for automating AI research. The debate reveals how hard it is to measure progress on work that lacks clear right answers.
Judgment calls are hard to score—and expensive to test
METR’s original suite scored task difficulty by timing how long a human would take. The new tasks often have no fixed completion time; some could take experts hundreds of hours. Paying people to complete them at scale would cost too much, so METR relied on estimates and single-expert scoring. For the philosophical argument task, just one person rated most responses. A broader panel might produce different results.
The constraints mean METR could include only a handful of open-ended tasks. Matt Putz, a member of METR’s technical staff, told MIT Technology Review that clean trends for such tasks remain elusive.
Anthropic study sparked fears—then METR tempered them
Earlier this month, METR published a heavily redacted report suggesting AI could cut the time needed for AI research at Anthropic in half. The claim raised alarms about AI systems designing their own successors without human input. METR’s new data pours cold water on that scenario—for now. The models still can’t match humans at open-ended research tasks.
OpenAI’s experience mirrors METR’s findings. In a blog post last week, the lab noted its models perform worse on the real-world tasks its employees assign than on METR’s verifiable benchmarks. The gap suggests that even fast progress in coding won’t directly transfer to judgment-heavy work.
Experts call the shift a necessary step
Sayash Kapoor, an incoming professor at UC Berkeley, praised METR for expanding its testing to include open-ended tasks. He told MIT Technology Review that AI still can’t conduct original computer-science research on its own. His own preprint study last month showed similar limitations.
Najoung Kim, a linguist and computer scientist at Boston University, agreed the changes were a step forward but cautioned against extrapolating too far. Because open-ended tasks are subjective, they don’t yet produce the clean, repeatable trends METR found for coding and math.
Key takeaways
- AI’s exponential gains are real—but only for tasks with clear success criteria like coding, math, or data analysis.
- Open-ended work, from research design to philosophical reasoning, still requires human judgment that models can’t replicate.
- METR’s new tests are a rare attempt to measure AI in ambiguous domains, but the results show progress is slower and harder to track.
- Startups building AI tools for verifiable tasks should expect rapid improvement; those targeting nuanced decision-making will need to plan for a longer runway and ongoing human oversight.
Related on Lazyfounder
Sources
- MIT Technology Review · 2026-11-20
The most misunderstood graph in AI just got a major update
This story is an original summary drafted with AI by Lazyfounder from the reporting listed above and checked by automated validation. Facts are attributed to their original publishers; sections marked as analysis are Lazyfounder's. Where a source is in another language, facts were machine-translated and quotations are reported, not reproduced. Read the original coverage via the links, and see our AI policy and corrections policy.
About the author
Editor, Lazyfounder
Tarun Mottlia edits LazyFounders, covering Indian startups, funding rounds, AI and product launches. Every story on the site is AI-assisted and checked against its cited sources before publication.
More stories by Tarun MottliaGet the LazyFounder Brief
Startup, funding and AI news in a five-minute read. Join the early-access list.


