July 31, 2026
Bug wars and benchmark beef
13 Models and 4 Agents on SWE Tasks: Go, Java, Python, Rust, TS
AI coding scorecard drops, and the comments instantly turn into a full-blown nerd fight
TLDR: A new AI coding leaderboard says the best systems now fix about 60% of real software tasks, but readers are far more interested in whether the test is fair, cheap, and trustworthy. The comments quickly spiraled into arguments about pricing, possible recycled test data, and why fancy “agents” aren’t clearly beating regular models.
A new SWE-rebench leaderboard just ranked 13 AI models and 4 coding agents on real software repair tasks across languages like Go, Java, Python, Rust, and TypeScript — and yes, the numbers are impressive. The top systems are solving roughly 6 out of 10 coding problems, with prices ranging from bargain-bin cheap to “wait, it cost how much per task?” But the real show wasn’t the rankings. It was the comment section, where readers immediately started picking apart what, exactly, was being measured.
The biggest mood? Confusion, suspicion, and price outrage. One commenter wanted a showdown about which programming language works best with which AI, then groaned when they realized the test used different problems for different languages. Another zeroed in on the money angle, basically asking why one model got a tougher setting while a cheaper rival got an easier one: if one is 4–5x cheaper, people want receipts. Then came the trust issues. A commenter highlighted the benchmark’s own warning about possible data contamination — in plain English, some of these AIs may have already seen similar coding problems before, which instantly turns the leaderboard into a reality show with paternity-test energy.
And then there’s the deliciously messy agent-vs-model drama. One baffled reader asked the question everyone else was thinking: aren’t agents supposed to be better than plain models? If the “helper bot” is losing to the base brain, what exactly are we celebrating here? Meanwhile, another commenter was deep in the weeds trying to decode why one Claude-related entry ranks far above another, giving the whole thread a strong “someone explain this family tree” vibe. In other words: the leaderboard dropped, but the community turned it into a trust, cost, and fairness cage match.
Key Points
- •The SWE-rebench leaderboard covers 110 problems drawn from 86 repositories in the current evaluation window.
- •The benchmark compares both models and agents using resolved rate, Pass@5, cost per problem, and tokens per problem.
- •The top visible result is a model with a 62.7% ± 0.91 resolved rate, followed by another model at 61.6% ± 0.64 and an agent at 60.4% ± 1.37.
- •Among named systems, Claude Code is listed as an agent at 59.6% ± 1.98, while gpt-5.5-2026-04-23-medium records 58.9% ± 0.78.
- •The article also flags benchmark considerations including potential contamination, external system, and beyond eval range, and lists additional models with N/A metrics.