OpenAI no longer recommends SWE-Bench Pro

OpenAI just did a very public breakup with SWE-Bench Pro, the coding test it had previously pointed people toward after losing faith in an older benchmark. In its new write-up, OpenAI says a detailed audit found about 30% of the tasks were broken, meaning the scoreboard may have been giving everyone a shaky picture of how good these systems really are at software work. In plain English: the test many people were using to judge progress may have had a lot of bad questions.

But the real show was in the comments, where the mood was less shocked and more "called it". One of the sharpest reactions basically translated the whole saga as: other labs simply got better at gaming the benchmark than OpenAI did. Ouch. Another commenter zoomed out and argued that true machine intelligence won’t be proven by acing a bunch of canned tests anyway, because real life throws messy, unknown problems at you.

Then came the comedy. One user joked that their personal benchmark is whether an AI can recreate a 1990s Super Soaker with parts from Home Depot, while another said they use it to design real-world parts from descriptions and the models still flop. That sparked the broader hot take hanging over the thread: maybe public benchmarks keep getting hyped, milked, and abandoned, while serious users are increasingly building their own private tests to see what actually works. The crowd verdict? Less faith in flashy leaderboards, more side-eye, sarcasm, and DIY reality checks.