February 24, 2026
Speed or spin? Pick a side
We Are Changing Our Developer Productivity Experiment Design
Devs refuse no‑AI work; speedup hints spark chaos
TLDR: METR says AI may now speed up coding, but refusal to do no‑AI tasks makes the data biased and fuzzy. Commenters split between “it’s real speed” and “quality counts,” with warnings that rate limits and costs could slam the party soon.
The nerd fight of the week: METR’s latest update says their “AI slows devs” finding from early 2025 (about 19% slower) may be flipping — but the data is messy. Why? More developers now flat‑out won’t take tasks if AI is banned, pay dropped from $150 to $50 an hour, and timing gets weird when people run multiple bots at once. Repeat participants show a hint of speedup (about 18% faster), new folks show a small nudge, and METR calls it a lower bound because the biggest AI wins are likely being filtered out. Cue the comments section setting itself on fire.
Fans cheer that the vibes finally match reality. As ej88 notes, repeat devs went from “0–40% slowdown” to “-10–40% speedup,” and sgillen dunks on last year’s skeptics: the findings are essentially reversed. But others aren’t buying the confetti. softwaredoug is baffled by selection bias questions, and camgunz throws cold water: unless we measure the whole software life cycle (SDLC — the full process from design to bugs), including defect rates, it’s just bot‑assisted busywork. Meanwhile, arctic‑true drops the meme of the day: rate limits will hit like a truck once labs chase profits. The mood? Half “I can’t code without AI,” half “show me the bug count,” with a dash of “free lunch is ending.”
Key Points
- •METR’s early 2025 study found AI use made tasks 19% slower on average (CI: +2% to +39%).
- •A new study launched in August 2025 with 10 repeat and 47 new developers paid $50/hour, using task randomization between “AI allowed” and “AI disallowed.”
- •Selection effects emerged: many developers refused to participate or submit tasks that could be assigned without AI; 30%–50% reported withholding tasks for this reason.
- •Raw results suggest potential speedups: repeat developers showed an estimated -18% (faster) with AI (CI: -38% to +9%); new recruits showed -4% (CI: -15% to +9%).
- •METR deems the new data an unreliable, likely lower-bound estimate due to refusal bias, reduced pay-driven selection, and unreliable time tracking with multiple concurrent AI agents, and is redesigning the study.