June 4, 2026
Fast kernel, slow clap
I made a kernel 2.2x faster. It made my training loop 3x slower
He sped up one tiny part, and the internet cackled when the whole thing got worse
TLDR: A developer made one part of an AI training system much faster, only to make the full process far slower because it accidentally disabled another hidden speed boost. Commenters were split between praising the honest write-up and roasting it as overlong, obvious, and weirdly chatbot-coded.
A programmer set out to make an artificial intelligence training system faster and actually did speed up one crucial piece by more than 2 times. Then came the plot twist: when that shiny upgrade was plugged into the full training loop, the whole process got nearly 3 times slower. Cue the comments section, where readers treated this like a classic tech horror story: the fix that breaks everything else.
The biggest crowd reaction wasn’t even about the custom speedup trick. It was about a much simpler discovery: using a pre-allocated memory cache sent graphics card use from 26% to 86%, which one commenter called the biggest real win in the entire project. That turned the vibe from “wow, fancy engineering” to “wait, the boring fix beat the flashy one?” A few readers loved the deep profiling and the honesty of publishing a failure. Others were less charitable, saying the post felt bloated, obvious to anyone who has done performance testing, and stuffed with dramatic AI-style wording.
And then came the sharpest jab: one commenter said the whole thing read like someone asked a chatbot how to optimize code and then tried every suggestion until a few happened to stick. Ouch. So the real drama here isn’t just the slowdown—it’s the age-old internet fight over whether this is a brave, transparent engineering diary or a long-winded lesson that could’ve been much shorter. Either way, the comments had a field day.
Key Points
- •The article describes building an RL post-training loop for Qwen2.5-0.5B-Instruct on GSM8K using Dr. GRPO on a single A10G GPU.
- •Before custom kernel work, the author reports a 4.8× improvement in the rollout phase by optimizing the training loop setup.
- •The article says rollout dominates wall time because generation is a sequential per-token decode process, while updates are a few large batched GPU operations.
- •It explains PPO, GRPO, and Dr. GRPO, including GRPO’s removal of the value network and the bias issues Dr. GRPO is intended to address.
- •A fused decode-attention kernel measured 2.2× faster than the SDPA baseline in microbenchmarks but made integrated decode nearly 3× slower in Hugging Face generate after breaking an auto-compile path.