July 19, 2026
When good math goes feral
From Muon to Gradient Clipping: Some Thoughts on QK Stability
A fancy AI training idea hit a meltdown moment, and the crowd has thoughts
TLDR: The post argues that a promising new way to train AI models can become dangerously unstable in key parts of Transformer systems, forcing researchers to rethink the method. Commenters loved the honesty but split hard between “important breakthrough” and “yet another fancy idea that needs emergency duct tape.”
A new post about AI training has the machine-learning crowd doing what it does best: turning a math problem into a full-on comment-section soap opera. The author digs into why Muon — a buzzy alternative to the usual training tools — can go gloriously wrong when used on two crucial parts of a Transformer model, the pieces that help it decide what information to pay attention to. In plain English: the theory looks elegant, but in real training runs, things can wobble, explode, and sometimes face-plant.
That gap between beautiful idea and messy reality is exactly where the community pounced. One camp was basically cheering: finally, someone admitting that a clever optimization trick can still blow up when it meets the real world. Another camp rolled its eyes and called it a classic case of researchers falling in love with neat math while engineers are left babysitting unstable training jobs at 3 a.m. The hottest disagreement? Whether this is a deep insight that could improve future AI systems, or just another chapter in the never-ending saga of “works in theory, add gradient clipping in practice.”
And yes, the jokes arrived right on schedule. Commenters compared the optimizer to a sports car with no brakes, while others joked that every advanced AI paper eventually rediscovers the same emergency fix: “clip it and pray.” Even people who liked the write-up were having fun with the drama, saying the real star wasn’t Muon at all — it was the community’s favorite plot twist: theory versus vibes.
Key Points
- •The article says the Muon optimizer was originally proposed by Keller Jordan and Jeremy Bernstein and is derived from a function-space view of optimization.
- •It reports that later work such as Kimi K2 demonstrated promise for Muon-based ideas but also exposed instability when original Muon is applied directly to Transformer Q and K matrices.
- •Traditional optimizers like SGD and Adam constrain update size in parameter space, commonly through the Frobenius norm.
- •Muon instead constrains the functional effect of an update by bounding output change for every unit-norm input, which leads to a spectral norm constraint.
- •The article states that this constrained optimization has an analytic solution via the matrix sign function msign, which keeps gradient singular vectors while setting singular values to 1.