July 28, 2026
Attention? More like notation warfare
You Could Have Come Up with Kimi Delta Attention
Readers saw the headline, saw the math, and immediately called bluff on “you could too”
TLDR: The article argues that a newer AI memory method can be understood with simple logic, not wizardry, and explains why it helps models remember more efficiently. But the comments stole the show, with readers split between “nice explanation” and “sir, I blacked out at the physics symbols.”
A spicy little math post tried to do something very bold: tell readers that a new AI memory trick called Kimi Delta Attention is not magic, and that if you follow the logic step by step, you basically could have invented it yourself. The internet’s response? Absolutely not, babe. The biggest laugh in the comments came from one brutally honest reader who summed up the mood with, “I could not, in fact, have come up with Kimi Delta Attention.” That was the vibe: less Eureka!, more I opened this and my soul left my body.
To be fair, the article’s actual point is surprisingly practical. It tries to explain how chatbots can remember earlier words without checking every single previous word each time, which gets slow and expensive. The author says this newer method updates memory more carefully instead of just piling on more stuff and causing confusion. But before readers could even get to the “aha” moment, the real drama erupted over the presentation. One camp was instantly distracted by the author using quantum-style bra-ket notation—yes, the angle-bracket math that looks like physics wandered into AI class. One commenter flat-out asked, “why are you using braket notation?” while another joked that if the author needs a warning label at the top to explain the symbols, you know you’re in deep.
Not everyone was grumpy, though. A few readers actually loved the article’s math-vs-physics toggle, calling it a neat peace treaty for confused engineers and notation snobs alike. And one commenter voiced the bigger everyday frustration: modern AI diagrams look like mysterious boxes nested inside boxes, and regular humans just want to know what happens live and what gets saved for later. In other words, the article wanted to demystify cutting-edge AI math—but the comments turned it into a referendum on whether “clear explanation” and “fancy symbols” can ever peacefully coexist.
Key Points
- •The article explains Kimi Delta Attention by deriving it from the DeltaNet family of linear attention methods rather than treating its final equation as opaque.
- •It contrasts standard softmax attention, which has quadratic key-query interactions over a sequence, with linear attention that summarizes history in a fixed-size state of summed outer products.
- •The article identifies softmax normalization as the main reason standard attention cannot be easily rearranged into a recurrent state update.
- •It shows that naive linear-attention writes are additive and can cause interference because new writes do not overwrite previous associations when keys are not orthogonal.
- •DeltaNet addresses this by reading the current memory prediction for a key, computing the error to the target value, and writing only that delta, optionally scaled by a coefficient βt.