July 28, 2026
Attention drama just got linear
Kimi Linear: An Expressive, Efficient Attention Architecture
AI nerds are calling this a monster win — and telling doubters to keep up
TLDR: Kimi Linear claims it can make AI models faster, lighter, and even better at handling long text, which is a big deal for building cheaper and stronger tools. In the comments, fans are hyping it as proof Kimi’s success is real, while swatting away skeptics and arguing this “old” paper secretly powers the company’s newer hits.
The big claim here is simple enough even for non-experts: Moonshot says its new system, Kimi Linear, is not just cheaper and faster than the usual way large AI models pay attention to text — it can actually do a better job too. That’s a flashy promise in a field where “faster and cheaper” often comes with a catch. According to the paper, it cuts the model’s memory burden by up to 75% and can spit out long answers far faster, while still beating the old standard. In plain English: less strain, more speed, and allegedly better results.
But the real fireworks are in the comments, where the vibe is part victory lap, part subtweet. One fan rolled in with “Another banger from Zhang et. al”, basically crowning the authors as repeat hitmakers. Another commenter pushed back against a recurring rumor clouding Kimi’s rise, snapping, “If you want to believe that the success of Kimi is about distillation attacks, ignore this.” Translation: some people have been trying to explain away Kimi’s momentum, and supporters are clearly over it.
Then there’s the classic tech-forum twist: the paper may be “old,” but commenters insist it’s suddenly very relevant because newer work like Kimi K3 appears to build directly on it. One builder even chimed in with a practical flex, saying they started using it internally and saw strong results before an even newer follow-up, Gated DeltaNet 2, raised the bar again. So yes, the paper is about AI architecture — but the comments read like a fandom arguing over whether they spotted the breakout star before everyone else.
Key Points
- •The article introduces Kimi Linear, a hybrid linear attention architecture reported to outperform full attention under fair comparisons across short-context, long-context, and RL scaling settings.
- •Kimi Linear is centered on Kimi Delta Attention, which extends Gated DeltaNet with finer-grained gating to use finite-state RNN memory more effectively.
- •A custom chunkwise algorithm with a specialized Diagonal-Plus-Low-Rank transition-matrix variant is used to improve hardware efficiency and reduce computation.
- •The reported pretrained model has 3B activated parameters and 48B total parameters, combining KDA and Multi-Head Latent Attention in a layerwise hybrid design.
- •The authors report up to 75% lower KV cache usage and up to 6× decoding throughput at 1M context, and they released open-source kernels, vLLM implementations, and checkpoints.