August 3, 2026
Cache me outside, Cloudflare
Smaller, faster, safer: running Kimi and GLM at scale
Cloudflare says its AI got cheaper and bigger, but commenters smell a catch
TLDR: Cloudflare says it found ways to run huge AI systems more cheaply and handle much longer chats without hurting quality. Commenters weren’t ready to clap: the big fight was whether this is honest optimization or hidden downgrade territory, with extra grumbling over invisible pricing.
Cloudflare rolled out a brag post about making giant AI models like Kimi and GLM smaller, faster, and cheaper to run near users. In plain English: it found ways to squeeze more conversations into the same expensive chips without, it says, hurting answer quality. That should have been a straightforward win. Instead, the comments turned into a mini courtroom drama over whether the company is being clever, sneaky, or both.
The loudest reaction? Trust issues. One commenter said their “slop detector went off” halfway through the post, which is about as devastating as a tomato thrown from the back row. Another accused Cloudflare of quietly serving compressed versions of models while still marketing the originals, arguing that if the AI is being slimmed down behind the scenes, users deserve a big obvious label on the product page. That sparked the core fight: is this smart engineering, or a transparency problem dressed up as a victory lap?
Not everyone came with pitchforks. Some readers gave Cloudflare credit for at least talking openly about one of AI’s less glamorous secrets: providers often cut corners to save memory. But even the supportive crowd wanted receipts, saying the company’s tests didn’t prove enough about how this affects long chats and coding tools. Then there was the wonderfully petty side quest: someone clicked pricing and found… no pricing, prompting a classic internet “why make me log in just to see the number?” groan. Add in a job-seeker asking what title even fits this kind of work, and the vibe was equal parts tech debate, consumer complaint, and comment-section roast.
Key Points
- •Cloudflare says it serves large open models on Workers AI using additional memory optimizations beyond separating prefill and decode phases.
- •The article highlights three techniques: KV-cache quantization, model-weight compression, and protections for shared cache state when more requests are packed onto hardware.
- •For Kimi K2.6, storing the KV cache in FP8 instead of BF16 reportedly doubles in-memory context capacity from about 686,000 to 1.37 million tokens.
- •Cloudflare reports that FP8 KV-cache storage enabled 64 concurrent decode requests on H200 hardware, reaching 2,192 tokens per second, while BF16 ran out of memory beyond 32 concurrent requests.
- •For GLM 5.2, compressing weights from FP8 to INT4 reduced checkpoint size from 705 GB to 421 GB and per-GPU memory from about 88 GB to 52 GB in an 8-way tensor-parallel deployment.