April 9, 2026
Streams, screams, and protobuf dreams
Progressive encoding and decoding of 'repeated' protobuffer fields
Trying to stream mega‑files without melting RAM, and the comments go wild
TLDR: A dev explains how to stream massive Protobuf trace files piece‑by‑piece so RAM doesn’t explode, especially for Google’s Perfetto. Comments erupt into tool drops—a mini Protobuf VM and a new BLIP format—and a tug‑of‑war over hacking Protobuf to stream versus switching to faster, zero‑copy alternatives.
A developer set out to solve a painfully relatable problem: how to write and read gigantic trace files without loading the whole thing into memory. The culprit is a popular data format, Protobuf, where a long list of items (called a “repeated field”) usually gets packed as one big bundle. The post breaks down a plan to stream those pieces bit‑by‑bit, especially for Google’s Perfetto trace viewer, which can spit out millions of events. Translation: fewer crashes, less fan screaming, happier laptops.
But the real show? The comments. One reader swaggered in with a DIY power move—a tiny virtual machine for Protobuf that filters data in streaming pipelines and even in WebAssembly (think: mini apps in the browser or servers). They dropped a link and the mic, then added the kicker: “100% written by Claude.” Cue gasps, cheers, and side‑eye. Another commenter pitched their own invention, BLIP, a new way to pack big numbers and a matching container format—clearly hoping to become the new hotness in “how to ship huge data” BLIP.
Meanwhile, the peanut gallery split into camps. Some shouted “just use Cap’n Proto” (an alternative built for zero‑copy speed). Others loved the gritty, pragmatic “make Protobuf streamable” approach. A few joked the thread was a stealth startup pitch: not just solving streaming—selling a format, a VM, and maybe a dream. Either way, the RAM‑meltdown memes wrote themselves.
Key Points
- •Perfetto traces are composed of long sequences of TracePacket messages contained in a Trace message with a repeated field.
- •Standard protobuf APIs typically require constructing the entire Trace in memory before serialization, causing issues for gigabyte-scale traces.
- •The article seeks progressive (streaming) encoding and decoding of repeated protobuf fields to write/read TracePackets incrementally.
- •Most protobuf libraries do not directly support streaming access to repeated fields in this manner.
- •The piece begins detailing the protobuf wire format, introducing VARINTs as foundational to understanding how streaming could be achieved.