November 2, 2025
Logs on a crash diet
We reduced a container image from 800GB to 2GB
800GB dev box crash diet sparks cheers, jeers, and privacy panic
TLDR: Sealos shrank a runaway container from 800GB to 2GB by killing bloated login-failure logs and flattening layers. The community split between “you misconfigured it,” worried privacy watchdogs, and “read the article” defenders, making this a cautionary tale about dev box design and transparency in cloud tooling.
Sealos says they slimmed a grotesquely bloated container—from a jaw-dropping 800GB down to 2GB—after mystery logs ballooned like a digital hoarder’s basement. The post explains that a log of failed logins swelled to 11GB and got copied across layers, multiplying the mess. They used a custom tool, image-manip, to “squash” layers (think: flattening stacks of changes) and surgically delete junk, and the cluster breathed again. Cue the comments: the loudest camp called it a “misconfig confession,” with one reader dismissing the write-up as basically “we broke it, we fixed it.” Another chorus piled on with “if your image is 800GB you’re doing something wrong,” turning the thread into a meme factory about “container weight loss plans” and “390:1 diets.” But defenders pointed out nuance: untrimmed explained the dev boxes need an SSH server because popular IDEs connect over SSH, which likely attracted bot storms that spammed failed logins and blew up logs. Then came the privacy panic: the idea of engineers peeking into customer images set off alarm bells about secrets and repos. Finally, the nerd police arrived: “read the article,” they scolded, noting the 11GB log got amplified by copy behavior—not a single 800GB image. Drama, receipts, and a very hungry log file. Read the full story at Sealos.
Key Points
- •Sealos faced persistent disk exhaustion in a development Kubernetes node despite expanding storage.
- •Investigation with iotop showed containerd writing >100MB/s, indicating container-level issues.
- •du scans of overlayfs snapshots revealed repeated 11GB /var/log/btmp files causing bloat.
- •Sealos built a custom tool, image-manip, to remove files and squash OCI image layers.
- •An 800GB, 272-layer image was reduced to 2.05GB (390:1), resolving production alerts and stabilizing environments.