October 31, 2025
Namespace Hoarders: Kube Edition
How We Found 7 TiB of Memory Just Sitting Around
Found 7 TB hiding; crowd argues smart hack vs fix Kubernetes
TLDR: A team saved massive memory by stopping a logging tool from watching namespace labels and using a simple name check instead. Commenters split: some cheer smart profiling; others demand a deeper Kubernetes fix, while nerds warn tag and naming “cardinality” keeps biting big systems—this matters for stability at scale.
Engineers dug through a maze of Kubernetes “namespaces” (think: folders for apps) and uncovered a shocking truth: 7 TiB of memory was being chewed up by lots of tiny watchers peeking at those folders. The fix? Tell the logging tool Vector to stop obsessing over namespace labels and use a simple name check instead. Cue the comment section confetti and chaos. One camp cheered the sleuthing—“profiling and digging deep” wins again, said shanemhansen—like watching a detective find money behind the couch. Another camp, led by nitinreddy88, wasn’t buying the quick patch vibes: “Why is a label causing this much memory? Fix Kubernetes,” they argued, calling for a bigger redesign that helps everyone. Then came the math nerds. hinkley rolled in with brainy talk about keys and how “human‑friendly” naming explodes memory, plus a nod to Prometheus metrics changes—because tag explosion isn’t just a Kubernetes problem. The memes? People joked about “Namespace Hoarders: Kubernetes Edition,” imagining daemonsets (apps that run on every machine) all binge‑watching the same list and crashing the API server (the cluster’s control tower) during rollouts. It’s the classic tech soap: clever hack vs systemic fix, with a side of cardinality discourse and couch‑cushion treasure. Read it, then decide if you want duct tape or a new house.
Key Points
- •High namespace counts cause processes that listwatch namespaces/netpols to consume large memory and stress the kube-apiserver.
- •Daemonsets exacerbate the issue because each pod listwatches the same resources, increasing memory usage with node count and risking apiserver overload during restarts.
- •Calico’s memory usage was reduced through collaboration with maintainers and testing in a staging cluster with several hundred thousand namespaces.
- •Vector was found to listwatch namespaces to use namespace labels in logs; the team replaced this with a namespace name prefix check to avoid label lookups.
- •A new Vector configuration option to disable insertion of namespace-derived fields reduces kube-apiserver load and daemonset memory usage in large clusters.