December 27, 2025
The Grinch stole their certs
The Dangers of SSL Certificates
Holiday cert flop crashes Bazel builds; users roast auto-renew and demand real backups
TLDR: Bazel’s download sites went dark when an SSL certificate expired, breaking builds until Google rushed a fix. Commenters clashed over causes and cures: better alerting and expiry monitoring vs overlapping backup certs—or even keeping HTTP as a fallback—amid warnings that shorter cert lifetimes could worsen outages
Boxing Day turned into a certpocalypse for Google’s Bazel: SSL certificates on bcr.bazel.build and releases.bazel.build expired, and builds across the world faceplanted. On GitHub, Xùdōng Yáng said auto‑renew was “bricked” by new subdomains and the alerts didn’t fire—so holiday on‑call engineers scrambled through docs to fix it. The community came in hot. One camp rolled its eyes at the “SSL is uniquely dangerous” angle, arguing this was just bad monitoring and a single broken alert chain. Another camp shouted that certificates are a single point of failure and pitched overlapping “backup certs” so one expiring doesn’t nuke everything at once. Then a spicy third camp said the quiet part loud: bring back plain old HTTP as a safety net, because “HTTPS‑only does more harm than good” when the lock icon turns on you. War stories poured in—on‑call nightmares, subdomain snafus, and “set‑it‑and‑forget‑it” turning into set‑it‑and‑regret‑it. Someone even warned lifetimes could shrink to 47 days by 2029, cue the collective groan. Meme energy: “Boxing Day KO,” “the Grinch stole their certs,” and “cron job ate my holiday.” The vibe? Fix your alerts, plan for failure, and maybe stop trusting magic renewals
Key Points
- •An SSL certificate for bcr.bazel.build and releases.bazel.build expired, disrupting Bazel build workflows.
- •An automated certificate renewal process failed after new subdomains were added.
- •Renewal failure alerts did not trigger, allowing the certificate to expire unnoticed.
- •Bazel team members unfamiliar with certificate management had to quickly address the issue using documentation and permissions.
- •The article describes certificate expiration as an abrupt, time-driven failure that can affect all users simultaneously.