The Dangers of SSL Certificates

Holiday cert flop crashes Bazel builds; users roast auto-renew and demand real backups

TLDR: Bazel’s download sites went dark when an SSL certificate expired, breaking builds until Google rushed a fix. Commenters clashed over causes and cures: better alerting and expiry monitoring vs overlapping backup certs—or even keeping HTTP as a fallback—amid warnings that shorter cert lifetimes could worsen outages

Boxing Day turned into a certpocalypse for Google’s Bazel: SSL certificates on bcr.bazel.build and releases.bazel.build expired, and builds across the world faceplanted. On GitHub, Xùdōng Yáng said auto‑renew was “bricked” by new subdomains and the alerts didn’t fire—so holiday on‑call engineers scrambled through docs to fix it. The community came in hot. One camp rolled its eyes at the “SSL is uniquely dangerous” angle, arguing this was just bad monitoring and a single broken alert chain. Another camp shouted that certificates are a single point of failure and pitched overlapping “backup certs” so one expiring doesn’t nuke everything at once. Then a spicy third camp said the quiet part loud: bring back plain old HTTP as a safety net, because “HTTPS‑only does more harm than good” when the lock icon turns on you. War stories poured in—on‑call nightmares, subdomain snafus, and “set‑it‑and‑forget‑it” turning into set‑it‑and‑regret‑it. Someone even warned lifetimes could shrink to 47 days by 2029, cue the collective groan. Meme energy: “Boxing Day KO,” “the Grinch stole their certs,” and “cron job ate my holiday.” The vibe? Fix your alerts, plan for failure, and maybe stop trusting magic renewals

Key Points

  • An SSL certificate for bcr.bazel.build and releases.bazel.build expired, disrupting Bazel build workflows.
  • An automated certificate renewal process failed after new subdomains were added.
  • Renewal failure alerts did not trigger, allowing the certificate to expire unnoticed.
  • Bazel team members unfamiliar with certificate management had to quickly address the issue using documentation and permissions.
  • The article describes certificate expiration as an abrupt, time-driven failure that can affect all users simultaneously.

Hottest takes

“it sounds like this team was relying only on a ‘failed to renew’ alert/monitor” — loloquwowndueo
“Having only one SSL certificate is a single point of failure” — flowerlad
“HTTPS-only does more harm than good” — superkuh
Made with <3 by @siedrix and @shesho from CDMX. Powered by Forge&Hive.