December 15, 2025
mTLS? More like WTFLS
Debugging TLS failures in distroless containers
Prod outage: app didn’t trust the server; skinny containers blamed
TLDR: The failure came from a production container missing trusted certificate roots, so it didn’t believe the server. Comments split between dunking on minimal “distroless” images and calling for better certificate management, with hot takes on pinning trust and memes about TLS always being the culprit.
The article says the team chased a secure connection failure and discovered the real villain wasn’t their client certificate, but trust: the app in production didn’t believe the server’s certificate authority (the “who vouches for you” part). Cue the comments: half the crowd roasted distroless (super-minimal containers) for shipping without the list of trusted certificate roots, while others defended minimal images and scolded teams for not baking in the cert bundle. One popular joke: “mTLS (mutual TLS, where both sides show ID) = make Things Less Simple.” Another: “It’s always DNS? Nah, it’s always CA.” Links flew to past fiascos like the Let’s Encrypt root expiration, and to the distroless project, with SREs swapping war stories of midnight openssl s_client spelunking. Strong opinions? Security purists demanded pinning the exact certificate or authority instead of trusting “whatever the OS ships.” Pragmatists begged teams to use a normal base image and avoid heroic debugging at 3 AM. The thread devolved into memes about “works on my machine” versus “prod forgot /etc/ssl.” Drama, dunking, and a reminder: trust chains matter, especially when your container is too skinny to know who to trust.
Key Points
- •A production TLS handshake failure occurred when the client did not trust the server’s CA root certificate.
- •Services use mTLS over HTTP with third-party providers, often implicating client certificates in issues, but not in this case.
- •TLS fundamentals are reviewed, including certificate contents, CA trust, and handshake verification steps.
- •The issue was reproduced locally using OpenSSL’s s_client command, which succeeded, isolating the problem to environment differences.
- •Linux maintains a CA trust store (e.g., /etc/ssl/certs/ca-certificates.crt), and verifying the server’s certificate chain (root and intermediates) is crucial.