Trust store rot: when a root CA expires and takes your outbound traffic with it

Certificate monitoring almost always points one direction: outward. You watch the leaf on port 443, the cert you present to the world. Expiry thresholds, chain validation, OCSP stapling — the runbook is all about what you serve. And the dashboards stay green, because every cert you hand out really is valid.

But TLS is a two-way relationship, and your systems spend most of their day as clients, not servers. Every outbound call has to validate a chain against a trust store you own and have to keep current. Stripe's API. Your cloud provider's control plane. A webhook endpoint, a package registry, some internal service three teams over. All of them lean on that store, and the store is full of CA certificates. Those have notAfter dates too. When a root or a cross-sign inside the store expires, your outbound traffic starts failing while every certificate you serve stays perfectly valid. And nothing on your monitoring lights up, because your monitoring is pointed the wrong way.

This is trust store rot, and it has taken down big chunks of the internet at least twice in recent memory.

Two roots, two bad afternoons

On May 30, 2020, the AddTrust External CA Root expired. It was one of the old Comodo/Sectigo anchors, and Sectigo had long since moved new issuance over to the newer USERTrust RSA Certification Authority root. On paper the migration was clean: new leaves chained to USERTrust, which most trust stores already carried. But servers kept stapling the old AddTrust cross-signed intermediate into the chain they sent, and plenty of clients kept building the path all the way up to the now-expired AddTrust root. Stripe integrations, Heroku, Roku boxes, a pile of Node.js services — all of them throwing errors against remote certificates that were, by any reasonable definition, still valid.

Sixteen months later, on September 30, 2021, DST Root CA X3 expired and the whole thing happened again, bigger. DST Root CA X3 was the IdenTrust root that cross-signed Let's Encrypt's ISRG Root X1 back when almost nothing trusted ISRG directly. By 2021 most modern systems had ISRG Root X1 in their store and could build a clean path to it. But a long tail of clients couldn't, or wouldn't, and they went down hard.

In both cases the error nearly everyone saw was certificate has expired. Here's what made it infuriating: the certificate the remote server was actually serving hadn't expired at all. A root three links up the chain had. Same failure, a decade of engineering apart.

The alternate-chain problem

The culprit in both incidents was chain building. More precisely, chain building in OpenSSL 1.0.2 and older.

When a server sends you a certificate, there's often more than one valid path from that leaf up to a root you trust. In the Let's Encrypt case one path went up through DST Root CA X3 (expired) and another went up through ISRG Root X1 (valid, and probably already in your store). A sane client notices the expired path, throws it out, and tries the other one. OpenSSL before 1.1.0 didn't do that reliably. It would latch onto the first chain it built, and if that chain happened to end at an expired root, it returned certificate has expired and quit — never mind the perfectly good alternate path sitting right next to it. OpenSSL 1.1.0 and up, and basically every modern TLS stack, handle alternate chains fine. So the breakage clustered exactly where you'd expect: old runtimes. OpenSSL 1.0.2, LibreSSL, some ancient curl baked into a container image nobody had rebuilt in two years.

Servers made it worse by keeping the retired cross-signed intermediate in the chain they sent, basically nudging naive clients toward the dead path. And there's a genuinely weird footnote: Android before 7.1.1 kept working straight through the DST expiry, and not by luck. Let's Encrypt deliberately obtained a cross-sign that stayed valid past the root's own expiry, because old Android doesn't enforce notAfter on trust anchors at all. So the platforms that broke were, in a way, the ones doing it more correctly. Sit with that one for a second the next time you're sure your assumptions are sound.

Reading which end is broken

Two error strings do most of the diagnostic work, and they mean opposite things.

• certificate has expired: some certificate in the chain your client actually built is past its notAfter. If the remote leaf is obviously fine, the dead link is almost always an intermediate or a root: a trust-store or alternate-chain problem on your end, or a server stapling a stale intermediate.
• unable to get local issuer certificate: the opposite. Your client built a partial chain and then couldn't find the issuer to finish it. Either the server didn't send the intermediate it should have, or your trust store is missing the root that anchors the path. A missing link, not an expired one.

Mix these two up and you'll burn an afternoon. Expired means something in the chain died of old age. Missing means the link was never there to begin with.

One host can disagree with itself

Here's what makes trust store management genuinely annoying: there is no single trust store on a modern host. There are several, and they drift apart on their own.

The OS ships one through the ca-certificates package, the bundle that update-ca-trust regenerates on RHEL or update-ca-certificates on Debian. But your container base image froze a copy of that package at whatever version existed the day it was built. Run a two-year-old python:3.9-slim and you're carrying a two-year-old CA bundle. On top of that, individual runtimes ship their own stores and ignore the OS entirely. Python's certifi vendors Mozilla's bundle and pins it per release. Node compiles its own root list into the binary and only checks the system store if you pass --use-openssl-ca. Java keeps its roots in a cacerts keystore inside $JAVA_HOME, on its own update schedule, often owned by a completely different team.

So one box can hold three or four trust stores that flatly disagree with each other. Your curl says the connection's fine, your Python service throws, and your JVM app has been quietly failing a webhook for a week. When you go to audit trust, "the host" isn't the unit of measurement. The runtime is.

Why 47-day certs turn a rare event into a schedule

Root expiries like AddTrust and DST are once-a-decade events. Intermediate rotation isn't, and it's about to get a lot more frequent.

CA/Browser Forum ballot SC-081v3 is dragging maximum TLS leaf lifetimes down from 398 days toward 47 days by March 2029. The 200-day cap is already live as of March 2026, and 100 days lands in March 2027. All of that is about the certs everyone serves, but the knock-on effect lands squarely on trust stores. When leaves rotate eight to ten times more often, CAs rotate and retire their issuing intermediates far more aggressively too. New intermediates appear and old ones age out. Cross-sign arrangements churn. Every one of those changes is another chance for a client with a stale bundle or a pinned intermediate to fall over.

And if any of your services pin a specific intermediate — people do this constantly, usually as a well-meaning "extra security" thing — you've just signed up to break every single time your CA rotates that intermediate. Under the new rules that's a standing appointment, not a once-a-decade surprise. Pinning a leaf is fragile. Pinning an intermediate is a time bomb, and the fuse keeps getting shorter.

What to actually do

The fix is boring. That's the good news: mostly hygiene, plus one change to what you monitor.

• Audit trust stores per runtime, not per host. Enumerate the OS bundle, then every certifi, every cacerts, every Node binary, and check which CA certificates each one carries and when those anchors expire. Everybody skips this, because it's tedious and invisible right up until it isn't.
• Keep ca-certificates current and rebuild your base images. A CA bundle frozen in a container layer is the single most common way stale trust gets shipped to prod. Pin a base image by digest and never rebuild it, and you're pinning a trust store without meaning to.
• Stop pinning intermediates. Pin the leaf's public key, or the root, if you have to pin anything at all. Pinning an intermediate guarantees breakage on the CA's rotation schedule, and that schedule is only getting busier.
• Test against expired-root fixtures before the real root expires. Build a chain that ends in an already-expired CA and point a service at it in staging. A stack that handles alternate chains shrugs it off. A stack carrying OpenSSL 1.0.2 somewhere falls over — and now you've found out on a Tuesday you picked, instead of during an incident.
• Probe the intermediates your critical dependencies chain to. This is the direction cert monitoring always forgets. The certs you serve are only half your exposure; the other half is the chains you depend on every time you make an outbound call. When Stripe or your cloud provider rotates to a new intermediate, you want to have watched it coming, not hear about it from a wall of failed API calls. That's the whole point of watching the outbound direction: CertPulse probes an external HTTPS endpoint, walks the full chain, and inspects every issuer in it, so a remote-side rotation shows up as a notification instead of an outage.

Every cert dashboard watches the door you're standing in. Trust store rot walks in through the door you forgot you were walking out of.