Reloading for a new certificate: what actually happens to in-flight connections

renewal is the part everyone instruments. the acme client exits zero, the dashboard goes green, the expiry alert clears. but the certificate that matters is the one sitting in the serving process's memory, not the one on disk, and those two can disagree for weeks before anybody notices.

i watched a team burn forty minutes on a browser warning while staring at a fullchain.pem with a notAfter three months out. the file was fine. the nginx worker holding that connection had been up since before the renewal and was still handing out the old chain, because the deploy hook that should have reloaded it had been failing quietly since nobody could say when. the acme client's exit code tells you it got a certificate. it says nothing about whether anything started serving it.

that gap is about to get eight times more chances to bite. so: what actually happens when you reload each of the common proxies, and what happens to the connections that were already open when you did.

proxies read certs once

every proxy in wide use reads certificate material at config load time and holds the parsed ssl context in memory. nginx does it in the master before dropping privileges, which is why your private key can be 0600 root-owned and workers still serve tls. haproxy parses at startup. envoy builds transport socket contexts from bootstrap or xds. none of them stat the file per handshake, and you really don't want them to.

so writing a new pem to disk changes nothing about what clients see. something has to tell the process to go read it again. if your monitoring watches files or queries the acm api, it's watching the wrong side of that line.

one exception. nginx will read from disk on every handshake if you put variables in ssl_certificate, because it can't cache what it can't resolve at config time. the docs say this outright: no caching on that path, use with care. it's a legitimate trick for a big sni fleet if you want file-watch semantics, and it costs you real cpu per handshake.

what each proxy actually does on reload

nginx

sighup (or nginx -s reload) makes the master parse the new config, open new listening sockets and cert files, spawn fresh workers, then tell the old workers to shut down gracefully. old workers stop accepting and keep serving what they already have until those connections close.

how long is that? worker_shutdown_timeout. default: unset. as in no timeout at all. a draining worker with an open websocket sits there holding the old cert context for as long as the client keeps the socket alive. hours. days, if the keepalive is generous and the client is a well-behaved daemon.

that's also the mechanism behind a failure i've watched happen twice now. reload often enough with long-lived connections in the mix and shutting-down workers pile up. each one carries its own copy of the config and the cert data. worker count creeps, rss climbs, and eventually the oom killer picks the process with the fattest footprint, which is the one serving your traffic. if you reload on any kind of cadence, bound worker_shutdown_timeout and accept that you're hanging up on a few connections.

and if your listen directives use reuseport, reload closes and reopens the per-worker listening sockets, and there's a window where the kernel can steer a syn at a socket that's on its way out. brief. rare. also the difference between "zero downtime certificate rotation" as a claim and as something you measured.

haproxy

haproxy's master-worker reload is the best design in this whole post. sigusr2 forks a new worker and the listening socket fds get handed over instead of closed and reopened. set expose-fd listeners on the stats socket and the new process inherits them. no unbound window, no connection refused. old workers finish their in-flight work and exit, bounded by hard-stop-after if you set it, which you should, same pileup reason as nginx.

better than that: haproxy lets you skip the reload. the runtime api takes set ssl cert then commit ssl cert and swaps the certificate in memory for new handshakes, no new process anywhere. it's transactional, so the material gets validated before commit and a failed commit leaves the old cert alone.

the trap is that it only touches memory. don't write the file to disk too and the next real reload quietly puts you back on the old certificate. i've seen that turn into an expired-cert page days after a rotation everyone remembered doing right.

envoy

envoy gives you two answers and they are not the same answer. hot restart runs a second envoy next to the first and passes listening sockets over a unix domain socket, with --drain-time-s (default 600) and --parent-shutdown-time-s (default 900) deciding how long the parent hangs around. it works. it's also an entire process swap to change a file.

sds is the right tool. secret discovery service hands certificates over dynamically, from a management server or from files with watched_directory, and envoy updates the transport socket context in place. no restart, no doubled memory. the file variant watches for move events on the directory, so you have to swap by symlink rename rather than writing in place. kubernetes secret projection already does exactly that, which is why it feels effortless there and fiddly everywhere else.

what sds won't do is reach back into existing connections. a new secret applies to new handshakes. sessions negotiated before the update keep what they negotiated.

apache

apachectl graceful sends sigusr1: the parent re-reads config including cert files, children finish their current request and exit. for plain http/1.1 that's a bounded wait. for anything going through mod_proxy_wstunnel, "current request" means the entire tunnel and you're back in nginx land. and if your key has a passphrase, a restart blocks on a prompt with nobody there to type it.

connections that outlive the reload

this is the bit that breaks "reloaded, therefore rotated."

a websocket opened before the reload keeps talking over a tls session negotiated against the old certificate. a grpc client sitting on an http/2 connection with a long keepalive does the same. session resumption stretches the window past the connection itself: with tls 1.3, a resumed session doesn't send the certificate at all. rfc 8446 caps ticket_lifetime at seven days. most servers set something far shorter, but the ceiling is what your threat model has to assume.

for expiry this rarely burns you, because a client that already trusts a session isn't rechecking notAfter mid-stream. it just means the words are wrong. "renewed" doesn't mean "serving." "reloaded" doesn't mean "the old cert is gone."

for revocation it's the entire problem. if you're rotating because a key leaked, the goal is that the compromised certificate stops being presented, and a graceful reload with unbounded drain does not do that. old workers keep serving the old chain to whoever's still attached. compromise rotation needs bounded drain or a hard restart, plus proof that the old serial isn't on the wire anywhere. that got more urgent when let's encrypt shut down its ocsp responders in 2025 in favor of crls, and browser crl aggregation updates on its own schedule, not yours. a revoked cert does not stop working quickly. you have to stop serving it.

failure modes worth rehearsing

the half-written pem. a deploy hook copies fullchain.pem into place non-atomically and races a reload firing on a timer. the proxy reads a truncated file. write to a temp file on the same filesystem, rename() it into place, then reload. atomic on posix. and not before.

the config test that lies. nginx -t and haproxy -c check that the certificate parses and that the key matches. they do not check that the chain is complete or in the right order. a fullchain.pem missing its intermediate passes every local check and then fails for every client that doesn't already have that intermediate cached. which usually means: fine in your browser, broken for java clients and for curl in a minimal container. validate the chain against a real trust store before you reload.

memory doubling. any fork-based reload has the old and new config in ram at the same time. terminating a few thousand certs with fat san lists inside a container with a tight memory limit, that's a real spike at the exact moment a restart hurts most.

reload storms. a hundred hosts share a cron line, all renew at 03:00, hammer the ca, then reload on the same second behind the same load balancer. certbot has been saying randomize it forever. the packaged systemd timer ships a randomized delay, the classic cron snippet sleeps a random interval up to twelve hours. put that same jitter in your deploy hooks, not just the renewal.

multiplying by eight

at 398 days, most hosts reloaded for a certificate once a year. a latent reload bug got one shot annually, and if it went off at 2am you probably blamed something else and moved on.

sc-081v3 changes the arithmetic. 200 days max has been in effect since march 2026. 100 days lands march 2027, 47 days march 2029, with domain validation reuse dropping to 10 days at the same time. 47 days is roughly eight rotations a year as a floor, eleven or twelve if you renew at two-thirds of lifetime like a reasonable person. every reload path you've never actually tested gets an order of magnitude more chances to surprise you.

what i'd do before then.

jitter the deploy hooks, not just the renewal timers.

pick proxies that don't need a reload at all. haproxy's runtime api, envoy sds with a watched directory, caddy and traefik doing their own issuance in-process. a rotation that never forks a process has no drain semantics to get wrong.

verify from outside. the acme client's exit code says a file got written. the reload's exit code says a signal got delivered. neither says anything about what the socket returns. the check that closes the loop is a tls probe that connects with the right sni, reads the leaf's serial and notAfter, and compares both against what you deployed. probe each backend directly, not the vip. otherwise you hit four healthy nodes and never find the fifth that never reloaded.

that outside view is the whole reason certpulse's endpoint monitoring reads what the server actually presents, per endpoint, on a schedule. a renewal that succeeded on disk and failed at the socket shows up as a mismatch instead of as a page three weeks later.

the filesystem is not the source of truth. the handshake is.

-- alex

go openssl s_client your own edge tonight. one of those nodes is lying to you.