Security

Your ACME account key can revoke every certificate it issued

September 24, 20269 min readCertPulse Engineering

every team i've worked with has a story about a private key that turned up somewhere it shouldn't. nobody has a story about their acme account key, because nobody thinks about the thing. it got created once, on whatever box happened to run certbot first, and it's been signing every request that box sends to the ca ever since. five years, maybe more.

that key is not the certificate's private key. it's the credential that says you are you, to the ca. and rfc 8555 hands it more authority than most people assume.

what the account key actually authorizes

every request an acme client sends is a jws signed by the account key. new orders, challenge responses, finalize, revoke. the account url the ca gave you at registration is the kid in the protected header of nearly all of them. sign with that key and you are that account.

that cashes out three ways.

rate limits count against the account. not the host, not the ip. let's encrypt's new-orders limit (300 per account per three hours, at time of writing) and the failed-authorization limits are all account-scoped. so forty machines carrying the same baked-in key share one bucket. one broken renewal loop on one of them starves the other thirty-nine, and the error you get back says "rate limit", not "it was host 12".

authorizations cache against the account too. let's encrypt reuses a successful domain authorization for 30 days, but only for the account that earned it. which is why a fresh account has to revalidate everything from zero, and that's precisely what you don't want when you're already reissuing under pressure.

third one is the part that should change where you keep the file. section 7.6: a ca must accept a revocation request from the account that issued the certificate. no revalidation of domain control. no possession of the cert's private key. the account key by itself is enough to revoke anything it issued.

one wrinkle. let's encrypt won't accept keyCompromise unless the request is signed by the certificate's own private key, because that reason code gets the key blocklisted. the issuing account can still revoke with unspecified, superseded, or cessationOfOperation. a browser doesn't care which one you picked. the reason code doesn't soften the outage.

so: anyone who can read that file can revoke every certificate your account has ever issued, across every domain, in a scripted loop that fits on one screen. they never touch your servers. they never need a single tls private key.

go find your copies

certbot keeps it under /etc/letsencrypt/accounts/, in a directory named after the acme server, then a hash of the account. the file is private_key.json, a jwk sitting there in plaintext, mode 0600. the regr.json next to it has your account url in it. you'll want that later, trust me.

acme.sh puts account.key in ~/.acme.sh/ca/acme-v02.api.letsencrypt.org/directory/ on recent versions, or straight in ~/.acme.sh/ on older ones. note the tilde. installed as root? then it's in /root, which is the exact directory people exclude from backups and forget to exclude from container images.

cert-manager stores it in whatever secret your issuer or clusterissuer names in privateKeySecretRef. clusterissuer defaults to the cert-manager namespace. it's an ordinary kubernetes secret, so: base64, not encryption. anyone with get on secrets in that namespace has your key.

caddy writes it into its data directory, under acme//users//. run caddy in a container with no persistent volume and it registers a brand new account on every restart, which is its own quiet little problem.

traefik throws the account key and every issued certificate into acme.json together.

and then the one that actually bites. the golden ami, or the base image, built on a host that had already run certbot. the key got baked into a layer, the image scaled to forty instances, and now forty machines hold a credential that can revoke your production certs. including whatever an autoscaler spins up at 3am and hands to anyone who can reach it. i have watched this happen in a shop where the tls private keys were properly in vault, locked down, audited, and the account key was sitting in a base image on an internal registry that half the company could pull from.

shared or per-host

shared account: one credential to protect, one caa record to pin, one rate-limit bucket you can actually reason about. and a blast radius covering everything you own.

per-host accounts: blast radius stops at that host's certificates. in exchange you hit the new-registrations-per-ip ceiling (10 per 3 hours at let's encrypt) the first time you scale up behind nat, you throw away authorization reuse, and account-level pinning turns into a nightmare you maintain forever.

for most teams the answer is neither. stop shipping the account key around. centralize issuance in cert-manager, a vault pki mount, an internal renewal service, whatever fits, and hand hosts certificates instead of the authority to mint them. more work up front. it's also the only version that survives 47-day certificates, where you're renewing four-odd times as often as you are now.

rollover, and what survives it

section 7.3.5 defines key-change. you build an inner jws signed by the new key, holding the account url and the old public key, wrap that in an outer jws signed by the old key, and post the whole thing to the key-change endpoint. the ca checks both signatures and swaps the key on the account.

the part that matters: the account url doesn't change. authorizations, order history, rate-limit state, contacts, external account binding, caa pinning, all of it survives. rollover is cheap and boring. if you rotate every other credential on a schedule, there's no argument for this one being special. certbot ships it as update_account --new-key; acme.sh and lego have their own.

catch is, key-change needs the old key. if you lost it, rollover isn't on the table. you register a new account, get a new account url, and everything in that previous paragraph resets to zero.

deactivation (7.3.6) is the other lever: post the account url with status "deactivated". irreversible, and the ca stops accepting anything from that account. it does not revoke what's already out there. you reach for it for exactly one reason, the 7.6 problem: deactivating a leaked account key is how you take the revoke button away from whoever has your key. order of operations matters here. stand up the new account, update caa, reissue under it, then deactivate the old one. deactivate first and you've locked yourself out of your own escape route.

honestly, for most teams, losing a let's encrypt account key is a shrug. register a new one, eat the revalidation, done before lunch. two things turn that shrug into a page at 2am.

the two things

caa account pinning

rfc 8657 adds an accounturi parameter to caa. you put it on your issue record and the ca will only issue for that domain when the request comes from that specific account url. good control. a compromised acme client elsewhere in your org, or a validation bypass against somebody else's account, can't get a cert for your name.

but it pins the account url, not the key. key rollover: fine, nothing notices. losing the key and registering fresh: not fine. the new account has a different url, and every issuance for that domain fails caa until dns is updated. if your zone lives with a different team, behind an iac repo with a required review and a change window, you now have a certificate outage whose critical path runs through a pull request. i've never seen that resolve quickly.

with 200-day certs you find out about it eventually. under the 47-day regime landing in 2029, clients start renewing well before expiry, so you find out in weeks instead of seasons. faster feedback. much less slack.

if you use accounturi, put the account url in the runbook, not just in regr.json on some host. and when you rotate accounts on purpose, add the new accounturi next to the old one, wait out the ttl, then drop the old.

external account binding

eab (7.3.4) is how commercial cas tie an acme account to a paying customer. zerossl, google trust services, ssl.com, sectigo, digicert all want it. you pull a key identifier and an hmac secret out of the ca's portal, your client sticks an externalAccountBinding field in the newaccount request, and the ca wires that acme account to your billing relationship.

here's why it matters when the key is gone: registering a replacement account requires a valid eab credential. google trust services issues those through gcloud, single-use per account. zerossl lets you regenerate from the dashboard. others want a support ticket, during business hours, in a timezone that is probably not yours. and the portal login is, every time i've looked, in exactly one person's password manager, and that person is somewhere over the atlantic with their phone off.

so the eab key identifier and hmac secret go in the vault next to the account key. document which ca account they belong to. and find out now, not later, whether your ca self-services replacements.

treating it like the credential it is

the account key belongs in your secrets store, backed up the way your other secrets are backed up. not living exclusively on a stateful volume nobody has ever restored from. test the restore. a backup of /etc/letsencrypt that quietly skips the accounts directory is a very popular way to discover this entire post.

audit file permissions, sure. more useful: audit who can read the directory it sits in. on kubernetes that's rbac on secrets in the issuer's namespace. in images, it's checking your build doesn't copy a state directory forward into a layer.

log your acme client invocations somewhere central. certbot writes to /var/log/letsencrypt/letsencrypt.log, acme.sh logs when you ask it nicely. a revocation that came from your own infrastructure shows up there. one issued by someone else holding a copy of your key shows up nowhere you control, because no public ca gives you an account activity feed. you cannot ask let's encrypt what your account has been up to. that's it, that door is closed.

which leaves one outside signal. certificate transparency. every issuance lands in a log within minutes, so unexpected certs for your domains are visible whether or not your pipeline made them. revocation is the worse half. with ocsp retired at let's encrypt, a mass revocation gives you no clean signal at all. it shows up as firefox users breaking on crlite while chrome users tell you everything's fine, which is a genuinely miserable way to find out. crl checking and endpoint probing are what close that hole.

that hole is what certpulse is for, and i'll be upfront that it's mine: ct watching so issuance you didn't authorize arrives as an alert instead of a support ticket, and endpoint probes so chain and revocation state are something you observe rather than something you assume.

anyway. go open your accounts directory. count how many hosts have a copy of that file.

that number is your blast radius.

-- alex

the credential you never think about is the one with no logs.

This is why we built CertPulse

CertPulse connects to your AWS, Azure, and GCP accounts, enumerates every certificate, monitors your external endpoints, and watches Certificate Transparency logs. One dashboard for every cert. Alerts when auto-renewal fails. Alerts when certs approach expiry. Alerts when someone issues a cert for your domain that you didn't request.

If you're looking for complete certificate visibility without maintaining scripts, we can get you there in about 5 minutes.