When cert-manager stops renewing: reading the Certificate, Order, and Challenge chain

cert-manager breaks politely. that's the whole problem with it.

no crashloopbackoff, no 500s, nothing pages. the Certificate object sits there with Ready: True while the Order underneath it has been wedged in the same state since the tuesday before last. everything looks fine right up until the morning your existing cert runs out and the ingress starts serving something that expired four hours ago.

i've walked into this more than once. it's always the same story: someone changed an ingress annotation, or rotated a secret, or rebuilt a namespace, and the renewal path quietly died while the still-valid cert kept every dashboard green. here's how to actually read the chain, and what it usually turns out to be.

Ready does not mean what your dashboard thinks

the Certificate's Ready condition describes the cert currently sitting in the target Secret. does it exist, does it match the spec, is it still inside its validity window. that's the entire scope. it says nothing about whether the last renewal attempt worked.

so a cert that's been failing renewal for two weeks and a cert that renewed cleanly last night look identical. both True. and Ready is what certmanager_certificate_ready_status exports, which is what almost every cluster dashboard graphs. the panel stays green until the day it doesn't, and by then you're counting hours.

the condition you want is Issuing. cert-manager sets it when it starts a new issuance and clears it when the new cert lands in the Secret. Issuing: True for more than an hour or two means a renewal that isn't completing. if you change one alert after reading this, change that one. pair it with certmanager_certificate_expiration_timestamp_seconds and you'll catch both the stall and the slow bleed.

the chain, top to bottom

the ACME flow is four resources deep and the error you need is basically never on the top one.

a Certificate owns a CertificateRequest, which is one attempt at issuance. for an ACME issuer that CertificateRequest owns an Order. the Order owns one Challenge per DNS name. every layer has its own conditions and its own events, and the sentence you're actually hunting for tends to live two or three levels down.

so the loop is: describe the Certificate, find the CertificateRequest it points at, describe that, find the Order, describe that, list Challenges, describe those. read the events at each layer, not just the conditions. the reason string on a Challenge is where the real error text hides.

two things will bite you on the way down.

CertificateRequests get pruned. revisionHistoryLimit on the Certificate decides how many stick around, and if someone set it to 1 to keep the namespace tidy, yesterday's failed attempt is already gone and you're staring at the retry instead of the original. leave it unset and you get the other flavour of pain: a heap of them, sorted by nothing useful, and you're squinting at revision numbers.

Challenges are ephemeral on purpose. when an Order finalizes, pass or fail, cert-manager deletes the Challenge resources along with every solver pod, service and ingress it made. debugging after the fact means debugging nothing. which is why a cert stuck pending is the easy case and a cert that failed and retried is the miserable one. the stuck one still has its Challenge sitting right there with reason filled in.

cmctl status certificate walks the whole thing in one shot: conditions, issuer, current CertificateRequest, Order, Challenges, events, Secret details. it's the first command i run now, every time. cmctl renew forces a fresh issuance without the delete-the-Secret-and-pray ritual, and that matters, because deleting the Secret throws away your working cert while you debug the broken renewal. don't do that at 2am. i have done that at 2am.

if the chain is just empty, no CertificateRequest at all, stop walking. go read the controller logs. that's a reconcile failure, not an ACME failure, and it fails for completely different reasons.

the failure modes worth memorizing

HTTP-01 solver unreachable

cert-manager spins up a temporary pod, service and ingress to serve the token under /.well-known/acme-challenge. the Challenge reason usually says it's waiting for propagation and hands you the exact GET it tried plus how it went wrong.

nine times out of ten something else is shadowing the path. a catch-all rule on the same host, a rewrite-target annotation mangling it, a config snippet doing its own routing. the challenge gets swallowed before the solver ever sees it. runner-up cause: ingressClassName mismatch, where the solver ingress comes up with no class or the wrong one and your controller ignores it completely. set the class explicitly in the solver config and that whole category disappears.

the redirect-to-HTTPS thing is less scary than the folklore. let's encrypt follows redirects, http to https included, and doesn't check the cert on the far end, so a plain ssl-redirect annotation is normally survivable. what actually kills you is a redirect that changes the path or the host. a permanent-redirect annotation aimed at your marketing site. a rewrite that eats the acme-challenge path on the way through.

and before any of that happens, cert-manager runs its own self-check from inside the cluster. if your cluster DNS resolves the public hostname to some internal address that isn't the ingress, the self-check fails and cert-manager never even asks the CA to validate. the Challenge sits pending forever and there's nothing in the ACME server's logs, because there was never a request.

DNS-01 stuck on propagation

same shape, different plumbing. the Challenge says it's waiting for DNS-01 propagation and the TXT record is either missing or invisible to whatever resolver cert-manager is asking.

long TTLs are the boring answer. old TXT record for the same validation name cached at an hour, you wait an hour, congratulations. split-horizon DNS is the fun one. your internal resolver answers authoritatively for the zone and has never heard of the record your provider just wrote externally. cert-manager ships flags for this: point dns01-recursive-nameservers at public resolvers and set dns01-recursive-nameservers-only so the propagation check stops asking the cluster. if you use CNAME delegation into a dedicated validation zone, that pair isn't optional, it's load-bearing.

the ACME account key vanished

your Issuer or ClusterIssuer references a Secret holding the ACME account private key. rebuild the namespace, restore from a backup that skipped it, let a sync tool prune it, and cert-manager just registers a brand new account on the next reconcile. you keep working, technically. you also lose every cached validated authorization on the old account, so every pending renewal re-validates from zero.

worse case: the key exists but doesn't match the account URI in the Issuer status. then you get account does not exist or an unauthorized error from the ACME server and the Issuer itself goes not-ready. back the Secret up. it's a few hundred bytes and it's the only thread connecting you to your account's history.

rate limit lockout

a misconfigured Issuer retrying in a loop will find let's encrypt's limits fast. the two that actually bite are the duplicate certificate limit (five identical name sets a week) and the failed-validation limit (five failures per account per hostname per hour). you get a rateLimited problem type with a message that says plainly what happened, cert-manager backs off, and the window is already burnt.

let's encrypt keeps revising the limit structure, so go read their current docs instead of trusting a number you memorized in 2021. the lesson underneath doesn't move: test Issuer changes against staging. always. a staging ClusterIssuer costs nothing and exists for exactly this moment.

reconciliation stopped entirely

no CertificateRequest ever appears, so the controller isn't getting that far. webhooks first: failed calling webhook with a context deadline or an x509 error means the webhook's serving cert or CA bundle drifted, usually after an upgrade or a cainjector restart. RBAC gaps show up as cert-manager being unable to create the solver ingress in one specific namespace. and admission policy is an increasingly common cause. a kyverno or gatekeeper rule demanding resource limits or particular labels will cheerfully reject the solver pod, and the rejection lands in the Order's events, which is nowhere near where you'd think to look.

renewal timing, and why it's about to get worse

renewBefore is a duration. not a percentage. that distinction is going to cost people real money over the next three years.

by default cert-manager renews at two-thirds of the cert lifetime, so a 90-day cert goes around day 60 with a month of slack. plenty of teams hardcoded renewBefore: 720h to spell that out. now watch what that value does when the CA starts handing out 47-day certs: renewBefore is longer than the cert's entire life, cert-manager logs a warning, falls back to the two-thirds default, and your explicit intent is silently discarded. better than the alternative, sure. you still lost control of the timing and nobody told you.

under SC-081v3 the 200-day phase went live in march 2026. 100 days lands march 2027. 47 days march 2029. every hardcoded renewBefore in your repos is sitting on a timer. recent cert-manager versions added renewBeforePercentage for precisely this, and it's the field to migrate to, because it scales with whatever the CA decides to give you.

the other timing problem is one GitOps invented. apply two hundred Certificates in one sync and they all get notAfter timestamps within seconds of each other, so they all renew the same day, the same hour, forever. at 90 days that's four irritating days a year. at 47 days it's every six weeks or so, and the whole burst has to clear your ACME account limits and your DNS provider's API limits simultaneously. route 53 throttles. cloudflare throttles. a DNS-01 fleet renewing in lockstep is an excellent way to discover exactly where those ceilings are, at the worst possible time. jitter them on purpose by varying renewBefore or renewBeforePercentage across the fleet. a few percent turns a spike into a trickle.

a Secret is not a served certificate

this is the part everyone skips and it's the part that generates the genuinely baffling incidents.

cert-manager's job ends when it writes the key pair into a kubernetes Secret. whether anything starts serving that key pair is entirely somebody else's problem.

ingress controllers mostly handle it. ingress-nginx watches the Secrets it references and reloads. envoy-based gateways pull over SDS and pick up the change without a restart. fine, no notes.

workloads mounting the Secret as a volume are where it comes apart. the kubelet does refresh the projected files, usually inside a minute. your process opened those files at startup and has no intention of ever reading them again. it needs a signal, a restart, or something like Reloader watching the Secret and rolling the Deployment. and if anybody mounted the cert with subPath, the file never updates at all. not slowly. never. subPath mounts don't receive updates from the API server, full stop, so that pod serves whatever it booted with until something replaces it, which might be months from now.

which is the whole reason i care about this. every check i've described asks kubernetes what kubernetes believes. the only check that counts asks the endpoint what it's presenting: connect, read the leaf, compare notAfter and the serial against what cert-manager thinks it issued. internal metrics tell you the control loop is alive. an external probe tells you the truth, and it catches the ugly case where cert-manager did everything perfectly and users still get a bad cert. it also catches the certs nobody ever put in cert-manager, which in every environment i've looked at is a bigger pile than anyone wants to admit. that's the gap certpulse sits in next to your prometheus rules, not on top of them. cert-manager metrics for the pipeline, endpoint probes for the result.

two things to do this week. alert on Issuing, not just Ready. then go check what your services are actually serving, from outside the cluster, because a correct Secret has never once been the same thing as a live certificate.

green dashboards are a story the cluster tells itself. go open a socket and ask.

-- alex