The certificates you can't automate: appliances, vendor portals, and the manual tail

every certificate automation story gets told from the 95% that works. cert-manager is wired up, acm renews everything behind an alb, let's encrypt fires at 30 days remaining and nobody notices, dashboard is green. then someone pages at 2am because the vpn concentrator stopped accepting connections. or prometheus can't scrape a node because the bmc's web cert died. or a customer emails to say your status page went untrusted around midnight, because the pem somebody pasted into a vendor portal last spring ran out.

the pages come from the tail. they always have.

the f5 management interface. the palo alto management interface. idrac, ilo, and whatever else is answering on 443 in that rack. the vpn concentrator. the java appliance from 2016 with a jks keystore and a renewal runbook whose only named contact left in 2021. the saas vendor where "bring your own domain" means logging into an admin panel, pasting a cert, a key, and a chain into three textareas, clicking save, and hoping.

at 398 days, every one of those was an annual annoyance. calendar reminder fires, someone does the thing, it works for another year. that model is dead. sc-081v3 already cut us to 200-day maximum lifetimes back in march. march 2027 takes it to 100. march 2029 takes it to 47, with a 10-day reuse window on domain validation. a cert you renew by hand at 47 days is roughly eight touches a year. multiply that by however many appliances are in your tail and you've just created a part-time job that nobody has been assigned and nobody will volunteer for.

inventory by renewal mechanism, not by location

most cert inventories are organized by where the thing lives. this aws account, that cluster, this vlan, that vendor. it's the natural way to build the list and it's close to useless for planning. two certs in the same vpc can have completely different renewal stories, and the one that wakes you up is almost never the one you'd guess from its address.

rebuild the list around how the certificate actually gets replaced. four buckets.

fully automated. acme clients, cert-manager, acm, key vault with an integrated ca, gcp certificate manager. no human in the loop, and when it fails it fails with a retryable error. these need monitoring. they don't need scheduling.

api-automatable but not automated yet. there's a management api that can install a cert, but today a person clicks through a gui to do it. this is where the actual work lives, and it's bigger than anyone guesses on the first pass.

human-only. no api, or an api that can't touch certs, or a vendor portal with zero programmatic path. someone logs in and pastes.

unknown. nobody currently employed knows how this cert gets renewed, or who does it, or what breaks when it doesn't.

that last bucket is the whole point of the exercise. in every estate i've walked into, it's the largest one on day one, and it stays large because nothing in the org chart forces it to shrink. a certificate nobody knows how to renew is an outage with a date on it.

each entry needs four fields past the expiry date, and honestly expiry is the least interesting of the five. renewal method: which api, which portal, which keytool incantation. credential path: where does the operator get the account or key that lets them do this, and does that credential expire too? the named human who can do it this week, and i mean a human, not "the network team." and duration. does this take five minutes or does it need a maintenance window because the appliance bounces its web server and drops every session on it?

duration is the field that turns a spreadsheet into a plan. a cert that needs a curl call and a cert that needs 45 minutes plus a cab approval are not the same animal, even though both get filed as "manual."

the api-automatable bucket is bigger than you think

teams write off appliance renewal way too early. the apis are there, and most of them are fine.

f5 has icontrol rest: upload the file, create the crypto cert and key objects, modify the client-ssl profile to point at them. pan-os gives you both an xml api and a rest api, and cert import works through either. modern bmcs speak redfish, and the dmtf spec includes a certificateservice with a replacecertificate action that idrac 9 and ilo 5/6 both implement. citrix adc has nitro. nearly every cdn and waf vendor exposes cert upload over api, because they had to build it for their own enterprise customers anyway.

so it's doable. now the honest part.

every vendor has its own auth model, its own idea of what "install a certificate" means, and its own reload behavior. on pan-os the import is trivial. the commit afterward is asynchronous, takes minutes on a busy box, and will cheerfully ship whatever half-finished candidate config a colleague left sitting in the running session at 4pm friday. on a bmc, replacing the web cert restarts the web server and kills your connection, so your automation has to read a dropped socket as success. that one cost me an afternoon. on an f5 the upload isn't the interesting question at all: the question is whether every ssl profile and every virtual server that pointed at the old cert now points at the new one, including the profile somebody cloned in 2021 and never told anyone about. jks keystores want the right alias, the keystore password from wherever that's buried, a pkcs12 conversion, and a jvm restart.

which means bespoke glue, per vendor. it breaks on firmware upgrades, because vendors reshuffle api surfaces between major versions and nobody at those companies treats cert installation as compatibility-critical. it needs stored privileged credentials, and a credential that can swap the certificate on your edge firewall is not a small thing to leave lying in a vault with loose acls. and you own it forever.

sometimes that's still cheaper than eight manual renewals a year on a box carrying production traffic. sometimes it very much isn't. writing and maintaining a nitro integration for one lab appliance nobody depends on is worse than just doing the clicking. decide device by device. and when the answer is no, say it out loud so the thing lands in the human-only bucket honestly, instead of rotting in a backlog ticket named "automate later" until it expires on a saturday.

two ways out

for the certs that genuinely can't be automated, quit trying to solve renewal and change the shape of the problem instead.

stop serving public certs from things that can't renew them

put a reverse proxy or load balancer you control in front of the appliance. the proxy terminates the publicly trusted cert and renews it like everything else you own. the appliance keeps a long-lived internal or self-signed cert on the back leg, and you pin the proxy's upstream verification to that specific cert or your internal ca. what you do not do is set insecure_skip_verify and walk away, which is exactly what happens nine times out of ten and is the reason this pattern occasionally makes things worse than what it replaced.

it doesn't fit everywhere. some appliances need to see the client cert for mutual tls. some vpn products refuse to have anything sitting between them and the client. but for management interfaces and admin panels, which is most of the tail by count, it works, and it drops the cert into a bucket you already solved.

not everything in the tail needs public trust

sc-081v3 governs publicly trusted certs. a management interface that's only reachable from a jump host or over the vpn has no browser trust requirement a private ca can't satisfy. the chrome and apple lifetime limits apply to chains ending at public roots; a chain ending at a root you installed yourself sits outside that policy entirely. an internal issuer can hand that interface something long-lived, or something properly short with automation you actually control end to end.

the catch is that you now own trust distribution, and trust distribution is where good intentions go to die. browsers on managed laptops are easy. the pain is everything that ignores the system trust store. java apps with their own cacerts. python tooling reading certifi. go binaries in scratch containers with whatever ca bundle the base image shipped. phones. monitoring agents. embedded junk with a five-year support contract. each one becomes a ticket the first time it can't validate, and the tickets don't arrive on the day you flip the switch, they arrive three weeks later from a team you've never met. if you aren't already running internal pki with a distribution story that works, spinning one up to fix twelve appliances is a bad trade.

making the leftovers survivable

whatever's still standing after those two escapes is your permanent manual tail. a few habits keep it from hurting.

alert earlier on manual certs than automated ones. different failure modes deserve different thresholds. an automated renewal that fails at 20 days has 20 days of retries in front of it. a manual renewal at 20 days needs a human, a change window, and maybe a vendor ticket with a 48-hour sla. so count change windows instead of days. at 47-day lifetimes with a biweekly window you get about three shots per cert, and one of those is going to land inside a change freeze or on a public holiday. alert at half the lifetime and page a person, not a channel. channels absorb alerts. people don't.

then treat "unknown owner" as a ticket with a due date. owner means someone who has personally done this renewal or can do it this quarter. not a team, not a dl, not "infra." empty owner field is a p3 with a deadline, and it's the single best predictor i know of a 2am page.

write the per-appliance steps down while you still remember them. exact menu path, exact chain order the vendor wants, which service restarts, what the failure looks like on screen, how long it takes. the person doing this in 47 days probably isn't you. tribal knowledge has a half-life of roughly one reorg.

all of it sits on top of the inventory problem, which is why the unknown bucket is worse than the technically hard ones. you cannot renew what you can't see. this is the boring reason certpulse has endpoint probes at all: a probe doesn't know or care whether a cert is automated, cloud-managed, or hand-pasted by a contractor in 2024. it connects, reads what's actually being served on the wire, and tells you when it dies. pair that against cloud inventory from acm, key vault, and certificate manager, and anything showing up in the probe results but in none of the cloud accounts is, by definition, your manual tail.

go find them now, while each one still only costs you twice a year.

the tail doesn't get shorter. the clock just gets faster.

-- alex