Your Kubernetes control plane has certificates too, and nothing is watching them

There's a kind of outage that only happens on clusters nobody has touched in a while. Workloads healthy. Ingress serving traffic. Every synthetic check green. And then kubectl says this:

Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2026-08-06T09:14:22Z is after 2026-07-31T11:02:07Z

If you drew the other half of the problem, you get error: You must be logged in to the server (Unauthorized) instead. That's an expired client certificate, and it will walk you through RBAC bindings and OIDC config for twenty minutes before it occurs to you to check a date.

Either way, the cluster is running and you can't administer it. Every certificate involved was created by kubeadm, lives for one year, and was never in anyone's inventory.

What kubeadm actually puts on disk

kubeadm init gives you three certificate authorities and about a dozen leaf certificates. The CAs are good for ten years. Everything they sign is good for one.

The CAs sit in /etc/kubernetes/pki: the cluster CA (ca.crt, ca.key), the front-proxy CA for the aggregation layer, and the etcd CA in the etcd subdirectory. Three independent trust roots, which is the right call. A compromised metrics-server aggregation path has no business reaching etcd. It's also three independent things you'll have to rotate someday.

The leaves, all one year:

• apiserver: the API server's serving certificate, with SANs for the service IP, the node name, kubernetes.default.svc.cluster.local, and your control plane endpoint
• apiserver-kubelet-client: how the API server authenticates to kubelets for logs and exec
• apiserver-etcd-client: how the API server authenticates to etcd. This is the one that takes the whole control plane down
• front-proxy-client: the aggregation layer's client identity
• etcd-server, etcd-peer, etcd-healthcheck-client: etcd's serving cert, its peer cert for the raft mesh, and the client cert kubeadm's own etcd tooling uses

The kubeconfigs live in /etc/kubernetes with the client certificate embedded straight into the YAML as base64: controller-manager.conf, scheduler.conf, admin.conf, and on 1.29 and later, super-admin.conf. Same one-year clock. That embedding is what makes them easy to miss. Nothing on the filesystem looks like a certificate, so a naive find for .crt files walks right past four identities.

And admin.conf has a problem the others don't. It isn't really a file on the control plane node. It's the file three engineers copied into their laptop's kube directory sometime in 2023. One of them has since left the company. One of them dropped it in a shared 1Password vault under a name nobody remembers. Renewing admin.conf on the node does nothing to any of those copies.

One thing on that list never expires: the service account key pair, sa.key and sa.pub. No certificate, no notAfter, no clock. Also no casual rotation path, because rotating it invalidates every service account token signed with the old key.

Why frequent upgraders never see this

Both kubeadm upgrade apply and kubeadm upgrade node renew every managed certificate as a side effect. The --certificate-renewal flag defaults to true. Upgrade the control plane and the one-year clock quietly resets to a fresh year.

So if you upgrade quarterly, this problem does not exist for you. You've never seen it, you don't have a runbook for it, and you won't think about it. The certificates renew themselves and you get the credit.

The teams that get hit are the ones doing everything else right by their own lights. They pinned a Kubernetes version because an operator wasn't compatible yet. They froze the platform during a migration. They inherited a cluster running a payments workload nobody wants to breathe on. Twelve months of "we'll upgrade next quarter," and a renewal mechanism they never knew they depended on stops firing.

The failure is gentle at first, which is the other half of the trap. Data plane traffic keeps flowing, because kube-proxy's iptables and nftables rules are already programmed, ingress controllers serve from cached state, and kubelets keep containers running against local state. Nothing pages. Meanwhile controller-manager and scheduler crashloop, nothing new schedules anywhere, HPAs freeze, and you have a cluster that is technically serving your customers while being completely unmanageable. If your alerting is all black-box HTTP probes, you find out when someone tries to deploy. Those probes aren't monitoring this. They're monitoring the one part that still works.

Checking, and then actually renewing

The command is kubeadm certs check-expiration, run on a control plane node. It reads the certificate files on disk and the certificates embedded inside the kubeconfigs, then prints a table: name, expiry timestamp, residual time, signing CA, and whether the cert is externally managed. A second table covers the three CAs.

That externally-managed column matters. If you're running an external CA and the signing key isn't on disk, kubeadm can tell you when things expire but can't do anything about it, and you get to go find whatever process issued them originally.

Renewal is kubeadm certs renew, either per-certificate (apiserver, apiserver-etcd-client, admin.conf, and so on) or with the all subcommand. It needs the relevant CA key present in /etc/kubernetes/pki. It rewrites the files and re-embeds fresh certs into the kubeconfigs.

And then nothing changes, because the running processes are still holding the old material in memory.

Recent API server versions do watch and reload their serving certificate from disk. Don't build a procedure on that. The API server's client certificates for etcd and the kubelet, plus the kubeconfigs that controller-manager and scheduler authenticate with, all need a process restart. Restarting the kubelet isn't reliably enough, because the kubelet only recreates a static pod when the manifest itself changes, and renewal doesn't touch the manifests. What actually works is moving the files out of /etc/kubernetes/manifests, waiting for the kubelet to tear the pods down, and moving them back. Ugly, and it's the documented move. On multi-node control planes, one node at a time, and confirm etcd has quorum before you touch the next one.

Then go update the human copies of admin.conf. kubeadm kubeconfig user will generate a fresh kubeconfig for a given username and group, which is what you should have been doing instead of handing out admin.conf in the first place. admin.conf is cluster-admin with no expiry story and no revocation story. Stop giving it to people.

The kubelet does its own thing, and does it worse

Kubelet certificates run on an entirely different system, and this is where a small misconfiguration becomes a staggered outage spread over days.

At join time, the kubelet uses a bootstrap token to submit a CSR against the kube-apiserver-client-kubelet signer. The controller-manager's approving controller auto-approves it, because kubeadm creates the RBAC for exactly that. The kubelet writes the issued cert to /var/lib/kubelet/pki as a timestamped pem file plus a kubelet-client-current.pem symlink pointing at it, and the kubelet's kubeconfig references the symlink rather than embedding anything. That indirection is the entire rotation mechanism. Somewhere around 70 to 90 percent of the certificate's lifetime, the kubelet requests a new one, writes a new timestamped file, and repoints the symlink.

Two things break it.

First: clusters created before kubeadm 1.17 have a kubelet.conf with an embedded client certificate instead of the pointer. Rotation happens on disk and the kubelet keeps authenticating with the frozen copy until that copy expires. Newer check-expiration builds warn about this by name, and an upgrade fixes it, which is cold comfort for the cluster that isn't getting upgraded. If check-expiration mentions kubelet.conf at all, you have it.

Second, and this is the common one: the auto-approval RBAC gets deleted by a hardening pass, or the controller-manager is already down for its own certificate reasons, and nothing approves the renewal CSRs. Look at the shape of that failure. Each node's certificate expires on its own join anniversary, so nodes drop to NotReady one or two at a time over days or weeks. It looks exactly like flaky hardware. Teams chase kernel logs and NIC firmware for a week before somebody thinks to run kubectl get csr.

Serving certificates are a separate, opt-in thing. By default the kubelet self-signs its HTTPS serving cert, which is the entire reason half the world runs metrics-server with TLS verification turned off. Set serverTLSBootstrap and the kubelet requests a real one from the kubelet-serving signer instead. The controller-manager will not auto-approve those, deliberately, because it can't validate the SANs being requested. So you need an approver: a cloud provider controller, something like kubelet-csr-approver, or a human running kubectl certificate approve. When that approver quietly dies, serving certs expire and kubectl logs and kubectl exec start throwing x509 errors while the node sits there Ready, because node status is reported over the client cert. Same error text, completely different cause, and easy to confuse at 2am.

So: kubectl get csr, look for Pending, note the requesting node identity and the signer name. One gotcha. Pending CSRs are garbage collected after 24 hours, so a nightly check can find an empty list while kubelets are still failing and re-requesting on their own schedule. Alert on the count over a window, not on a point-in-time snapshot.

Treat it as inventory

The fix isn't a better renewal procedure. It's treating control plane PKI as inventory you monitor, instead of something upgrades quietly handle on your behalf.

Emit metrics. Run check-expiration on a schedule, turn residual time into a gauge, done. A systemd timer writing into node_exporter's textfile collector is enough and takes an afternoon. If you'd rather deploy something, enix's x509-certificate-exporter runs as a DaemonSet and reads both the pki directory and the kubeconfigs, embedded certs included. The kubelet already exposes kubelet_certificate_manager_client_ttl_seconds and renewal error counters. The API server exposes a histogram of remaining lifetime on client certs presented at authentication time, which is a good way to turn up stale identities you had no idea were still connecting.

Set thresholds that assume you won't upgrade in time. Warn at 90 days, warn louder at 45, page at 14. If a scheduled upgrade renews everything before any of those fire, the alerts stay silent and cost you nothing.

Extend the defaults where it makes sense. ClusterConfiguration v1beta4 added certificateValidityPeriod and caCertificateValidityPeriod, so you can issue longer-lived leaves on clusters you know won't get upgraded often. It's a real security tradeoff, and a two-year certificate on an unmonitored cluster just moves the outage to 2028. Set it on purpose. Monitor it either way.

Write the CA rotation plan before you need it. The ten-year clock is real. kubeadm went GA in Kubernetes 1.13 in December 2018, which means a whole generation of clusters shares a birthday range and will hit CA expiry somewhere between 2027 and 2029. There is no kubeadm certs renew for a CA. The upstream manual procedure means distributing a new CA to every node, running a trust bundle containing both CAs through the transition, restarting everything, and reckoning with service account tokens. That's a maintenance window and a written plan, not an afternoon. Find your CA expiry dates now, while now is still years of margin instead of weeks.

One honest note, since we sell certificate monitoring: CertPulse will not find any of this. Our endpoint probes need a listener they can reach, and our cloud inventory integrations enumerate ACM, Key Vault, and GCP Certificate Manager. Your etcd peer certificate lives in none of those places and answers on none of those ports. It's the right tool for certificates terminating public traffic and the ones your cloud provider issued, and the wrong tool for internal cluster PKI. That gap belongs to your metrics pipeline, and closing it is a couple of hours of work you should schedule before the anniversary you haven't looked up yet.

Go run check-expiration on your oldest cluster. Thirty seconds, and you'll learn something you didn't want to know.