Multi-perspective validation: why your ACME challenge fails from a network you can't see

your certificate renewal failed. you ssh into the box, curl the challenge url, get a 200 with the exact token the client wrote, and stare at it. from where you're sitting the validation path is fine. the CA disagrees, and it's not wrong — it just looked from somewhere you didn't.

this is the most common shape of ACME validation failure i see now, and it comes straight out of a change in how public CAs do domain validation. since march 2025, corroborating validation from multiple network perspectives is a baseline requirement, not a let's encrypt house rule. if you're debugging with a single curl from your own network, you're testing a fraction of what the CA tests.

why one vantage point stopped being enough

the attack is simple and someone actually did it, not just wrote a paper wondering. princeton researchers showed in 2018 that an adversary who can influence BGP routing for a short window can hijack the path to a target's IP space, answer an HTTP-01 or DNS-01 challenge from that hijacked position, and walk away with a legitimately-issued certificate for a domain they don't control. the hijack doesn't need to last long. it doesn't need to be globally visible either. it needs to fool the one network location the CA validated from, and stay quiet enough that nobody notices a brief route leak. sub-prefix hijacks make it worse — only a slice of the internet sees the bad route, and the CA happens to be sitting in that slice.

the defense is to check from several places at once. if an attacker has to hijack the path to your prefix as seen from four geographically and topologically distinct networks at the same time, the attack gets loud. let's encrypt shipped multi-perspective validation to production in february 2020 and published data showing it caught real routing anomalies in the wild. the CA/browser forum then generalized it: ballot SC-067v3 put multi-perspective issuance corroboration into the TLS baseline requirements, phasing in from march 2025 and tightening the quorum over time. more remote perspectives as it goes, a small allowance for how many may fail, and a minimum diversity requirement so a CA can't run all its "perspectives" in one cloud region and call it a day.

i'm deliberately not printing the current numbers, because they change on a schedule and a blog post is a snapshot. read BR section 3.2.2.9 directly. the shape is what matters operationally: your domain is being probed from multiple networks you have no relationship with, and one of them failing is enough to fail issuance. CAA lookups get corroborated the same way, so inconsistent CAA answers fail for the same reason.

what actually breaks, and it's almost never the CA

the primary perspective succeeding while a remote one fails isn't a bug. that asymmetry is the entire point of the design, and it's telling you something true about your infrastructure: some networks can reach your validation path and others can't. here's where that comes from, roughly in order of how often i've watched it happen.

geo-blocking is the big one. somebody added a rule to drop traffic from regions the business doesn't serve. sounds reasonable, especially after an incident. but the CA's remote perspectives sit in cloud regions picked for diversity, not for overlap with your customer base. european company blocking APAC and south america? you've statistically guaranteed at least one perspective gets a connection reset. your primary perspective, probably closer to home, is fine.

bot management and WAF managed rules come next. cloudflare's bot fight mode is the classic. an ACME client isn't a browser, the validation request arrives from an ASN your bot scoring has never seen hitting your site, and the CA gets a 403 or a javascript interstitial instead of the token. it fails intermittently because scoring is probabilistic and runs per-perspective. same story with rate limiting: several perspectives requesting the identical path within a second or two of each other, from unrelated networks, looks quite a lot like a scan, and a per-region rate limit will happily block the second and third one.

then there's ipv6 that half exists. you have AAAA records. boulder prefers ipv6 and falls back to v4 only under specific failure conditions. if your ipv6 path works from some networks and blackholes from others — a broken transit route, a firewall rule that was only ever written for v4, an ALB with v6 listeners in exactly one AZ — the perspectives reaching you over v6 fail while the ones that fall back succeed. this one wastes the most time, because everything you test from your laptop works.

anycast DNS that hasn't converged is quieter. DNS-01 writes a TXT record and the client immediately tells the CA to go check. your provider's anycast fleet hasn't finished distributing that record to every node yet. the primary perspective queries a node that has it; a remote perspective in another region queries one that doesn't. route 53 is honest about this: a change batch sits at PENDING until it reports INSYNC, and any decent ACME client polls for it. plenty of providers offer no such signal, and their API returning 200 means "we accepted your write", not "the record is live everywhere".

split-horizon resolvers do the same thing from the inside. your internal view answers one way, the world answers another. you verify with dig on a jump host, get the record back, and never learn the external authoritative answer is stale.

how the failure reads

boulder is more helpful here than it gets credit for. when the primary perspective succeeds and a remote one doesn't, the problem detail is prefixed with "During secondary validation:" followed by the underlying error. that prefix is the whole diagnosis. your server answered the CA's primary check correctly, and a different network couldn't reproduce it. stop debugging your web server. start debugging what sits in front of it, and what your DNS looks like from somewhere else.

the trap is that these failures are usually intermittent. retry and it works, because a different set of perspectives got lucky, or the DNS record finally landed, or the bot score came out differently that time. that reads as client flakiness, so somebody wraps the renewal in a retry loop, the alert stops firing, and everyone moves on. the reachability asymmetry is still there. now it's just invisible.

that retry loop also spends budget you probably didn't think about. let's encrypt limits failed validations to five per account, per hostname, per hour. a loop hammering a validation path with a persistent problem will exhaust that and start getting "too many failed authorizations recently" — which looks far more alarming than the original error, shows up after you've already burned an hour, and points you at rate limits instead of at the actual cause.

debugging it without guessing

query your authoritative nameservers directly, not through a caching resolver. dig each NS by name with recursion disabled, one at a time, and compare the answers. a caching resolver will happily hand you a record that only one of your four nameservers actually has.

then check from outside your own network path. public resolvers in different regions, RIPE atlas one-off measurements if you want real geographic spread, a DNS propagation checker if you want fast and approximate. you're looking for disagreement, not presence. let's debug is worth knowing about: it runs a batch of checks against a hostname the way boulder would, including authoritative nameserver consistency and ipv6 reachability, and it'll tell you about problems your own curl can't see.

drop your TXT record TTLs before you start so propagation timing is predictable instead of a mystery, and make sure your ACME client actually waits for propagation rather than sleeping a fixed number of seconds and hoping. when you're picking a DNS provider for DNS-01, propagation consistency matters more than how pleasant the API is. a provider whose records land everywhere in five seconds and tells you when they have is worth more at renewal time than one with a nicer SDK.

for HTTP-01, audit every layer in front of the challenge path and assume each one is guilty until cleared: geo rules, bot management, IP allowlists, rate limiting, managed WAF rulesets, and whatever "block unusual user agents" toggle somebody flipped during a scraping incident. every one of them needs an explicit exemption for the acme-challenge path, scoped to that path and nothing else. it serves opaque tokens and nothing else, so exempting it costs you nothing. exempting the whole well-known directory, or the whole host, costs you plenty.

and test in staging. let's encrypt's staging environment does multi-perspective validation exactly the way production does, with rate limits generous enough to iterate against.

why this gets worse on the 47-day clock

a validation path that fails one attempt in twenty is a curiosity when you renew annually. you retry, it works, you forget. under the 200-day maximum already in effect you're renewing a few times a year. at 100 days from march 2027 and 47 days from march 2029, you're looking at roughly eight renewals per certificate per year, and that same one-in-twenty failure rate turns into a recurring incident across an estate of a few hundred certs. multiply by every cert sitting behind the same WAF policy and it stops being intermittent at all.

the monitoring signal isn't the ACME client's exit code — the retry loop already swallowed that. it's renewal failures recurring on the same hostnames, and gaps in certificate transparency issuance for domains you expect to see reissued on a cadence. if a cert should have renewed nine days ago and no new CT entry has appeared for it, something upstream is failing quietly. that absence is what certpulse is built to notice: renewal-failure alerting on the cloud side, CT log watching on the public side, so a validation problem shows up as a signal instead of as an outage six weeks later.

fix the asymmetry instead of papering over it with retries. if some part of the internet can't reach your validation path, that's worth knowing on its own, certificates or not.