The anatomy of an enterprise SSO outage

When SSO goes down, everything goes down, which is a sentence that sounds like vendor fearmongering until the Tuesday it happens to you. Nobody can reach email, the VPN, the payroll system, or, in one incident we assisted with this year, the ticketing system where the outage was supposed to be tracked. Authr sits in the authentication path for hundreds of enterprises now, north of 120 million authentications a month, which gives us an unusual dataset: we see how SSO fails across every IdP, every protocol, and every flavor of enterprise network. This post is the field guide.

The four horsemen, ranked by frequency

• Certificate expiry: the undisputed champion. A signing certificate somewhere in the federation quietly ages out, usually one the current team did not know existed, often on the partner's side.
• Clock skew: an IdP's NTP drifts, assertions arrive from 90 seconds in the future, and every relying party correctly rejects them. Correct behavior, total outage.
• Metadata drift: someone rotates a key, changes an endpoint, or re-provisions an IdP, and the other side of the federation finds out empirically.
• The upstream dependency nobody drew: the IdP is fine, but the LDAP directory, HSM, or database behind it is not, and the failure surfaces as mysterious intermittent login errors.

Why SSO outages are uniquely nasty

Two properties make these incidents worse than ordinary downtime. First, the blast radius is everything at once, including your remote workforce's only ways of talking to each other, which in 2022 means the whole company. Second, and less appreciated: the responders are locked out too. The on-call engineer cannot reach the dashboard, because the dashboard is behind the SSO that is down. We call this the locksmith problem, and every enterprise should have a documented, regularly tested break-glass path that does not depend on the system being repaired. Most discover they need one at the worst possible moment.

Here is a lightly sanitized log from a certificate-expiry incident we helped a customer diagnose in September. Note the timestamps: the certificate expired at midnight UTC, and the outage began at exactly 00:00:00. Cryptography is nothing if not punctual.

23:59:58 saml.verify ok    idp=partner-hub-prod assertions=214/min
00:00:00 saml.verify FAIL  idp=partner-hub-prod reason=cert_expired
         cert cn=sso-signing-2019 notAfter=2022-09-30T00:00:00Z
00:00:03 authr.alert page  rule=verify_fail_rate cell=us-e-1
00:00:41 authr.mitigate    grace_verify=on ttl=4h (prior cert, flagged)
00:19:12 idp cert rotated by customer; grace_verify=off

What we do about it, structurally

Every failure mode above is boring, which is excellent news, because boring failures can be engineered away. Authr monitors certificate expiry across every federation we touch and starts escalating 60 days out, to us and to the partner. We tolerate bounded clock skew and alert on drift trends before they cross the threshold. Metadata changes are detected and diffed automatically. And the mitigation in that log, accepting a just-expired certificate for a bounded window while screaming about it, is a deliberate product decision about which failure is worse at minute one of an outage. Across our 9 regions, this machinery now catches a would-be outage somewhere in our customer base roughly every week. It is the least glamorous engineering we do, and I would argue the most valuable.