Zero-downtime key rotation, explained

Signing key rotation has a reputation as a risky ceremony, the kind of thing teams schedule for a quiet Sunday and announce in advance, as if apologizing. Ours runs every 30 days per cell, fully automated, under full production traffic, and has for two years. Nobody schedules it. Most of the team could not tell you when it last happened, which is the entire point. Here is how it works, including the failure we designed around.

The problem is not signing, it is verifying

Rotating the key you sign with is trivial: generate, swap, done. The hard part is the thousands of relying services across our customers that verify those tokens, every one of them caching our JWKS document on its own schedule, with its own bugs. Some refresh hourly. Some refresh daily. Some, and I say this with affection, fetched the keys once at boot in 2021 and are still going. If you cut over signing the moment a new key exists, every verifier with a stale cache starts rejecting valid tokens, and you have converted your key hygiene into your customers' outage.

Overlap is the whole algorithm

The fix is embarrassingly simple to state: never require anyone to know about a key any sooner than the slowest plausible cache refresh. Every rotation is a staged overlap.

// rotation phases per cell (durations from measured cache behavior)
const ROTATION = [
  { phase: "publish",  action: "new key in JWKS, old key still signs", holdHours: 48 },
  { phase: "cutover",  action: "new key signs; old key verifies only", holdHours: 0 },
  { phase: "grace",    action: "old key remains in JWKS for stale caches", holdHours: 72 },
  { phase: "retire",   action: "old key removed, shards destroyed, event logged", holdHours: 0 },
] as const;

Publish for 48 hours before the new key signs anything. Sign, while the old key remains published for another 72. Only then retire it, with destruction of the key shards written to Audit Fabric like everything else. The durations are not folklore; we measure real JWKS fetch intervals across relying parties and set the windows past the observed p99.9. When we find a verifier outside even that window, our support team gets a proactive alert with the customer's name on it before their users notice a thing.

The failure we actually had

In 2022, before this design fully hardened, a rotation in one region exposed a wonderful bug: a customer's API gateway cached our JWKS correctly but keyed the cache on the wrong field, so it treated the updated document as identical and never picked up the new key. Tokens started failing at exactly the cutover minute. The lesson was not to fix that gateway, though we helped. The lesson was that we own the consequences of every verifier's bugs, deserved or not. That incident is why the grace window exists, why cutover is per-cell rather than global, and why rotation halts automatically if verification failure rates twitch by even a basis point.

Rotation frequency is a security control, but the deeper argument for automating it is Nassim Taleb with a pager: a rotation you run monthly is a rotation you can run in an emergency, at 2 a.m., during a suspected compromise, without a war room. We have done that once. It took 11 minutes, and it looked exactly like the boring monthly one. That is what the two years of engineering were for.