How we cut p99 token latency to 38ms

At the start of this year, our p99 token issuance latency was 142ms. Today it is 38ms, measured at the edge across all 14 regions, and it has held there for six weeks under a traffic load that recently crossed 600 million authentications a month. This is the story of the difference, told in the order we found the problems, which is never the order you expect.

Why the p99 and not the median

Our median was already 19ms, and if we quoted that number this post would not exist. But an authentication sits in front of everything a user does, and a single login can fan out into a dozen token operations across a session. At that fan-out, one slow operation in a hundred means a large fraction of user sessions eat at least one stall. Tail latency is not a statistical curiosity; it is the latency your unluckiest users experience every single day, and enterprises are made entirely of users who file tickets.

The hunt, in three findings

Finding one was signing queue convoy. Token signing ran through per-region HSM pools, and under burst load, requests queued behind slow outliers. The fix was hedging: if a signing request has not returned in 12ms, fire a second to a different HSM and take whichever answers first. Duplicate work on 2 percent of requests bought back 40ms at the tail.

Finding two was cold policy caches after deploys. Every rollout emptied the edge caches that hold tenant signing configuration, so the p99 spiked on a schedule that exactly matched our release train. We felt appropriately silly about this one. Caches are now warmed from a snapshot before a node takes traffic.

Finding three, the shameful one, was a mutex around metrics. Our instrumentation library serialized histogram updates behind a single lock, so under load, the code that measured latency was manufacturing it. The trace that caught it is framed on the wall in the Dublin office.

trace 8f3a12 (token-mint, eu-west) — before / after
  parse+validate      1.1ms   1.1ms
  policy fetch       31.4ms   2.0ms   (cache warm at boot)
  sign (HSM)         88.2ms  14.3ms   (hedged @12ms)
  metrics record     18.9ms   0.1ms   (lock-free histograms)
  total p99         142ms     38ms

Holding the line

Latency work that is not defended is a loan, not an asset. Every one of these fixes now ships with a regression gate: the deploy pipeline runs a production-shadow load test per region, and a p99 over 45ms blocks the train, no exceptions, no overrides in the release tooling. The gate has fired four times since June. All four would have been slow-burn regressions we caught in a war room three months later. I will take the blocked deploys.