Postmortem culture at Authr

We sell five nines. When your product is the front door to a thousand companies, an incident review is not a process artifact, it is the mechanism by which the SLA stays true next year. So we take postmortems unreasonably seriously, and this post describes how, including the parts that took us years to get right.

Every near-miss is an incident

The rule that shapes everything else: we write postmortems for events with zero customer impact. If an error budget burned faster than forecast, if a failover worked but took longer than rehearsed, if an engineer caught a bad config in the last review before rollout, that gets the same treatment as an outage. Last quarter we wrote 31 postmortems. Two involved measurable customer impact. The other 29 are why there were only two.

Near-misses are cheaper teachers than outages, but only if you treat them with the same respect. A culture that only reviews visible failures is optimizing for luck.

The document is a conversation, not a verdict

Every postmortem at Authr follows the same skeleton, and the skeleton is deliberately hostile to hand-waving. The phrase human error is banned as a root cause; if a person made a mistake, the document must explain what made the mistake easy to make and hard to catch. Here is the template header, unchanged since 2023.

PM-2025-0114  |  severity: SEV-3 (near-miss)  |  status: review
impact: none external; cell ap-se-2 error budget -22% in 18 min
trigger: config push enabled strict SAML audience check on legacy IdP pool
detection: burn-rate alert (4m) — NOT the canary (gap, see AI-3)
actions: 6 items, 2 blocking next config-push train
blameless check: passed — no names in causal chain

Note the detection line. We always record what caught the problem versus what should have. The gap between those two is, in my experience, the highest-value output of the entire exercise.

Action items with teeth

The industry failure mode is a beautiful postmortem whose action items die in the backlog. Our fix is structural: blocking action items gate the systems they name. If a postmortem marks an item as blocking the config-push train, the train literally does not depart until the item ships. Product roadmap does not get a vote. This costs us real feature velocity a few times a year, visibly and painfully, and it is the single strongest signal leadership can send about what we actually value.

Twice a year we also run the aggregate review: every postmortem from the trailing six months goes on one wall, and we look for rhyming causes. That review is where cell admission control came from, and where our current investment in configuration canaries was born. Individual incidents teach tactics. The aggregate teaches strategy.

None of this works without safety. Engineers here volunteer their own mistakes in writing because eight years of history says the document will be about the system, not the person, and because the sharpest postmortem authors keep getting promoted. Culture is just the accumulated record of what got rewarded.