How to Write an Incident Postmortem (with a Template)
StackSide

Start from a blameless premise
The goal is to find why the system let the mistake reach users, not who made it. Judge each decision by what the person knew at the time, not by how it turned out.
Keep observations and decisions apart in the timeline
Put what was observed and what was decided in separate columns. When they are mixed, you cannot later tell whether a decision was wrong or the information was missing.
Root cause and detection gaps
Name one direct cause, then write down why it took so long to notice. Long incidents usually come from detection gaps, not from the original bug. The same discipline applies to planned changes: write down what you would have seen if the change had gone wrong.
Selective Test Execution at Stripe: Fast CI for a 50M-line Ruby monorepo
Stack: Ruby, CI/CD, Bazel
Blast radius and recovery
Separate the blast radius into users, features, and data. Large batch systems fail in a different way: one bad change can stall many jobs at once, so count affected work items as well as users. Meta’s ingestion migration is a useful reference for scoping a change this large before it starts.
Migrating Data Ingestion Systems at Meta Scale
Stack: MySQL, Scuba, Distributed Systems
Follow-up actions with owners
Every follow-up needs an owner and a done-when condition. “Improve monitoring” is not an action; “alert when the metric crosses this threshold, and reproduce the threshold in a test” is. GitHub’s staged CSS work shows the same idea of shrinking risk in steps.
Improving site performance by shipping more CSS
Stack: CSS Modules, CSS-in-JS, TypeScript
Template (copy)
## 1. Summary - What broke, and what users saw - Which features and how many users were affected ## 2. Timeline | Time | What we observed | What we decided and did | |---|---|---| ## 3. Root cause - Direct cause - Why detection was late ## 4. Blast radius - Users affected, data loss (yes or no), recovery steps ## 5. Follow-up actions | Action | Owner | Done when | |---|---|---|
## 1. Summary
- What broke, and what users saw
- Which features and how many users were affected
## 2. Timeline
| Time | What we observed | What we decided and did |
|---|---|---|
## 3. Root cause
- Direct cause
- Why detection was late
## 4. Blast radius
- Users affected, data loss (yes or no), recovery steps
## 5. Follow-up actions
| Action | Owner | Done when |
|---|---|---|
Cases quoted in this guide are compiled automatically from public posts and videos and may contain errors. Check each original link. Read the disclaimer
New cases, once a week
Get newly added topics and cases by email once a week.
Once a week. New topics and cases only. Unsubscribe anytime.
Sign in with Google to subscribe. We store your email, name and profile photo. You must be 14 or older. Privacy policy