Skip to content
StackSide

How to Write an Incident Postmortem (with a Template)

StackSide

Share

Start from a blameless premise

The goal is to find why the system let the mistake reach users, not who made it. Judge each decision by what the person knew at the time, not by how it turned out.

Keep observations and decisions apart in the timeline

Put what was observed and what was decided in separate columns. When they are mixed, you cannot later tell whether a decision was wrong or the information was missing.

Root cause and detection gaps

Name one direct cause, then write down why it took so long to notice. Long incidents usually come from detection gaps, not from the original bug. The same discipline applies to planned changes: write down what you would have seen if the change had gone wrong.

Stripe · Mentioned

Selective Test Execution at Stripe: Fast CI for a 50M-line Ruby monorepo

Stack: Ruby, CI/CD, Bazel

Blast radius and recovery

Separate the blast radius into users, features, and data. Large batch systems fail in a different way: one bad change can stall many jobs at once, so count affected work items as well as users. Meta’s ingestion migration is a useful reference for scoping a change this large before it starts.

Meta · Migrated

Migrating Data Ingestion Systems at Meta Scale

Stack: MySQL, Scuba, Distributed Systems

Follow-up actions with owners

Every follow-up needs an owner and a done-when condition. “Improve monitoring” is not an action; “alert when the metric crosses this threshold, and reproduce the threshold in a test” is. GitHub’s staged CSS work shows the same idea of shrinking risk in steps.

GitHub · Removed

Improving site performance by shipping more CSS

Stack: CSS Modules, CSS-in-JS, TypeScript

Template (copy)

## 1. Summary

- What broke, and what users saw
- Which features and how many users were affected

## 2. Timeline

| Time | What we observed | What we decided and did |
|---|---|---|

## 3. Root cause

- Direct cause
- Why detection was late

## 4. Blast radius

- Users affected, data loss (yes or no), recovery steps

## 5. Follow-up actions

| Action | Owner | Done when |
|---|---|---|
Template
## 1. Summary

- What broke, and what users saw
- Which features and how many users were affected

## 2. Timeline

| Time | What we observed | What we decided and did |
|---|---|---|

## 3. Root cause

- Direct cause
- Why detection was late

## 4. Blast radius

- Users affected, data loss (yes or no), recovery steps

## 5. Follow-up actions

| Action | Owner | Done when |
|---|---|---|

Cases quoted in this guide are compiled automatically from public posts and videos and may contain errors. Check each original link. Read the disclaimer

Related topics

New cases, once a week

Get newly added topics and cases by email once a week.

Once a week. New topics and cases only. Unsubscribe anytime.

Sign in with Google to subscribe. We store your email, name and profile photo. You must be 14 or older. Privacy policy