Incident-Postmortem

# Incident Post-Mortem

Guide a team through writing a structured, blameless post-mortem after a production incident. The output is a document that builds shared understanding, identifies root causes without blame, and produces concrete action items to prevent recurrence.

Blameless Principle

Systems fail, not people. The goal is to understand HOW the incident happened — not WHO caused it. Avoid language like "X forgot to", "Y should have known". Use "the system did not", "the process lacked", "the alert did not fire".

When to Use

- Production outage or service degradation has been resolved

- A significant near-miss occurred (would have been an incident if caught later)

- User-facing errors, data loss, or SLA breach happened

- Team wants to capture learnings before context fades

**Not for:** Minor bugs caught in staging, planned maintenance windows, or incidents with no learning value.

Input Requirements

Gather these details before writing the post-mortem. Ask for anything missing:

Incident Metadata

- Incident title (short, descriptive)

- Date and time of detection (with timezone)

- Date and time of resolution

- Severity / impact level (P1–P4 or equivalent)

- Incident commander / on-call owner

Impact

- Affected services and systems

- User-facing impact (errors, slowness, full outage)

- Estimated number of users affected

- Data loss or corruption (yes/no, scope)

- SLA/SLO breach (yes/no, by how much)

Timeline Events

Key moments to reconstruct:

- First symptom occurred

- Alert fired (or was noticed manually)

- On-call paged / incident declared

- Investigation started

- Root cause identified

- Mitigation applied

- Full resolution confirmed

- Customer communication sent (if any)

Contributing Factors

Ask the team: "What made this worse than it needed to be?" — not "who failed". Examples:

- Alert threshold too high / alert didn't fire

- Runbook was missing or outdated

- Deploy lacked a feature flag for rollback

- Monitoring didn't cover this failure mode

- On-call handoff missed context

Process

Step 1 — Gather Metadata

If the user has not provided full incident details, ask for them section by section. Don't proceed to writing until you have: title, times, severity, affected services, and at least a rough timeline.

Step 2 — Reconstruct Timeline

Work with the user to build a precise chronological timeline. For each event:

- Exact time (UTC preferred)

- What happened (system event or human action)

- Who observed it or took the action

- Link to log / alert / Slack message if available

Flag gaps: "We don't know what happened between 14:32 and 14:47 — worth checking logs."

Step 3 — Root Cause Analysis

Use the **5 Whys** iteratively:

Why did users see 500 errors?
→ The API pods were crash-looping.

Why were they crash-looping?
→ Memory limit was exceeded.

Why was the limit exceeded?
→ A new query was loading full result sets into memory.

Why wasn't this caught before deploy?
→ Load tests only covered the p50 case, not high-cardinality accounts.

Why did load tests only cover p50?
→ We had no test fixtures for large accounts.

Stop when you reach a system/process gap you can fix. The last "why" should point to an action item.

Distinguish:

- **Root cause** — the deepest systemic gap (one or two)

- **Contributing factors** — conditions that made it worse but aren't the root cause

Step 4 — Impact Quantification

Help the user be precise:

- Duration: detection to resolution (not symptom start to resolution — separate these)

- Error rate at peak vs. normal baseline

- Percentage of traffic affected

- Revenue / business impact if known

Step 5 — Action Items

For each root cause and contributing factor, generate at least one action item:

| # | Action | Owner | Due Date | Priority |

|---|--------|-------|----------|----------|

| 1 | Add load test fixtures for accounts > 10k records | @eng-team | 2026-07-01 | High |

| 2 | Lower memory alert threshold from 90% to 75% | @platf