incident-postmortem skill produces a blameless post-incident review that transforms a production incident from a painful failure into a permanent improvement in system reliability. The core principle is that incidents are caused by systemic failures, not individual mistakes. Attacking people instead of systems destroys the psychological safety that enables honest postmortems — and an honest postmortem is the only one worth writing. This is a reactive skill only: it is invoked after a real production incident (or a near-miss), never scheduled upfront.
Type: Component · Estimated time: 1–2 hours · Standards: Google SRE blameless post-mortem culture; Five Whys RCA · Output file:
.engineering-docs/20-incident-postmortem-[INC-ID].md · Reactive only — never plan this document in advance. Invoke it after a real production incident or a near-miss that reveals a system gap.Best for
Production incidents
After any production incident affecting users or breaching an SLA — from a full payment outage (SEV-1) to a 5-minute performance degradation (SEV-4).
Near-miss reviews
After any near-miss caught before user impact — a bad deployment stopped in staging, an alert that self-resolved, a backup gap found during a drill.
Reliability culture
Building an organizational practice of continuous reliability improvement where every incident produces durable improvements, not just a fix.
Compliance reporting
Satisfying compliance or contractual obligations for incident reporting, with documented root cause, impact quantification, and remediation actions.
What it produces
The generated postmortem covers the full lifecycle of an incident from detection through systemic remediation:How to invoke it
Pass the incident name or ID and a description of what happened. The richer the context — timeline points, monitoring data, what was done to mitigate — the more complete and accurate the first draft will be.Example scenarios
Payment processing outage
Payment processing outage
Write a post-mortem for last night's payment processing outageProduces a SEV-2 postmortem with a timestamped timeline from alert to all-clear, Five Whys analysis tracing from connection pool exhaustion back to a missing index in staging, and action items for connection utilization alerting, slow query logging, and staging data volume parity.Database connection exhaustion
Database connection exhaustion
Document the root cause of the database connection exhaustion incidentProduces a precise root cause analysis showing why the query was not caught in testing, why monitoring did not alert before the pool was fully exhausted, and why staging did not replicate production data volume — with the deployment control gate that would have caught it.API timeout affecting a subset of users
API timeout affecting a subset of users
Create a blameless review of the API timeout that affected 15% of usersProduces a SEV-3 postmortem with quantified impact (users affected, requests failed), a timeline from first timeout to mitigation, and root causes classified for trending: observability gap (no p99 alert at warning threshold) and architecture gap (no circuit breaker on the slow downstream dependency).Key concepts
Blameless culture
People do not cause incidents. Systems do. When an engineer makes a mistake that leads to an incident, the real questions are:- Why did the system allow that mistake to have this impact?
- What alerting, testing, or deployment control would have prevented it?
- What process would have caught this before production?
Five Whys analysis
The skill enforces at least four layers of “Why?” past the proximate cause. The proximate cause is almost always “a bad change was deployed.” The real causes are deeper:
Each root cause is classified into a standard category for quarterly trending:
Testing gap · Observability gap · Deployment control gap · Architecture gap · Process gap · Knowledge gap · Third-party dependency.
Severity classification
The postmortem includes a structured decision tree that classifies severity objectively — before the full impact is known — and supports re-classification as more information becomes available:Incident Commander role
For SEV-1 and SEV-2 incidents, an Incident Commander (IC) is designated. The IC owns the incident response — coordination, communication, go/no-go decisions — without performing the technical debugging. The postmortem documents IC designation time, responsibilities checklist, and any IC handoffs (shift change, fatigue) with status briefing summaries.Action items with teeth
Every action item must have three things to be valid: a named owner (a person, not a team), a due date, and a measurable success criterion that proves it is done. “Improve monitoring” is not an action item. “Add DB connection utilization alert at 80% threshold and verify it fires in staging test by YYYY-MM-DD, owned by [Name]” is an action item. Action items are tracked in a system that supports deadlines and assignees (Jira, Linear, GitHub Issues — not a wiki table), reviewed weekly by the postmortem author, and escalated to engineering management if overdue by more than two weeks.Near-miss postmortems
Near-misses are incidents caught before user impact. They deserve postmortems because a near-miss today is a real incident tomorrow if the underlying gap is not fixed. The format is identical to a full postmortem; the Impact Assessment documents potential impact rather than actual impact. Organizations that only write postmortems for real incidents inadvertently incentivize under-reporting.Statistical tracking across postmortems
The postmortem includes a statistical analysis section for quarterly trending: MTTD, MTTR, root cause category, and action item completion rate. These metrics, aggregated across incidents, reveal systemic patterns that no single postmortem can surface on its own.Interview process
The agent asks targeted questions before drafting, first loading prior.engineering-docs/ files to extract the known tech stack, SLO targets, and runbook procedures:
1
Context loading
Reads existing documents (technical-runbook, SLO document, deployment-plan) to understand the service’s alert thresholds, on-call procedures, and any prior related incidents.
2
Symptom and detection (≤ 2–3 questions)
Confirms how the incident was first noticed (automated alert, customer support ticket, internal observation) and through which channel.
3
Timeline points
Collects key timestamps: incident start, detection, first mitigation, full resolution, and any re-classification events.
4
Failure impact
Quantifies the business impact — number of failed requests, affected users, revenue estimate, SLA breach status — in concrete terms, not approximations.
5
Document generation
Writes the timeline, performs Five Whys, quantifies impact, classifies root causes, and generates specific action items with owners.
6
Revision and sharing
Incorporates feedback; checks cross-document consistency (action items must align with runbook procedures and system architecture); shares with the team within 24–48 hours of resolution.
Output structure
The generated.engineering-docs/20-incident-postmortem-[INC-ID].md mirrors the full template:
20-incident-postmortem-INC-[ID].md (structure)
Handoff
Reads from
17-technical-runbook.md— operational procedures followed (or not) during the incident19-slo-error-budget.md— SLO targets to measure impact and classify breach severity16-deployment-plan.md— deployment that may have triggered the incident- Monitoring data — timeline, metrics, log excerpts
Feeds into
17-technical-runbook.md— updated response procedures based on lessons learned15-test-strategy.md— new test cases to prevent recurrence of root causes7-system-architecture.md— architecture changes to reduce blast radius18-disaster-recovery.md— DR gaps discovered during the incident
Quality gate
Before marking the postmortemfinal, verify every item in this checklist:
- The timeline has precise timestamps for every event from first symptom to all-clear
- The Five Whys analysis goes at least 4 layers deep past the proximate cause
- Every action item has a named owner, due date, and measurable success criterion
- The document contains zero blameful language targeting individuals
- Impact is quantified with concrete metrics (users affected, transactions failed, revenue impact)
- Severity is classified using the decision tree and justified
- Incident Commander is designated (for SEV-1/SEV-2) and their responsibilities are documented
- Action items are tracked in a system with deadlines and a weekly review cadence
- Root cause categories are assigned for statistical trending
- Near-miss incidents have a postmortem if they reveal system gaps
- Incident response tooling is documented (PagerDuty/OpsGenie/incident.io configuration)
Timing: Publish the postmortem draft within 24–48 hours of incident resolution while the timeline is fresh. Hold the postmortem review meeting within 48–72 hours. Action items enter the team’s next sprint planning cycle immediately.