Start with the decision#
Optimize first for bounded user impact and coordinated recovery; investigate deeply after the service is stable and turn learning into owned prevention work.
Mechanism and state flow#
- Detect an objective symptom and declare the incident early.
- Assign incident command, operations, and communication roles.
- Mitigate with the safest reversible action that reduces impact.
- Recover service and data, then validate user-visible health.
- Document timeline, contributing conditions, detection gaps, and action items without blame.
Concrete example#
Corrupt configuration rollout
Error rate rises after a config push. The incident commander freezes changes, operations rolls back the config, communications posts impact updates, and the team verifies error rate and queued work before closing.
Architecture consequence: The postmortem adds schema validation, a canary, rollback ownership, and a dated action owner instead of merely naming the bad config.
Trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| Rollback | A recent reversible change is the likely trigger. | May reintroduce the defect the change was meant to fix. |
| Fail over | A healthy independent region or replica exists. | Replication lag and capacity uncertainty. |
| Degrade | A noncritical dependency drives the incident. | Reduced product behavior and recovery reconciliation. |
Failure diagnosis#
- Failure
- Responders pursue root cause while impact keeps growing.
- Impact
- Recovery slows, communication fragments, and risky simultaneous changes compound the outage.
- Detection
- No incident commander, no current mitigation hypothesis, or time since last stakeholder update exceeds policy.
- Mitigation
- Separate command from operations, prefer one reversible mitigation at a time, maintain a timeline, and declare explicit verification criteria.
- Trade-off
- Structure adds process during small incidents but prevents chaos during large ones.
Design checkpoint#
Work the scenario before revealing the checklist
- Define the user-impact signal and declaration threshold.
- Assign command, operations, and communications roles.
- Choose a reversible first mitigation and rollback trigger.
- Write three measurable postmortem actions with owners and due dates.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- How do you distinguish mitigation from permanent remediation?
- What proves data recovery is complete?
- How do you keep a postmortem blameless but actionable?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- Incident Management GuideGoogle SRE
- Postmortem Culture: Learning from FailureGoogle SRE
- Continuous Archiving and Point-in-Time RecoveryPostgreSQL
Sources reviewed 2026-09-04. Validate product limits, legal obligations, and operational assumptions for the system you are designing.