Mental model#
An alert is a request for timely human action. Page on a user-visible symptom or imminent exhaustion with a documented response; use tickets and dashboards for conditions that do not require interruption.
Mechanism and state flow#
- Choose the service-level symptom and the urgency of the required response.
- Measure the condition over windows that reject short noise but detect harmful burn quickly.
- Route to the owning team with severity, affected scope, links, and a tested runbook.
- Deduplicate related notifications and define escalation when acknowledgement or mitigation does not occur.
- Review pages for actionability, false positives, missed incidents, and operator load.
Concrete example#
Page on fast error-budget burn
A checkout service pages when failures are consuming the monthly error budget at a rate that will exhaust it quickly, confirmed over both a short and longer window. One transient error spike does not page.
Architecture consequence: Tie paging urgency to user harm and budget burn rather than a raw CPU threshold.
Scaling and operational trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| Symptom alert | Users are affected or an SLO is at risk. | Diagnosis begins after the alert and requires linked evidence. |
| Cause alert | A resource condition reliably predicts imminent harm and has one owner action. | Infrastructure thresholds often create noise without user impact. |
| Ticket | Action is needed but not immediately. | The condition may wait until business hours. |
Failure diagnosis#
- Failure
- Every instance pages independently during a regional dependency outage.
- Impact
- Operators receive hundreds of notifications and cannot see the common symptom.
- Detection
- Alert volume and shared dependency errors spike across services at once.
- Mitigation
- Aggregate by service and region, inhibit derivative pages, and preserve one incident with linked affected components.
- Trade-off
- Aggressive grouping can hide a distinct simultaneous failure.
Checklist exercise#
Work the scenario before revealing your answer
- Classify five candidate conditions as page, ticket, dashboard, or discard.
- For each page, name the user symptom, response deadline, owner action, and stop condition.
- Write a runbook first step that can be completed without guessing dashboard names.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- What action should the page recipient take now?
- Why is this threshold tied to user harm?
- How do you detect both fast and slow burns?
- What inhibits downstream alert storms?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- Alerting on SLOsGoogle SRE Workbook
- Monitoring distributed systemsGoogle SRE
Sources reviewed 2026-09-04. Product-specific controls and threat assumptions still require current verification.