Review the important failures, not every imaginable one#
- Failure
- Cache cluster unavailable
- Impact
- Database demand may jump far beyond safe capacity
- Detection
- Cache error rate rises while database load and tail latency follow
- Mitigation
- Serve bounded stale local data, rate-limit misses, and shed optional work
- Trade-off
- Users may see stale results and some uncached requests fail
| Failure | Reasoning path |
|---|---|
| Recommendation unavailable | Remove optional module; preserve core response |
| Replica lags | State the consistency and session behavior |
| Broker unavailable | Decide whether the request can commit; consider the outbox only when requirements fit |
| Region disappears | Route, capacity, replication lag, sessions, RPO/RTO |
For each: normal behavior → failure or ambiguity → impact → detection → containment → recovery → new trade-off. Prioritize failures that threaten the core product or cause cascading load.
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- Addressing cascading failuresGoogle SRE
- Reliability design patternsMicrosoft Azure Architecture Center
Reach this point to mark the lesson as read.