A healthy caller can depend on a partially broken system#
Service B stops returning, but it has not crashed. Service A keeps sockets and workers occupied, requests accumulate, and the healthy API eventually becomes unavailable too. Distributed failures include crashes, delay, dropped packets, duplicate delivery, stale replicas, overloaded dependencies, unavailable zones, and network partitions.
Ask harder questions than “is it up?” What if B finishes but its response is lost? What if the database commits but its acknowledgement disappears? What if A reaches the database while B cannot? Each observer may have a different, incomplete view.
- Can it fail or become slow?
- Can requests be duplicated or responses be lost?
- Can data become stale?
- Can this dependency become overloaded?
- What is the fallback?
- What protects the rest of the system?
- How does it recover?
- Failure
- Service B stops responding
- Impact
- Service A exhausts its request workers
- Detection
- Dependency latency and deadline errors rise
- Mitigation
- Bound the call, isolate resources, and return an intentional fallback
- Trade-off
- The response may be incomplete or stale
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- Addressing cascading failuresGoogle SRE
- Reliability design patternsMicrosoft Azure Architecture Center
Reach this point to mark the lesson as read.