Start with the decision#
Separate namespace and block metadata from bulk data transfer, then design placement, replication, and recovery around large sequential files rather than small transactional objects.
Mechanism and state flow#
- A metadata service maps paths and files to immutable or append-oriented blocks.
- Clients request block locations, then transfer bytes directly with data nodes.
- Data nodes heartbeat and report block inventory.
- Placement spreads replicas across failure domains while considering locality and write cost.
- The controller re-replicates under-replicated blocks and rebalances capacity.
Concrete example#
One terabyte analytics file
The file is split into large blocks. Metadata returns nearby healthy replicas, readers stream blocks in parallel, and a failed data node triggers re-replication from surviving copies without moving user bytes through the namespace service.
Architecture consequence: Metadata capacity, small-file pressure, rack placement, and recovery bandwidth become explicit design constraints.
Trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| Large blocks | Files are large and scans dominate. | Poor fit for many tiny files and fine-grained updates. |
| Replication | Fast recovery and simple reads matter. | Multiplicative storage and network writes. |
| Erasure coding | Cold durable data dominates cost. | Encoding, repair, and degraded-read complexity. |
Failure diagnosis#
- Failure
- A rack fails while many blocks have replicas concentrated in it.
- Impact
- Data becomes unavailable or recovery saturates cross-rack bandwidth.
- Detection
- Heartbeats disappear, block reports age, and under-replicated block count rises.
- Mitigation
- Place across failure domains, reserve recovery bandwidth, prioritize rare blocks, and throttle re-replication storms.
- Trade-off
- Cross-rack placement improves fault tolerance but increases write traffic.
Design checkpoint#
Work the scenario before revealing the checklist
- Choose block size and expected file-size distribution.
- Define metadata durability and failover.
- Place three replicas across two or more failure domains.
- Estimate recovery bandwidth after losing one rack.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- Why should bulk data bypass the metadata service?
- What makes small files expensive?
- How do you prevent recovery traffic from harming foreground reads?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- HDFS ArchitectureApache Hadoop
- The Google File SystemGoogle Research
Sources reviewed 2026-09-04. Validate product limits, legal obligations, and operational assumptions for the system you are designing.