Start with the decision#
For a write-heavy system, define the durable record and the permissible delay before derived state. The safest high-throughput path is often a small append followed by asynchronous, replayable materialization.
Mechanism and state flow#
- Validate and deduplicate at admission, then append the smallest durable event or command record.
- Partition by a stable high-cardinality key and bound any tenant or key that can dominate it.
- Process derived indexes, aggregates, and notifications with idempotent consumers.
- Compact, retain, or archive append-only data under a stated replay and audit policy.
- Track acceptance rate, durability lag, backlog age, hot partitions, and materialization freshness.
Concrete example#
Device telemetry ingestion
Gateways batch measurements, the service partitions by device hash and time bucket, and a durable log records accepted data. Stream processors update live alerts; batch compaction creates efficient historical files.
Architecture consequence: Keep raw durable input available for replay while treating live and historical views as replaceable projections.
Trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| Append-only log | Writes are naturally ordered facts and replay has value. | Retention, compaction, schema evolution, and duplicate processing are permanent responsibilities. |
| In-place update | A current value with tight per-key consistency is the product requirement. | Hot records, locks, and indexes become throughput limits. |
| Client or gateway batch | Small writes can tolerate a bounded collection delay. | Partial loss and retry semantics move closer to the client. |
Failure diagnosis#
- Failure
- A slow downstream indexer causes an unbounded durable backlog.
- Impact
- Freshness degrades, storage cost rises, and eventual recovery can overload the index.
- Detection
- Oldest-message age grows even when accepted write rate is stable.
- Mitigation
- Apply admission policy, isolate consumer pools, scale from backlog age, and replay in bounded partitions.
- Trade-off
- Rejecting or sampling low-value writes protects recovery but loses accepted volume.
Design checkpoint#
Work the scenario before revealing the checklist
- Define the minimum durable record and acknowledgment boundary.
- Choose partition and ordering scope for one device or tenant.
- List every derived view and prove its consumer is idempotent.
- Set backlog and replay limits before estimating worker count.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- Can the source of truth be replayed?
- What compacts old writes?
- Where does backpressure reach the caller?
- How do schema changes coexist in the log?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- Designing partition keys to distribute workloadsAWS Documentation
- Competing Consumers patternMicrosoft Azure Architecture Center
Sources reviewed 2026-09-04. Service limits and product-specific guarantees still require current verification.