Start with the decision#
Recompute derived state from an authoritative source with a versioned, restartable process that coexists with live writes and proves completeness before cutover.
Mechanism and state flow#
- Name the authoritative source and the target derivation version.
- Capture a change-log position or make target writes version-aware.
- Scan stable key ranges in bounded idempotent batches.
- Apply concurrent changes without letting older backfill output win.
- Compare invariants, lag, and samples; switch readers; retain a rollback window.
Concrete example#
Rebuild a search index
Create index_v2 with a new analyzer. Bulk index records by primary-key range while CDC applies newer versions. Compare document counts, sampled queries, and maximum lag before moving a small reader cohort.
Architecture consequence: An alias switch is reversible, and a stale bulk record cannot overwrite a newer CDC update.
Trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| In-place repair | The transformation is small and individually reversible. | Mixed old and new state is harder to reason about. |
| Parallel target | Validation and instant rollback matter. | Temporary duplicate storage and writes. |
| Offline rebuild | Readers can tolerate unavailability or stale snapshots. | Simpler consistency but explicit downtime or freshness loss. |
Failure diagnosis#
- Failure
- A restarted worker repeats a range and writes an older projection over a live update.
- Impact
- The rebuilt target silently regresses individual entities.
- Detection
- Target version is below source version, checksum mismatches, or shadow queries diverge.
- Mitigation
- Checkpoint ranges, attach source versions, use conditional upserts, and make every batch replay-safe.
- Trade-off
- Version checks add write cost but make retries and parallelism safe.
Design checkpoint#
Work the scenario before revealing the checklist
- Name the source of truth and target derivation version.
- Choose stable range boundaries and checkpoint state.
- Define how live changes overtake snapshot work.
- Write cutover, abort, and rollback thresholds.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- What if the source row changes twice during its batch?
- How do deletes reach the rebuilt target?
- Which parity check catches semantically wrong but well-formed output?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
Sources reviewed 2026-09-04. Validate product limits, legal obligations, and operational assumptions for the system you are designing.