Start with the decision#
Change a live data contract in stages: expand compatibility, move and verify data, switch readers, then contract only after old writers are gone.
Mechanism and state flow#
- Add backward-compatible schema or a target store.
- Make new writes compatible with both representations and observe failures.
- Backfill historical rows in bounded, restartable batches while capturing concurrent changes.
- Compare counts, checksums, invariants, and lag before switching reads.
- Stop old writers, keep a rollback window, then remove the old representation.
Concrete example#
Split a display name
Add given_name and family_name while retaining display_name. New code writes both forms, a checkpointed worker backfills old rows, and shadow reads compare reconstructed names before the new columns become authoritative.
Architecture consequence: Every deploy remains reversible until measured parity and old-client retirement make contraction safe.
Trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| Expand/contract | Application versions overlap and downtime is unacceptable. | Temporary dual representation and longer operational ownership. |
| Maintenance cutover | The dataset is small and an explicit outage is acceptable. | Simple implementation but direct availability impact. |
| New store plus CDC | The storage engine or access model changes. | Lag, reconciliation, and dual-system failure modes. |
Failure diagnosis#
- Failure
- Backfill writes stale values after a live update.
- Impact
- The target passes row-count checks but contains older business state.
- Detection
- Version mismatches, invariant comparison, CDC lag, and sampled source-target checksums.
- Mitigation
- Use version-aware upserts or capture a change-log position, make batches idempotent, and verify after catch-up.
- Trade-off
- Extra metadata and write coordination buy an auditable cutover.
Design checkpoint#
Work the scenario before revealing the checklist
- Name every simultaneously deployed reader and writer.
- Choose a restartable cursor and per-row idempotency rule.
- Define parity metrics and a cutover threshold.
- Write rollback steps for each phase, not only the final deploy.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- How do you prevent a backfill from overwriting a newer write?
- When can the old column or store actually be removed?
- What happens when change capture falls behind?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
Sources reviewed 2026-09-04. Validate product limits, legal obligations, and operational assumptions for the system you are designing.