Start with the decision#
A long-running job must survive process restarts, deployment, cancellation, and partial progress. Model it as persisted state with checkpoints rather than one extended function invocation.
Mechanism and state flow#
- Create a durable operation with an immutable input reference and explicit pending state.
- Divide work into restartable stages or chunks and checkpoint committed progress.
- Lease one stage at a time, heartbeat while useful work continues, and reclaim expired leases.
- Treat cancellation as a requested transition that workers observe at safe boundaries.
- Publish terminal output atomically with the succeeded state and retain enough evidence to diagnose or resume failure.
Concrete example#
Re-index a tenant's search corpus
A coordinator snapshots the target version and creates shard tasks. Workers checkpoint completed ranges into a new index. Validation compares counts and samples before one atomic alias switch publishes the result.
Architecture consequence: Keep the old serving index until the complete new version is validated, making rollback a pointer change rather than another rebuild.
Trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| Fine checkpoints | Work is expensive and replay must be small. | Checkpoint I/O and state transitions increase. |
| Coarse checkpoints | Stages are cheap and naturally idempotent. | Failures repeat more work. |
| One long lease | Work cannot expose a safe restart boundary. | Recovery is slow and false worker death is hard to distinguish. |
Failure diagnosis#
- Failure
- A deployment kills a worker after it wrote output but before recording the checkpoint.
- Impact
- The replacement repeats the chunk and may corrupt or duplicate output.
- Detection
- Output contains duplicate chunk identity or the checkpoint trails committed artifact versions.
- Mitigation
- Write deterministic chunk keys and make checkpoint publication conditional on the expected job revision.
- Trade-off
- Deterministic staging consumes temporary storage and metadata.
Design checkpoint#
Work the scenario before revealing the checklist
- Draw all job states, legal transitions, and terminal states.
- Choose checkpoint boundaries and prove every stage is restartable.
- Define cancellation behavior during each stage.
- Explain how output becomes visible and how an old version is retained.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- What happens during a deployment?
- How do you distinguish a slow worker from a dead one?
- Can progress move backward?
- Where is the point of no return for cancellation?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- Asynchronous Request-Reply patternMicrosoft Azure Architecture Center
- Best practices for background jobsMicrosoft Azure Architecture Center
Sources reviewed 2026-09-04. Service limits and product-specific guarantees still require current verification.