Start with the decision#
Use a durable workflow when a business outcome spans multiple independently failing steps. Decide which transitions are reversible, which need compensation, and which require human resolution.
Mechanism and state flow#
- Persist workflow state, input identity, and the next permitted transition before dispatching work.
- Execute one idempotent local transaction per step and record its outcome.
- Retry transient failures within a bounded policy; compensate prior reversible steps when the outcome cannot complete.
- Treat compensation as new work that can fail, not as an automatic database rollback.
- Expose stuck state, operator action, audit history, and terminal business meaning.
Concrete example#
Reserve, charge, and fulfill an order
Inventory reservation commits locally, payment uses an idempotency key, and fulfillment begins only after charge. If fulfillment becomes impossible, the workflow requests a refund and releases stock, recording each result.
Architecture consequence: The user sees pending, completed, or needs-attention truth while the system retains enough state to resume without replaying successful effects.
Trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| Orchestration | One owner must enforce a visible sequence and recovery policy. | The coordinator becomes critical workflow infrastructure. |
| Choreography | Independent reactions are loosely coupled and no single sequence owns the outcome. | Global state and failure diagnosis become harder. |
| Distributed transaction | Participants truly support one bounded atomic protocol and availability cost is acceptable. | Coupling and blocking often make external services a poor fit. |
Failure diagnosis#
- Failure
- Payment succeeds, fulfillment fails, and the refund request times out.
- Impact
- The customer is charged without delivery and an automatic retry may duplicate the refund.
- Detection
- Workflow age and a nonterminal compensation state breach their objective.
- Mitigation
- Use the original payment identity for idempotent refund, retry with bounds, and route exhausted cases to an owned reconciliation queue.
- Trade-off
- Human recovery preserves correctness but increases operating cost and completion time.
Design checkpoint#
Work the scenario before revealing the checklist
- List every state and legal transition for an order.
- Mark each side effect as idempotent, compensatable, irreversible, or manual.
- Define timeout and retry policy separately for each step.
- Create an operator view for workflows past their expected age.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- Who owns the end-to-end outcome?
- Can compensation itself fail?
- What does the customer see between steps?
- How do deploys and duplicate messages affect the state machine?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- Saga distributed transactions patternMicrosoft Azure Architecture Center
- Making retries safe with idempotent APIsAmazon Builders' Library
Sources reviewed 2026-09-04. Service limits and product-specific guarantees still require current verification.