Start with the decision#
Move work to a background job when the caller does not need its completed effect before receiving a truthful response. Define durable acceptance, result state, retries, and ownership before removing the work from the request.
Mechanism and state flow#
- Validate authorization and input synchronously, then commit a job record or message with an idempotency identity.
- Return an accepted state that names what has and has not completed.
- Lease work to bounded consumers and renew or redeliver it when a worker disappears.
- Make side effects idempotent, classify retryable failures, and quarantine exhausted jobs.
- Persist terminal state and expose progress or notification only when the product needs it.
Concrete example#
Generate a monthly account export
The API validates the owner and creates an export job, then returns its identifier. A worker reads a consistent snapshot, writes the encrypted archive, records expiry, and marks the job complete. The UI polls a status resource.
Architecture consequence: Treat the export and its downloadable artifact as a stateful product workflow, not a fire-and-forget function call.
Trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| Inline work | Completion is fast, bounded, and required for the response. | The request inherits dependency latency and retry uncertainty. |
| Queued job | Durable acceptance may precede completion. | Status, duplicates, cancellation, backlog, and cleanup need design. |
| Scheduled sweep | Work can be discovered periodically from authoritative state. | Discovery delay and repeated scans trade simplicity for freshness. |
Failure diagnosis#
- Failure
- A worker sends email and crashes before acknowledging the job.
- Impact
- Redelivery can send the same message twice.
- Detection
- One job identity appears in repeated attempts with an external side effect before terminal state.
- Mitigation
- Use an idempotency key at the provider boundary or an effect ledger that atomically records the claim.
- Trade-off
- Effect deduplication adds storage and may require provider cooperation.
Design checkpoint#
Work the scenario before revealing the checklist
- Write the API response for accepted, running, failed, and completed states.
- Choose the job identity, lease duration, retry classes, and dead-letter owner.
- List every side effect and its deduplication boundary.
- Set a backlog-age objective and overload response.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- What is guaranteed when the API returns?
- Who retries after a worker crash?
- Can the user cancel after work starts?
- How are abandoned results expired?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- Best practices for background jobsMicrosoft Azure Architecture Center
- Competing Consumers patternMicrosoft Azure Architecture Center
Sources reviewed 2026-09-04. Service limits and product-specific guarantees still require current verification.