Backend Engineering · Engineering Foundry challenge level: Intermediate

Idempotent Background Job Processor

Specify a job processor that survives retries, worker crashes, and duplicate delivery without performing external effects twice.

All challengesSuggested practice time: 65 minutes

Original prompt

What you are building

Design the interfaces and persistent state for a background job processor that charges an external API and records the result. Queue delivery is at least once, workers may crash at any point, and the external API supports caller-provided idempotency keys.

Engineering Foundry originalNo company association

Deliverables

Job creation and worker interfaces.
Persistent job and attempt data model.
State-transition and concurrency rules.
Idempotency and external-effect strategy.
Retry, dead-letter, and operator recovery behavior.
Testing and observability plan.

Constraints

Assume at-least-once queue delivery.
Do not rely on a distributed transaction with the external API.
Manual replay must preserve an audit trail.
Two workers may receive the same job concurrently.

Success criteria

External idempotency and internal state transitions reinforce each other.
Concurrency control is explicit.
Crash points are traced before and after the external call.
Tests include duplicate delivery, lease expiry, and ambiguous responses.

Suggested workflow

List every crash boundary.
Define job and attempt states.
Choose claim/lease and concurrency semantics.
Connect stable idempotency keys to the external call.
Design recovery, tests, metrics, and operator tools.

Attempt before reveal

Build your deliverable independently.

Use the prompt, constraints, and workflow first. Guidance offers considerations and questions - not a canonical solution.

Reveal guidanceDirections and questions to test

Failure modes to explore

Trace a crash after the external effect succeeds but before local success is recorded.
Keep the logical job identity separate from individual delivery attempts.
Consider compare-and-set state transitions or leases with fencing behavior.

Reveal common mistakesFailure patterns worth checking

Assuming queue acknowledgement makes the external effect exactly once.
Generating a new idempotency key for every retry.
Using a lease without handling a stale worker.
Discarding attempt history when an operator replays work.

Reveal stretch goalsOptional ways to extend the deliverable

Add dependency graphs between jobs.
Support fair scheduling and per-tenant concurrency limits.

Optional solution worksheet

Prepare your approach for discussion.

Capture your reasoning, then copy a plain-text summary you control.

Session only - not saved.Refreshing or leaving this page clears every worksheet field and self-review choice. Nothing is uploaded or officially submitted.

Approach summary

Key decision

Important tradeoff

Failure mode or edge case

Validation or testing plan

Reflection

Optional solution URLThis link remains in the current browser session and is included only if you copy the summary.

Nothing is sent when you copy.

Optional community discussion

Compare reasoning, not rankings.

Discuss this challenge in the Engineering Foundry community if you choose. Share your own work, explain assumptions, critique ideas rather than people, and remove confidential or proprietary material.

Discuss this challenge

Loading Engineering Foundry…

Backend Engineering · Engineering Foundry challenge level: Intermediate

Idempotent Background Job Processor

Specify a job processor that survives retries, worker crashes, and duplicate delivery without performing external effects twice.

All challengesSuggested practice time: 65 minutes

Original prompt

What you are building

Engineering Foundry originalNo company association

Deliverables

Job creation and worker interfaces.
Persistent job and attempt data model.
State-transition and concurrency rules.
Idempotency and external-effect strategy.
Retry, dead-letter, and operator recovery behavior.
Testing and observability plan.

Constraints

Assume at-least-once queue delivery.
Do not rely on a distributed transaction with the external API.
Manual replay must preserve an audit trail.
Two workers may receive the same job concurrently.

Success criteria

External idempotency and internal state transitions reinforce each other.
Concurrency control is explicit.
Crash points are traced before and after the external call.
Tests include duplicate delivery, lease expiry, and ambiguous responses.

Suggested workflow

List every crash boundary.
Define job and attempt states.
Choose claim/lease and concurrency semantics.
Connect stable idempotency keys to the external call.
Design recovery, tests, metrics, and operator tools.

Attempt before reveal

Build your deliverable independently.

Use the prompt, constraints, and workflow first. Guidance offers considerations and questions - not a canonical solution.

Reveal guidanceDirections and questions to test

Failure modes to explore

Trace a crash after the external effect succeeds but before local success is recorded.
Keep the logical job identity separate from individual delivery attempts.
Consider compare-and-set state transitions or leases with fencing behavior.

Reveal common mistakesFailure patterns worth checking

Assuming queue acknowledgement makes the external effect exactly once.
Generating a new idempotency key for every retry.
Using a lease without handling a stale worker.
Discarding attempt history when an operator replays work.

Reveal stretch goalsOptional ways to extend the deliverable

Add dependency graphs between jobs.
Support fair scheduling and per-tenant concurrency limits.

Optional solution worksheet

Prepare your approach for discussion.

Capture your reasoning, then copy a plain-text summary you control.

Session only - not saved.Refreshing or leaving this page clears every worksheet field and self-review choice. Nothing is uploaded or officially submitted.

Approach summary

Key decision

Important tradeoff

Failure mode or edge case

Validation or testing plan

Reflection

Optional solution URLThis link remains in the current browser session and is included only if you copy the summary.

Nothing is sent when you copy.

Optional community discussion

Compare reasoning, not rankings.

Discuss this challenge

Idempotent Background Job Processor

What you are building

Deliverables

Constraints

Success criteria

Suggested workflow

Build your deliverable independently.

Failure modes to explore

Backend Engineering deliverable self-review

Prepare your approach for discussion.

Compare reasoning, not rankings.

Idempotent Background Job Processor

What you are building

Deliverables

Constraints

Success criteria

Suggested workflow

Build your deliverable independently.

Failure modes to explore

Backend Engineering deliverable self-review

Prepare your approach for discussion.

Compare reasoning, not rankings.