Start with the decision#
Keep large bytes off the application request path. Separate resumable transfer, immutable object identity, asynchronous processing, and atomic result publication.
Mechanism and state flow#
- Authorize an upload intent and issue bounded object-store credentials or signed part operations.
- Upload chunks independently with checksums, size/type limits, and an explicit complete or abort step.
- Create a processing job only for a finalized immutable input version.
- Partition processing into idempotent chunks and stage results under the job version.
- Validate completeness, atomically publish the result pointer, and expire input, parts, and failed staging data by policy.
Concrete example#
Process a 40 GB video upload
The browser uploads numbered parts directly to object storage and completes with checksums. A job probes media metadata, creates segment tasks, writes renditions to a versioned prefix, validates the manifest, then switches the video's published pointer.
Architecture consequence: A failed segment retries independently, while viewers continue using the previous complete version and never see partial output.
Trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| Multipart direct upload | Files exceed a safe application-request size or networks are unreliable. | Orphaned parts, credential scope, checksums, and completion become explicit. |
| Parallel chunk processing | Chunks are independently transformable and merge order is known. | Skew and final assembly may dominate total time. |
| Single sequential worker | The format prevents safe splitting and volume is bounded. | Recovery repeats more work and one worker limits throughput. |
Failure diagnosis#
- Failure
- The publisher exposes the output prefix before every chunk finishes.
- Impact
- Readers observe missing segments or a mixture of job versions.
- Detection
- Manifest validation finds absent checksums while the public pointer already references the staging version.
- Mitigation
- Write immutable staged objects, validate the complete manifest, and conditionally swap one published pointer only after success.
- Trade-off
- Keeping old and staged versions consumes more temporary storage.
Design checkpoint#
Work the scenario before revealing the checklist
- Choose part size, maximum file size, checksum, and credential scope.
- Define the immutable input version and processing job identity.
- Draw chunk retry, cancellation, and cleanup paths.
- Specify the single atomic action that makes output visible.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- Who pays for abandoned multipart uploads?
- Can a file change after processing starts?
- How is malicious content isolated?
- What happens when one chunk is much slower than the rest?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- Uploading and copying objects using multipart uploadAWS Documentation
- Best practices for background jobsMicrosoft Azure Architecture Center
Sources reviewed 2026-09-04. Service limits and product-specific guarantees still require current verification.