Start with the decision#
Choose batch or streaming from the decision deadline, input boundedness, correction model, and cost - not from a desire to appear real time. A shared logical transform can still run under different operational SLOs.
Mechanism and state flow#
- Define event time, processing time, and the maximum useful result delay.
- Use a bounded source and finite run for batch; use durable offsets, watermarks, and windows for unbounded input.
- Choose how late data updates or retracts earlier results.
- Make sinks idempotent or transactional because workers and windows may retry.
- Retain replayable input and version transforms so corrected output can be rebuilt.
Concrete example#
Fraud signals and financial reconciliation
A streaming pipeline scores transactions within seconds and updates provisional features. A daily batch recomputes the ledger-derived truth, incorporates late events, and reports divergence from the online result.
Architecture consequence: Assign low latency to the provisional decision and exact reconciliation to the bounded authoritative run instead of pretending one path has both properties.
Trade-offs#
| Option | Choose when | Cost or limit |
|---|---|---|
| Batch | Input is bounded or the result may arrive on a schedule. | Freshness is limited by collection and run time. |
| Streaming | Each event must affect a decision within a tight deadline. | State, late data, backpressure, and continuous operations add complexity. |
| Shared model, separate pipelines | The transforms match but SLOs and resource isolation differ. | Outputs need explicit reconciliation and versioning. |
Failure diagnosis#
- Failure
- A late event arrives after a window was published and billed.
- Impact
- The aggregate disagrees with authoritative records or changes silently after consumers acted.
- Detection
- Late-data counters and batch reconciliation expose a versioned result difference.
- Mitigation
- Define allowed lateness, correction events, and a finalization policy; keep billing on a reconciled source if exactness is required.
- Trade-off
- Waiting longer improves completeness but delays final results.
Design checkpoint#
Work the scenario before revealing the checklist
- Name the business decision and its latest useful answer.
- Define event time, window, watermark, and late-data policy.
- Choose a replay source and output version key.
- Explain how a batch recomputation replaces or corrects streamed output.
Use these prompts to extend the design. The useful answer is a reasoned change, not a memorized sentence.
- Is the input truly unbounded?
- What happens to late or out-of-order events?
- Which output is authoritative?
- Can the sink tolerate a replay?
Related practice problems
Use the manifest-backed prompts below to apply this lesson in a complete design.
- Dataflow overviewGoogle Cloud Documentation
- Plan your Dataflow pipelineGoogle Cloud Documentation
Sources reviewed 2026-09-04. Service limits and product-specific guarantees still require current verification.