Core concept · Intermediate

Batch, Streaming & Online Inference

Distinguish batch scoring, streaming updates, precomputation, online inference, and hybrid paths.

Content reviewed September 2026Read to the end

What you will learn

Distinguish batch scoring, streaming updates, precomputation, online inference, and hybrid paths.

Decision trigger

What freshness or context requirement rules out precomputation?

Interview impact

Inference mode changes cost, availability, event-time correctness, caching, and fallback.

Latency and freshness selector

1Decision cadence
2Feature freshness
3Compute mode
4Serving path
5Fallback

Mental model

Compute no later - and no earlier - than the decision requires.

A demand forecast is forced through request-time inference even though decisions update once per day.

How it works

Map decision deadline, feature freshness, traffic, compute cost, and cacheability.
Select batch, precomputed, streaming, online, or hybrid paths and define late-data behavior.

Product consequences

Unnecessary real-time paths add latency without user value.

Operational consequences

Streaming introduces event time, late data, state recovery, and replay; online inference introduces saturation and availability concerns.

Alternatives

Serve precomputed results when per-request context adds little value.
Use streaming aggregates plus a simple online policy instead of online model inference.

Failure modes

A batch job misses its publish deadline with no stale-mode policy.
Late events corrupt streaming features.

Worked example

Demand forecasts run daily in batch.
Fraud uses streaming velocity features and online scoring; recommendation precomputes candidates and ranks with request context.

Interviewer probes

What is the freshness requirement?
Could this decision be precomputed?

Expected depth by level

Entry: Distinguish batch and online.
Mid-level: Add streaming, hybrid design, and late-data behavior.
Senior+: Address regional placement, replay, cost, and migration between modes.
Role emphasis: SWE-ML emphasizes data paths; MLE balances freshness and quality.

System Design prerequisites

Event streaming Caching

Sources and review notes

These sources support the technical claims on this page. Product-specific examples remain Engineering Foundry teaching scenarios.

Rules of Machine LearningGoogle for Developers · first-party engineering/science · reviewed 2026-09-04
Baselines, instrumentation, launch sequencing, and production iteration guidance.
Hidden Technical Debt in Machine Learning SystemsGoogle Research · original paper · reviewed 2026-09-04
Feedback loops, data dependencies, configuration debt, and system-level ML risks.

Reach this point to mark the lesson as read.

Finish this lesson

Record completion when you have worked through the material. Completion is learning progress, not evidence of mastery.

Loading completion…

Previous: Feature Stores & Training-Serving Consistency

Loading Engineering Foundry…

Sources and review notes

These sources support the technical claims on this page. Product-specific examples remain Engineering Foundry teaching scenarios.

Rules of Machine LearningGoogle for Developers · first-party engineering/science · reviewed 2026-09-04

Baselines, instrumentation, launch sequencing, and production iteration guidance.

Hidden Technical Debt in Machine Learning SystemsGoogle Research · original paper · reviewed 2026-09-04

Feedback loops, data dependencies, configuration debt, and system-level ML risks.

Decision trigger

Interview impact

Mental model

How it works

Product consequences

Operational consequences

Alternatives

Failure modes

Worked example

Interviewer probes

Expected depth by level

System Design prerequisites

Practice this concept

Sources and review notes

Finish this lesson

Decision trigger

Interview impact

Mental model

How it works

Product consequences

Operational consequences

Alternatives

Failure modes

Worked example

Interviewer probes

Expected depth by level

System Design prerequisites

Practice this concept

Sources and review notes

Finish this lesson