Core concept · Intermediate

Training Pipelines, Reproducibility & Lineage

Design a rerunnable pipeline with explicit publish gates and lineage.

Content reviewed September 2026Read to the end

What you will learn

Design a rerunnable pipeline with explicit publish gates and lineage.

Decision trigger

What failure or audit question would the current lineage be unable to answer?

Interview impact

Pipeline design determines repeatability, safe backfills, failure recovery, and auditability.

Artifact lineage graph

1Data + code + config
2Validation
3Train
4Evaluate
5Versioned artifact

Mental model

A model artifact is the output of a versioned graph, not an isolated file.

A notebook retrains a ranker, overwrites a mutable dataset, and publishes an artifact no one can reproduce.

How it works

Version code, configuration, input snapshots, feature definitions, environment, evaluation, and output artifact.
Make stages idempotent, validate before publish, isolate backfills, and retain lineage from artifact to inputs.

Product consequences

Reliable reruns shorten recovery from bad data or model regressions.

Operational consequences

Retries, quotas, backfills, artifact retention, and partial failure need explicit behavior.

Alternatives

Keep a simple scripted pipeline when scale and team boundaries do not justify a platform.
Use event-driven training only when events represent real evidence, not every data fluctuation.

Failure modes

A retry duplicates side effects or publishes twice.
Mutable inputs prevent reproducing a released artifact.

Worked example

A daily ranker validates schemas, builds point-in-time examples, trains, evaluates against baseline and slices, then registers only accepted artifacts.
A failed publish leaves the current production version untouched.

Interviewer probes

Which stages are safe to retry?
What evidence blocks publication?

Expected depth by level

Entry: Describe a repeatable train/evaluate flow.
Mid-level: Add lineage, retries, validation, and backfills.
Senior+: Address multi-tenant compute, governance, disaster recovery, and ownership.
Role emphasis: SWE-ML owns orchestration; MLE connects model and platform semantics.

System Design prerequisites

Job schedulers Idempotency

Sources and review notes

These sources support the technical claims on this page. Product-specific examples remain Engineering Foundry teaching scenarios.

The ML Test ScoreGoogle Research · original paper · reviewed 2026-09-04
Production-readiness tests and monitoring across data, models, infrastructure, and integration.
Model Registry WorkflowsMLflow · official documentation · reviewed 2026-09-04
Versioned model artifacts, aliases, environment promotion, and lifecycle metadata.

Reach this point to mark the lesson as read.

Finish this lesson

Record completion when you have worked through the material. Completion is learning progress, not evidence of mastery.

Loading completion…

Previous: Offline Evaluation by ML Task

Loading Engineering Foundry…

Sources and review notes

These sources support the technical claims on this page. Product-specific examples remain Engineering Foundry teaching scenarios.

The ML Test ScoreGoogle Research · original paper · reviewed 2026-09-04

Production-readiness tests and monitoring across data, models, infrastructure, and integration.

Model Registry WorkflowsMLflow · official documentation · reviewed 2026-09-04

Versioned model artifacts, aliases, environment promotion, and lifecycle metadata.

Decision trigger

Interview impact

Mental model

How it works

Product consequences

Operational consequences

Alternatives

Failure modes

Worked example

Interviewer probes

Expected depth by level

System Design prerequisites

Practice this concept

Sources and review notes

Finish this lesson

Decision trigger

Interview impact

Mental model

How it works

Product consequences

Operational consequences

Alternatives

Failure modes

Worked example

Interviewer probes

Expected depth by level

System Design prerequisites

Practice this concept

Sources and review notes

Finish this lesson