Core concept · Intermediate

Online Experimentation & Causal Validation

Know when an online experiment is appropriate and what can invalidate it.

Content reviewed September 2026Read to the end

What you will learn

Know when an online experiment is appropriate and what can invalidate it.

Decision trigger

What integrity or guardrail signal would stop the experiment before significance?

Interview impact

Experiment design connects product metrics, rollout safety, interference, and decision limits.

Experiment integrity path

1Eligible unit
2Random assignment
3Exposure
4Guardrail checks
5Causal analysis

Mental model

An experiment is a causal design with integrity checks, not a traffic split plus a dashboard.

A feed test randomizes requests even though users appear repeatedly and creators influence both treatment groups.

How it works

Predeclare hypothesis, primary metric, guardrails, randomization unit, eligibility, exposure, ramp, duration, and stop rules.
Check assignment and sample-ratio mismatch before interpreting effects; examine interference and long-term limits.

Product consequences

A valid experiment can reveal that offline gains do not improve user outcomes.

Operational consequences

Assignment, exposure logging, isolation, guardrails, and ramp controls must remain reliable throughout the test.

Alternatives

Use shadow or replay for technical validation without user impact.
Use quasi-experimental or staged observational evidence only with explicit causal limitations.

Failure modes

Sample-ratio mismatch invalidates the result.
Network effects violate independent treatment assumptions.

Worked example

Feed ranking randomizes by user, logs exposure, watches quality and latency guardrails, and ramps only after integrity checks.
Creator ecosystem effects are treated as an interference risk, not ignored.

Interviewer probes

What is the randomization and analysis unit?
How do you detect invalid assignment or interference?

Expected depth by level

Entry: State treatment, control, and one guardrail.
Mid-level: Add unit, exposure, SRM, ramp, and analysis integrity.
Senior+: Handle interference, long-term effects, organizational overlap, and decision governance.
Role emphasis: Applied Scientist deepens inference; SWE-ML owns trustworthy delivery and logging.

System Design prerequisites

Consistent hashing

Sources and review notes

These sources support the technical claims on this page. Product-specific examples remain Engineering Foundry teaching scenarios.

Diagnosing Sample Ratio Mismatch in Online Controlled ExperimentsMicrosoft Research · original paper · reviewed 2026-09-04
Experiment integrity, assignment checks, and sample-ratio mismatch diagnosis.
Patterns of Trustworthy Experimentation: Pre-Experiment StageMicrosoft Research · first-party engineering/science · reviewed 2026-09-04
Randomization units, guardrails, network effects, and pre-experiment validity checks.

Reach this point to mark the lesson as read.

Finish this lesson

Record completion when you have worked through the material. Completion is learning progress, not evidence of mastery.

Loading completion…

Previous: Model Registry, Deployment & Rollback

Loading Engineering Foundry…

Expected depth by level

Entry

State treatment, control, and one guardrail.

Mid-level

Add unit, exposure, SRM, ramp, and analysis integrity.

Senior+

Handle interference, long-term effects, organizational overlap, and decision governance.

Role emphasis

Applied Scientist deepens inference; SWE-ML owns trustworthy delivery and logging.

Sources and review notes

These sources support the technical claims on this page. Product-specific examples remain Engineering Foundry teaching scenarios.

Diagnosing Sample Ratio Mismatch in Online Controlled ExperimentsMicrosoft Research · original paper · reviewed 2026-09-04

Experiment integrity, assignment checks, and sample-ratio mismatch diagnosis.

Patterns of Trustworthy Experimentation: Pre-Experiment StageMicrosoft Research · first-party engineering/science · reviewed 2026-09-04

Randomization units, guardrails, network effects, and pre-experiment validity checks.

Decision trigger

Interview impact

Mental model

How it works

Product consequences

Operational consequences

Alternatives

Failure modes

Worked example

Interviewer probes

Expected depth by level

System Design prerequisites

Practice this concept

Sources and review notes

Finish this lesson

Decision trigger

Interview impact

Mental model

How it works

Product consequences

Operational consequences

Alternatives

Failure modes

Worked example

Interviewer probes

Expected depth by level

System Design prerequisites

Practice this concept

Sources and review notes

Finish this lesson