- Baseline
- The simplest credible rule, heuristic, domain method, or model that a more complex design must beat.
- Calibration
- Agreement between a predicted probability and observed outcome frequency for the relevant population and time.
- Canary
- A release step that sends a bounded share of real production traffic to a new version and can affect users.
- Counterfactual
- The outcome that would have occurred under a different action; usually not directly observed for the same event.
- Data drift
- A change in input data or feature distribution; it is a signal to diagnose, not automatic proof that retraining is needed.
- Embedding
- A learned vector representation optimized for a task; 2D diagrams are projections, not literal production geometry.
- Exposure
- A record that an item, action, or treatment was actually shown or applied, including relevant position and policy context.
- Feature freshness
- How old a feature value may be before it no longer satisfies the decision's quality or safety requirement.
- Guardrail
- A measure or hard constraint that can block or reverse an optimization-driven launch.
- Label maturation
- The policy and time window after which an observed outcome is trustworthy enough to use as a label.
- Leakage
- Information unavailable at the historical decision time entering training or evaluation and inflating offline results.
- NDCG@K
- A top-K ranking measure that gives more credit when relevant results appear earlier; relevance grades and K must be defined.
- Point-in-time join
- A historical join that returns only feature values available at or before each example's event time.
- Proxy objective
- A measurable target used in place of the desired product outcome; its mismatch and guardrails must remain explicit.
- Recall@K
- The share of relevant candidates retrieved within the first K results; often used before final ranking evaluation.
- Reranking
- A later, smaller-candidate ordering or policy stage that can use richer signals or constraints.
- Sample-ratio mismatch
- A statistically unexpected difference between intended and observed experiment allocation that can indicate invalid data or assignment.
- Shadow deployment
- Running a new version on copied production inputs without letting its output affect the user-visible decision.
- Training-serving skew
- A semantic or distribution mismatch between feature computation during training and production inference.
- Uncertainty / abstention
- Representing insufficient confidence or evidence and routing the decision to a safer fallback or review path.