ML Inference / Model Serving

Model serving is a latency and capacity system#

Online inference returns a prediction inside the request budget - for example fraud scoring. Async inference accepts a job and publishes a result later - for example document processing. Training is outside this lesson.

Capacity depends on model size, load time, CPU/GPU memory, request shapes, batching, and latency targets; avoid invented throughput constants. Dynamic batching can improve accelerator utilization but waits to form a batch, so it can increase individual request latency.

Version routes enable canary evaluation and rollback. Record latency, errors, saturation, and product-quality metrics by version. Autoscaling must account for expensive startup and loaded-model capacity, not request count alone.