Model serving and inference

Models as reliable inference services: serving architecture, SageMaker, open-weight LLM engines (vLLM, SGLang), TensorRT-LLM and edge, and GPUs inside a customer's own environment.

This module covers how trained models become reliable inference services: the architecture of online and batch serving, the managed route through SageMaker, and the specialised runtimes used for large language models. It moves from general serving patterns to LLM engines and then to compilation, quantization and edge targets, where latency and cost are won. It ends with GPU infrastructure and model serving for inference workloads in infrastructure the customer owns.

Lessons (5)

• Model serving architecture: inference APIs for batch and real-time demand · Practice

  Model serving turns a versioned model artifact into an inference API with defined latency, throughput and availability.

• Managed deployment and serving with Amazon SageMaker · Practice

  Amazon SageMaker offers managed hosting in which AWS runs the instances, load balancing and health management while the team supplies a container and a model artifact.

• Serving open-weight LLMs with vLLM and SGLang · Practice

  Serving open-weight LLMs is a distinct discipline because generation is autoregressive: each request holds GPU memory for its whole lifetime and produces output token by token.

• Low-latency inference: TensorRT-LLM, quantization and edge deployment · Practice

  When latency or hardware budget is tight, the model itself is optimised for the target device instead of only scaling the service around it.

• GPU infrastructure and model serving for inference workloads

  When a customer cannot or will not call a hosted model API, the forward deployed engineer has to run open-weight models on GPUs inside the customer environment.

Changes

• Version 1, 2026-10-14: re-cut from the role pathway groups into a topic module (site 2026.10.2); no lesson changed