Operating in production

After it ships: observability (platform and agent traces), model monitoring and drift, SLOs and on-call, debugging and postmortems, security and compliance, autoscaling and cost.

This module covers what happens after a system ships. It starts with seeing what the system does, through logs, metrics and traces with OpenTelemetry and Prometheus and through agent traces, and with knowing when a model degrades. It then covers keeping it reliable with SLOs, on-call runbooks, debugging and postmortems, keeping it secure and compliant, including in regulated industries, and paying a sensible price for it through autoscaling and cost efficiency.

Lessons (8)

• Observability: logs, metrics and traces with OpenTelemetry and Prometheus · Practice

  Observability is the ability to explain a system's internal state from the telemetry it emits, without shipping new code to ask a new question.

• Observability: logs, metrics and agent traces · Practice

  Observability is the ability to understand what a running system is doing from the telemetry it emits: logs, metrics and traces.

• Model monitoring: data quality, drift and alerting · Practice

  A model can be up, fast and error-free while its predictions get worse, because the data it sees has moved away from the data it was trained on.

• Reliability practice: SLOs, on-call runbooks and postmortems

  Reliability practice turns the vague goal of a stable service into numbers and routines: service level objectives that define good enough, alerts tied to those objectives, an on-call rotation with runbooks, and postmortems that turn incidents into fixes.

• Security, identity and access, and compliance for ML platforms · Practice

  An ML platform concentrates valuable things: training data, model weights, expensive compute and credentials to many systems.

• Autoscaling and cost efficiency: right-sizing, spot mix and GPU utilization · Practice

  ML workloads are expensive and bursty, so capacity has to follow demand at two levels: the number of pods and the number and type of nodes under them.

• Debugging production systems and running postmortems

  Debugging production systems is a disciplined loop of triage, observation, hypothesis and test, not a hunt driven by intuition.

• Security reviews, governance and compliance in regulated industries · Practice

  Regulated customers will not put an AI system into production until it passes their security and compliance review.

Changes

• Version 1, 2026-10-14: re-cut from the role pathway groups into a topic module (site 2026.10.2); no lesson changed