Reliability practice: SLOs, on-call runbooks and postmortems

Generated by Claude through GigAI 0.1.11.10 on 2026-10-12.

Concept: Reliability · also covers On-call, Site reliability engineering, Production systems · Step 30 of 35 in MLOps / ML platform engineer · version 1

On this page

Reliability practice turns the vague goal of a stable service into numbers and routines: service level objectives that define good enough, alerts tied to those objectives, an on-call rotation with runbooks, and postmortems that turn incidents into fixes. For ML platforms the same method applies, with objectives extended to cover freshness and model quality as well as availability and latency.

Read more

How it is done in practice

A platform team writes SLOs per service tier in a short document agreed with the teams that depend on it, encodes the indicators as recording rules, and generates burn-rate alerts from them so nobody hand-writes thresholds. Pages go to a rotation through an incident tool, with a secondary and an escalation path, and each alert links to a runbook kept in the same repository as the service. A weekly review looks at pages received, budget remaining and open postmortem actions. When a budget is exhausted, the agreed policy applies: feature rollouts pause and reliability work takes priority until the service is back inside its objective.

Technologies

PrometheusAlertmanagerGrafanaPagerDutyOpsgenieSlothPyrraOpenSLO

Diagram

flowchart LR
  S[SLI: good events over valid events] --> O[SLO target over 30 days]
  O --> B[Error budget]
  B --> R[Burn rate alerts]
  R -->|fast burn| P[Page on-call]
  R -->|slow burn| T[Ticket]
  P --> M[Mitigate with runbook]
  M --> PM[Blameless postmortem]
  PM --> A[Tracked action items]
  A --> S
flowchart LR
  S[SLI: good events over valid events] --> O[SLO target over 30 days]
  O --> B[Error budget]
  B --> R[Burn rate alerts]
  R -->|fast burn| P[Page on-call]
  R -->|slow burn| T[Ticket]
  P --> M[Mitigate with runbook]
  M --> PM[Blameless postmortem]
  PM --> A[Tracked action items]
  A --> S

Code

yaml
# 99.9 percent SLO, budget = 0.001
- alert: InferenceErrorBudgetFastBurn
  expr: |
    (
      sum(rate(inference_requests_total{outcome="error"}[1h]))
      / sum(rate(inference_requests_total[1h])) > 14.4 * 0.001
    ) and (
      sum(rate(inference_requests_total{outcome="error"}[5m]))
      / sum(rate(inference_requests_total[5m])) > 14.4 * 0.001
    )
  labels:
    severity: page
  annotations:
    summary: "2 percent of monthly budget burned in 1h"
    runbook: "runbooks/inference-errors"

Subtopics

SLIs, SLOs and error budgets

Definitions

A service level indicator is a measured quantity, best expressed as a ratio of good events to valid events: requests answered without error, or requests faster than a threshold. A service level objective is a target for that indicator over a window, such as 99.9 percent over 30 days. A service level agreement is a contract with consequences and is normally looser than the internal objective. The error budget is one minus the objective: the amount of failure allowed in the window. When the budget is spent, risky changes slow down, which gives product and platform teams a shared rule.

Choosing SLIs for ML systems

Pick a few indicators that reflect what users experience. For online inference: availability and latency at a high percentile measured at the gateway, not inside the model server. For LLM endpoints: time to first token and successful completion ratio. For batch scoring and feature pipelines: freshness, meaning the share of time the newest output is younger than a threshold, and correctness of the output. Use percentiles, not averages, because averages hide the slow tail. Do not set the target at 100 percent, and do not set it higher than your dependencies can support.

Alerting on SLOs with burn rates

Multiwindow, multi-burn-rate

Burn rate is how fast the error budget is consumed relative to a rate that would exhaust it exactly at the end of the window. A burn rate of 1 uses the whole 30-day budget in 30 days. The recommended pattern pages when 2 percent of the budget is consumed in one hour (burn rate 14.4) or 5 percent in six hours (burn rate 6), and opens a ticket when 10 percent is consumed in three days (burn rate 1). Each condition also checks a short window, one twelfth of the long one, so the alert resets quickly after recovery. This gives good precision and recall compared with simple threshold alerts.

On-call design and runbooks

Sustainable rotations and usable runbooks

On-call guidance from site reliability practice caps operational work at half of an engineer's time, keeps the number of incidents per shift low enough that each can be handled and followed up properly, and defines response times by service criticality. Every page must be actionable and linked to a runbook. A runbook for model serving states the symptom, the dashboard to open, the checks in order, and the safe mitigations: roll back to the previous model version, shift traffic away from a canary, scale replicas, fall back to a smaller model or a cached answer. Mitigate first, diagnose after.

Blameless postmortems

The written review

A postmortem is a written record of an incident: impact, timeline, root causes, what went well, what went badly, and action items with owners. Triggers are agreed beforehand, for example user-visible downtime beyond a threshold, data loss, a manual rollback, or a monitoring failure. Blameless means the document examines why the system allowed the mistake, not who made it, so people report honestly. Postmortems are reviewed by peers, stored where others can search them, and their action items are tracked to completion. For ML incidents, record the model version, data window and whether monitoring detected the problem before users did.

Sources

Related lessons

Model monitoring: data quality, drift and alertingSecurity, identity and access, and compliance for ML platforms