Reliability, SLOs and failure modes

Generated by Claude through GigAI (karthik446/gigai-learning-center 4c0cb514) on 2026-10-13.

Concept: SLOs, SLIs and error budgets · also covers Reliability, Site reliability engineering, Monitoring · Lesson 15 of 31 in System design · version 1

Practice this

On this page

Reliability becomes an engineering quantity once it is defined from the user's point of view and measured. Service level indicators measure what users experience, objectives set a target for them, and the error budget that results governs how much risk the team can take. Alerting is then built on the rate at which that budget is being spent, and design reviews enumerate failure modes against it.

Read more

How it is done in practice

A team writes down two or three SLIs per user-facing service, agrees the targets and an error budget policy with product owners, and encodes them as recording rules for the error ratio over each window. Multiwindow burn rate alerts route through Alertmanager, with fast burns paging the on-call engineer and slow burns filing tickets. Every page links to a runbook. A dashboard shows remaining budget, and a regular review looks at where budget went, which feeds postmortems and prioritization. Design reviews include a failure mode table per dependency, and failover paths are exercised on a schedule so the recovery time assumed in the SLO is a measured number.

Technologies

PrometheusAlertmanagerGrafanaOpenTelemetrySlothPyrraPagerDutyDatadog SLOsGoogle Cloud MonitoringThanos

Diagram

flowchart LR
    S[Services expose request and error counters] --> P[Prometheus scrape]
    P --> RR[Recording rules: error ratio per window]
    RR --> AR[Alerting rules: multiwindow burn rate]
    RR --> D[Dashboard: remaining error budget]
    AR --> AM[Alertmanager: group, inhibit, silence, route]
    AM -->|fast burn| PG[Page on-call]
    AM -->|slow burn| T[Ticket queue]
    D --> BR[Budget review and policy decisions]
flowchart LR
    S[Services expose request and error counters] --> P[Prometheus scrape]
    P --> RR[Recording rules: error ratio per window]
    RR --> AR[Alerting rules: multiwindow burn rate]
    RR --> D[Dashboard: remaining error budget]
    AR --> AM[Alertmanager: group, inhibit, silence, route]
    AM -->|fast burn| PG[Page on-call]
    AM -->|slow burn| T[Ticket queue]
    D --> BR[Budget review and policy decisions]

Code

yaml
groups:
  - name: slo-checkout
    rules:
      - record: job:slo_errors:ratio_rate1h
        expr: |
          sum(rate(http_requests_total{job="checkout",code=~"5.."}[1h]))
          / sum(rate(http_requests_total{job="checkout"}[1h]))
      - record: job:slo_errors:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
          / sum(rate(http_requests_total{job="checkout"}[5m]))
      - alert: CheckoutFastBurn
        # 99.9 percent SLO: budget 0.001, burn rate 14.4
        expr: |
          job:slo_errors:ratio_rate1h > (14.4 * 0.001)
          and job:slo_errors:ratio_rate5m > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: Checkout is burning error budget at 14.4x

Subtopics

SLIs, SLOs and SLAs

Three distinct things

A service level indicator is a measured quantity describing service behaviour, best expressed as a ratio of good events to valid events: requests answered successfully, requests faster than a threshold, records processed within a freshness bound. A service level objective is a target for an SLI over a window, such as 99.9 percent of requests succeeding over 30 days. A service level agreement is a contract with consequences for missing it, and is normally looser than the internal SLO so there is margin before penalties apply.

Choosing good indicators

Pick a small number of indicators tied to what users care about, by system type. Serving systems track availability, latency and throughput. Storage systems track latency, availability and durability. Pipelines track freshness and correctness. Measure latency as a distribution with percentiles, not averages, because an average hides the slow tail that users notice. Measure as close to the user as practical: load balancer metrics capture failures that server-side metrics miss when the server never saw the request. Do not target 100 percent. It is unreachable, users cannot perceive it through their own less reliable networks, and it leaves no room for change.

Error budgets and what they decide

From target to budget

The error budget is one minus the SLO. A 99.9 percent availability target over 30 days allows 0.1 percent of requests to fail, or roughly 43 minutes of full downtime. The budget turns reliability into a shared, spendable resource. Releases, experiments, migrations and planned maintenance all consume it, as do incidents. While budget remains, the team ships at full speed. When it is exhausted, an error budget policy agreed in advance takes effect, typically shifting effort to reliability work and slowing risky changes until the service is back within target.

Setting and revising targets

Start from current measured performance and user expectations, not from an aspiration. Publish a target somewhat looser than what the system achieves so there is headroom. Remember that a service cannot be more reliable than its hard dependencies combined, so targets must be consistent along the call chain. Systems that greatly exceed their target teach users to depend on the higher level, so either tighten the target or accept that the margin is not a promise. Review objectives periodically with product owners and adjust when they are neither driving decisions nor matching user happiness.

Burn rate alerting

Alert on budget consumption

Alerting on raw error rate thresholds is either noisy or slow. Burn rate is how fast the budget is being consumed relative to the rate that would exactly exhaust it at the end of the window. A burn rate of 1 uses the whole budget in 30 days, and 14.4 uses 2 percent of it in one hour. Alerting on burn rate ties each page to a concrete threat to the objective, and scales severity naturally: fast burns page a human, slow burns open a ticket.

Multiple windows and burn rates

The recommended scheme combines several burn rates, each checked over a long and a short window. For a 30 day objective, a typical set pages at 14.4 over one hour and at 6 over six hours, and tickets at 1 over three days. Each condition also requires the same burn rate over a short window of about one twelfth the length, such as five minutes for the one hour alert. The long window gives significance and the short one makes the alert reset quickly once the problem stops. Low-traffic services need care, since a few failed requests can look like a huge burn.

Implementing SLOs with recording rules, alerting rules and Alertmanager

Recording and alerting rules

Recording rules precompute expensive expressions on a schedule and store the result as a new time series. For SLOs, this means recording the error ratio per service over each window needed, such as five minutes, one hour and six hours, so dashboards and alerts query cheap precomputed series. Alerting rules evaluate an expression and fire when it returns results. A for clause requires the condition to hold for a duration before firing, moving the alert from pending to firing. Labels carry severity and routing information, and annotations carry human-readable summaries and runbook links.

Alertmanager

The metrics server only decides that an alert is firing. Alertmanager decides who hears about it. It deduplicates alerts and groups related ones into a single notification, so a failure across a hundred instances pages once, not a hundred times. Inhibition suppresses downstream alerts when a broader one is already firing, for example muting per-service alerts while a whole cluster is unreachable. Silences mute matching alerts for a time window during maintenance. A routing tree matches on labels to send notifications to the right receiver, such as a paging service for critical severity and a chat channel for tickets.

Reasoning about failure modes in a design

Enumerate, then bound the blast radius

For each component and dependency in a design, ask what happens when it is down, slow, returning errors, returning wrong data, or partitioned from its peers. Slow is usually worse than down, because it holds resources and evades simple health checks. Classify dependencies as hard, where the request cannot succeed without them, or soft, where the service can degrade by serving stale data, defaults or a reduced feature. Look for correlated failure: shared zones, shared config pushes, shared credentials and certificates that expire together. Then check each mode against the SLO: how it is detected, how long mitigation takes, and how much budget it would consume. Redundancy across zones, staged rollouts with automatic rollback and tested failover are the standard mitigations.

Sources

Related lessons

Practice this
Capacity planning and performanceSecurity basics in designs