Observability for designers

Generated by Claude through GigAI (karthik446/gigai-learning-center 4c0cb514) on 2026-10-13.

Concept: Observability · also covers Logs, metrics and tracing, Telemetry, Monitoring · Lesson 13 of 31 in System design · version 1

Practice this

On this page

Observability is the ability to understand what a system is doing internally from the telemetry it emits, without shipping new code to ask a new question. A designer decides up front which signals each component emits, how request context travels across service boundaries, and how telemetry is collected and stored. Those choices determine whether an incident is diagnosed in minutes or in hours.

Read more

How it is done in practice

A platform team publishes a small instrumentation library and a collector configuration, so every service gets request metrics, trace propagation and structured logs with trace ids by default. Services export OTLP to a node-local collector, which forwards to a gateway pool that samples traces and fans out to a metrics store, a trace store and a log store. Design reviews include a telemetry section: key metrics per component, label sets checked for cardinality, the SLO alerts, and the dashboards that exist before launch. After incidents the team asks which question could not be answered from telemetry and adds that signal.

Technologies

OpenTelemetryOpenTelemetry CollectorOTLPPrometheusPromQLAlertmanagerGrafanaJaegerGrafana TempoGrafana Loki

Diagram

flowchart LR
  A[Service A with SDK] -->|OTLP| AG[Node agent collector]
  B[Service B with SDK] -->|OTLP| AG
  AG --> GW[Gateway collector pool]
  GW -->|metrics| M[Metrics store]
  GW -->|sampled traces| T[Trace store]
  GW -->|logs| L[Log store]
  M --> D[Dashboards and alerts]
  T --> D
  L --> D
flowchart LR
  A[Service A with SDK] -->|OTLP| AG[Node agent collector]
  B[Service B with SDK] -->|OTLP| AG
  AG --> GW[Gateway collector pool]
  GW -->|metrics| M[Metrics store]
  GW -->|sampled traces| T[Trace store]
  GW -->|logs| L[Log store]
  M --> D[Dashboards and alerts]
  T --> D
  L --> D

Code

python
import time
from prometheus_client import Counter, Histogram, start_http_server

REQS = Counter("http_requests_total", "Requests", ["route", "code"])
LAT = Histogram("http_request_duration_seconds", "Latency", ["route"])

def handle(route, work):
    start = time.perf_counter()
    code = "200"
    try:
        return work()
    except Exception:
        code = "500"
        raise
    finally:
        REQS.labels(route, code).inc()
        LAT.labels(route).observe(time.perf_counter() - start)

start_http_server(8000)  # exposes /metrics for scraping

Subtopics

The signals: traces, metrics and logs

What each signal answers

A metric is a numeric measurement aggregated over time, cheap to store and ideal for dashboards and alerts: request rate, error rate, latency distribution, queue depth. A log is a timestamped record of a discrete event, best for the detail of one occurrence. A trace records the path of one request through many services as a tree of spans, each with a start time, a duration and attributes. Metrics tell you that something is wrong, traces tell you where, and logs tell you why. OpenTelemetry treats these as separate signals that share a common resource description and a common context, plus baggage for passing key-value pairs along a request.

Correlating signals

The signals are most useful when joined. Put the trace id and span id into every structured log line so a slow span leads straight to its logs. Use the same service name, version and environment attributes on all three signals so that a dashboard panel, a trace search and a log query can be filtered identically. Decide these naming conventions once, in a shared library or collector configuration, because fixing inconsistent attribute names across fifty services later is expensive.

Designing metrics: types, labels and cardinality

Metric types and what to measure

Prometheus offers counters (only go up, queried as a rate), gauges (go up and down, such as in-flight requests), and histograms (observations counted into buckets, from which quantiles are estimated on the server). For online serving systems, instrument request count, error count and latency per endpoint. For offline or batch systems, track items in, items out, time of last success and the age of the oldest unprocessed item. For queues and pools, track size, saturation and wait time. Measure latency with histograms rather than averages, since averages hide the tail that users actually feel.

Cardinality is the cost driver

Every unique combination of label values is a separate time series that the server must hold in memory and on disk. A label for HTTP method or status code is safe; a label for user id, request id or raw URL path creates unbounded series and can take the monitoring system down. Keep label values to a small, bounded set, use route templates instead of raw paths, and send high-cardinality detail to traces or logs instead. Prometheus scrapes targets over HTTP on an interval, so each instance exposes its current values and the server computes rates and aggregations with PromQL.

Trace context propagation and sampling

Propagation across boundaries

A trace only works if every hop forwards the context. On an outgoing call the client injects the trace id, the parent span id and sampling flags into request headers, and the server extracts them and starts a child span. The same must happen through message queues, where context goes into message headers, and across thread pools and async tasks inside a process. One service that drops the headers splits a trace in two, so propagation is a platform requirement, usually met with auto-instrumentation libraries and a shared middleware.

Sampling

Recording every span at high request rates is too expensive, so systems sample. Head sampling decides at the start of the request, for example keep one percent, and is cheap but blind to the outcome. Tail sampling buffers the spans of a trace and decides after it completes, so all errors and slow requests can be kept while most fast successes are dropped. Tail sampling needs all spans of a trace to reach the same collector instance, which shapes the collection topology.

The telemetry pipeline and the OpenTelemetry Collector

Receivers, processors, exporters

The OpenTelemetry Collector is a vendor-neutral process that receives, processes and exports telemetry. Receivers accept data in formats such as OTLP or scrape Prometheus endpoints. Processors batch, filter, sample, redact sensitive attributes and add resource metadata. Exporters send the result to one or more backends. Putting a collector between applications and backends means services export one protocol and never need redeploying when the backend changes, and that retries, batching and credentials live in one place.

Deployment patterns

Two patterns are common and often combined. An agent runs next to the workload, as a sidecar or one per node, receives telemetry over the local network and enriches it with host and orchestrator metadata. A gateway is a horizontally scaled pool of collectors per cluster or region that handles heavier work such as tail sampling and fan-out to several backends. Size the pipeline like any other data path: it needs memory limits, back-pressure handling and its own monitoring, or it will fail during the very incident it is meant to explain.

Alerting and dashboards that follow the design

Alert on symptoms, diagnose with causes

Page a human for user-visible symptoms: error ratio above the objective, latency above the objective, or work not completing. Cause-level signals such as CPU, disk or one replica restarting belong on dashboards and tickets unless they predict an imminent outage, such as a disk filling. For each component in the design, write down the handful of metrics that prove it is healthy, the alert that fires when it is not, and the first query an on-call engineer runs. If that list cannot be written, the component is not yet observable enough to ship.

Sources

Related lessons

Practice this
Rate limiting, overload and resilience patternsCapacity planning and performance