Practice: Observability for designers
Generated by Claude through GigAI (karthik446/gigai-learning-center 4c0cb514) on 2026-10-13.
Main tool: OpenTelemetry · OpenTelemetry is the one vendor-neutral standard that makes a designer decide signals, attributes, context propagation, sampling and the collection pipeline in a single model, so every subtopic of this lesson can be practised with it on a laptop. · Alternatives: Prometheus client libraries cover metrics only with a pull model and no traces, Jaeger or Zipkin native clients cover tracing only, Micrometer is a JVM-centred metrics and tracing facade, and vendor agents give turnkey collection but tie instrumentation to one backend, whereas OpenTelemetry is vendor-neutral across traces, metrics and logs.
Every step links into the tool's own documentation; links were checked on 2026-10-13 · version 1
Setup (one time)
1. Install the .NET SDK that the getting started guide lists and confirm the dotnet command works in a terminal.
The fetched hands-on guides for OpenTelemetry use .NET, so one SDK install lets you run every example in this path locally.
Explained in: Getting started with traces - Console | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: Asking dotnet for its version prints a version that meets the listed prerequisite.
2. Install and start Prometheus locally with the OTLP receiver enabled as the guide describes.
You need a local metrics backend so the metrics you design can be queried and charted without any paid cloud.
Explained in: Export to Prometheus and Grafana | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: The Prometheus web page opens in a browser on your laptop.
3. Install and start Jaeger locally, as a container or a downloaded binary, with OTLP ingestion enabled.
A local trace backend lets you see whole distributed traces instead of console output, which is what the later propagation and sampling steps need.
Explained in: Export to Jaeger | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: The Jaeger search page opens in a browser on your laptop.
4. Install and start Grafana locally and add Prometheus as a data source.
Dashboards are part of the design, and Grafana over your local Prometheus is the free way to practise them.
Explained in: Export to Prometheus and Grafana | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: Grafana reports that the Prometheus data source test succeeded.
Steps
1. Create a console application, add OpenTelemetry tracing with the console exporter, and run it.
This is the shortest possible first run: one tracer provider, one source, one span printed to the terminal. It shows the tracing pipeline before any backend is involved.
Explained in: Getting started with traces - Console | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: The terminal prints one activity with a trace id, a span id and the tags you set.
2. Write a one-page signal map for a small two-service design that lists which traces, metrics and logs each component emits and which question each one answers.
A designer chooses signals up front. Mapping each signal to a diagnostic question keeps you from emitting telemetry nobody will query.
Explained in: Signals | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: Every component in your sketch has at least one named trace, metric and log with the question it answers.
3. Create nested spans in the application and set span kind, attributes, an event and an error status on them.
Spans, their context and their attributes are the core object model of tracing, and the attribute choices you make here decide what you can filter on during an incident.
Explained in: Traces | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: The output shows a parent and a child span sharing one trace id, with the kind, attributes and status you set.
4. Enable the OpenTelemetry log bridge for the application logger and emit structured log records inside an active span.
Logs become far more useful when they carry the trace and span ids of the request that produced them. The bridge is how existing logging joins the other two signals.
Explained in: Logs | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: An exported log record shows named fields plus the same trace id as the span it was written in.
5. Create a counter, a histogram and a gauge on one meter, with a small fixed set of attributes on each.
Choosing the instrument type decides what aggregations are possible later: rates from counters, percentiles from histograms, current values from gauges.
Explained in: Metrics | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: The exported metrics show three instruments with the expected types and one series per attribute combination.
6. Configure a cardinality limit on the meter provider, then record a metric with an unbounded attribute such as a user id and observe the result.
High cardinality attributes are the most common way a metrics design blows up storage cost. Seeing the limit engage teaches you which attributes belong on spans instead.
Explained in: Metrics | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: Once the limit is reached, new attribute combinations are folded into an overflow series instead of creating new ones.
7. Run two local services where the first calls the second over HTTP, and inspect the trace context header that travels between them.
A trace only spans services when context is propagated on every hop. Verifying the header yourself shows exactly where a broken trace would break.
Explained in: Context propagation | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: Spans from both services share one trace id and the second service's span lists the first service's span as its parent.
8. Start a local OpenTelemetry Collector in a container, point both services at it over OTLP, and have it forward to your backends.
Putting a Collector between applications and backends moves batching, retries, filtering and backend choice out of application code, which is the central pipeline decision in the design.
Explained in: Collector | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: Telemetry still reaches the backends while the services are configured with only the Collector address.
9. Configure OTLP metrics export to Prometheus using environment variables only, so the same build runs unchanged with a different endpoint.
Keeping exporter settings in the environment makes the setup reproducible across laptops and CI, and keeps backend choice out of the code.
Explained in: OTLP metrics export to Prometheus | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: Restarting with only the environment changed makes the metrics appear in the Prometheus query page.
10. Run the tail-based sampling example so that failed requests are always kept while successful ones are mostly dropped.
Sampling decides which traces you still have during an incident. A tail decision keeps the errors that a head decision made at request start would have thrown away.
Explained in: Tail-based sampling | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: Under mixed traffic the exported traces include every failed request and only a fraction of the successful ones.
11. Create a Grafana dashboard over the exported metrics with one panel per service level indicator you would alert on, such as request rate, error ratio and latency percentile.
Dashboards and alerts should follow the design: each panel maps to a metric you chose on purpose and to a threshold someone would be paged for.
Explained in: Export to Prometheus and Grafana | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: The dashboard shows live panels that move when you send traffic, and each panel has a written alert threshold next to it.
12. Configure the trace export to Jaeger the way the production usage notes describe, with batching and an explicit endpoint, and run the whole stack end to end.
The quick-start defaults are not what you would ship. Finishing with the production guidance shows what changes between a demo and a deployment you could defend in a design review.
Explained in: Export to Jaeger | OpenTelemetry (opentelemetry.io), checked 2026-10-13
Done when: Jaeger shows complete multi-service traces from the full stack, and your configuration matches each point in the production usage notes.