Timeouts, retries and idempotency

Generated by Claude through GigAI (karthik446/gigai-learning-center 4c0cb514) on 2026-10-13.

Concept: Timeouts, retries and idempotency · also covers Reliability, Distributed systems, API design · Lesson 11 of 31 in System design · version 1

On this page

Remote calls fail in ways local calls do not: they can be slow, lost, or completed without the caller learning the outcome. Timeouts bound how long a caller waits, retries hide transient failures, and idempotency makes those retries safe. Applied carelessly, the same tools amplify an outage, so each needs limits: backoff, jitter, retry budgets and circuit breakers.

Read more

How it is done in practice

Teams put these policies in a shared client library or a service mesh sidecar rather than in each call site, so every outbound call gets a timeout, a bounded jittered retry and a breaker by default. Timeouts are derived from dashboards of dependency latency and revisited when the dependency changes. Mutating APIs accept an idempotency key, stored in the same database transaction as the business write, with a unique constraint doing the deduplication. Retry counts, breaker state and timeout rates are emitted as metrics, and game days or fault injection confirm that a slow dependency produces fast failures, not a retry storm.

Simulate it

A retry is extra load aimed at the dependency that is already struggling. Here the API retries its calls to a database that is about 62% busy at 60 requests a second, and the circuit breaker starts switched off.

  1. At the start nothing is wrong: failures stay at 0% and the database receives about 61 calls a second, the same as the offered load.
  2. Raise the offered load to 80 with 3 retries and wait. Goodput falls to 0 and the database is asked for about 313 calls a second although only about 78 requests a second were offered.
  3. Set the retries to 0 at the same load of 80 and the database copes: goodput stays at about 76 requests a second, with about 5% failing.
  4. Put the retries back to 3 and move the circuit breaker control down to 0.5. The breaker opens and the database drops to about 77 calls a second and about 50% busy, while clients still see about 56% of requests fail: the breaker protects the database, it does not make it faster.
  5. Move the breaker control back to 1, set the load to 60 and press Crash the database: the API sends about 240 calls a second at a database that is not there.
  6. Do the same with the breaker at 0.5 and almost no calls reach the dead database, because the open breaker turns them away first.

The simulation needs JavaScript; the steps above describe what it shows. Simulation engine: Breakscale (MIT, the Breakscale contributors), commit d2c876e0, vendored without its interface, analytics or link store. Nothing leaves this page.

Technologies

gRPC deadlinesEnvoyIstioResilience4jPollyTenacityAWS SDK retry modesFinaglePostgreSQL unique constraintsRedis

Diagram

sequenceDiagram
    participant C as Client
    participant S as API server
    participant D as Database
    C->>S: POST charge, key K1
    S->>D: begin, insert key K1, apply charge, commit
    D-->>S: committed
    S--xC: response lost, client times out
    Note over C: sleep random 0 to capped backoff
    C->>S: POST charge, key K1 again
    S->>D: look up key K1
    D-->>S: stored result
    S-->>C: same response, no second charge
sequenceDiagram
    participant C as Client
    participant S as API server
    participant D as Database
    C->>S: POST charge, key K1
    S->>D: begin, insert key K1, apply charge, commit
    D-->>S: committed
    S--xC: response lost, client times out
    Note over C: sleep random 0 to capped backoff
    C->>S: POST charge, key K1 again
    S->>D: look up key K1
    D-->>S: stored result
    S-->>C: same response, no second charge

Code

python
import random, time

RETRYABLE = (TimeoutError, ConnectionError)

def call_with_retry(fn, deadline_s=2.0, base=0.05, cap=1.0, attempts=4):
    start = time.monotonic()
    for attempt in range(attempts):
        remaining = deadline_s - (time.monotonic() - start)
        if remaining <= 0:
            break
        try:
            return fn(timeout=remaining)
        except RETRYABLE:
            # full jitter: uniform between 0 and capped exponential delay
            sleep = random.uniform(0, min(cap, base * 2 ** attempt))
            time.sleep(min(sleep, max(0.0, remaining)))
    raise TimeoutError("deadline exceeded or retries exhausted")

Subtopics

Timeouts and deadline propagation

Why every remote call needs a timeout

A call without a timeout ties up a thread, a connection and memory for as long as the remote side stays silent. Under a slow dependency those held resources pile up until the caller itself stops serving, which is how one slow service takes down its callers. Set both a connection timeout and a request timeout on every client. A common way to pick the request timeout is to start from the measured latency distribution of the dependency and choose a high percentile, such as p99.9, so that the rate of false timeouts is a known, accepted number rather than a guess.

Deadlines across a call chain

In a chain of services, independent per-hop timeouts waste work: a downstream service keeps computing an answer the original caller has already abandoned. Deadline propagation fixes this. The edge sets an absolute deadline, each hop passes the remaining time along with the request, and each service checks it before starting expensive work and cancels when it expires. Timeouts should also shrink toward the leaves, so an inner call cannot outlive the outer one. Watch for two traps: timeouts set too low cause retry storms when latency rises slightly, and timeouts that exclude DNS or TLS handshake time are not real bounds.

Retries with exponential backoff and jitter

What to retry and how to space it

Retry only failures that are likely transient: connection resets, timeouts, throttling responses and server-side unavailable errors. Do not retry validation or authorization errors, since the result will not change. Immediate retries hit a struggling service at its worst moment, so wait between attempts and grow the wait exponentially, for example base times two to the power of the attempt, up to a cap. Always bound the number of attempts, and stop early when the overall deadline is too close to fit another try.

Why jitter matters

Plain exponential backoff still synchronizes clients. If a thousand clients fail at the same instant, they all retry at one base interval, then two, then four, producing spikes of load separated by idle gaps. Jitter adds randomness to spread the retries out. With full jitter, the sleep is a uniform random value between zero and the capped exponential delay. Analysis of competing clients shows this cuts both the total number of calls and the time to complete the work compared with unjittered backoff. The same idea applies to any periodic work: timers, cache expiry and scheduled jobs should be jittered too.

Retry amplification and retry budgets

How retries multiply load

Retries are selfish: each client spends more of the server's capacity to raise its own chance of success. When a failure is caused by overload, that makes it worse. The effect compounds across layers. If each of five layers in a call stack retries three times, a failure at the bottom can produce up to 243 times the normal load on the deepest service. The usual fix is to retry at a single layer of the stack, typically the one closest to the failing call or the one that owns the user-facing request, and let other layers pass failures through.

Budgets instead of fixed counts

A per-request attempt limit does not cap fleet-wide retry load. A retry budget does. One form is a local token bucket: each success adds a fraction of a token, each retry spends one, and when the bucket is empty retries stop. Another is a ratio: retries may be at most a set fraction of total requests. During a real outage the budget drains quickly and the client falls back to roughly normal load, while isolated transient failures still get retried. Servers can help by returning an explicit signal that distinguishes overload from other failures.

Idempotency keys for safe retries

The problem

When a request times out, the caller cannot tell whether the server never saw it, failed partway, or finished and lost the response. Retrying a read is harmless, but retrying a create or a charge may apply it twice. An idempotent operation is one where repeating it has the same effect as doing it once. Some operations are naturally idempotent, such as setting a value or deleting by identifier. Others, such as appending, incrementing or charging, must be made idempotent by design.

How the key mechanism works

The client generates a unique key, usually a random UUID, and sends it with the mutating request, commonly in a header. The server records the key together with the outcome. If the same key arrives again, the server returns the stored result without redoing the work. Important details: the key lookup and the side effect must commit atomically, or a crash between them breaks the guarantee. A second request arriving while the first is in flight should be rejected or made to wait. The server should check that the parameters match the original request. Keys expire after a retention window, so clients must not retry beyond it.

Circuit breakers

States and transitions

A circuit breaker wraps calls to a dependency and tracks failures. In the closed state calls pass through. When failures cross a threshold, the breaker opens and calls fail immediately without touching the dependency, which frees caller resources and gives the dependency room to recover. After a reset timeout the breaker goes half-open and lets a trial call through: success closes it, failure reopens it. Thresholds can be consecutive failures or an error ratio over a window. Decide what the caller does while the breaker is open: return a cached or default value, queue the work, or fail fast with a clear error. Breaker state changes should be logged and exposed as metrics.

Sources

Related lessons

Data modelling across storesRate limiting, overload and resilience patterns