Rate limiting, overload and resilience patterns

Generated by Claude through GigAI (karthik446/gigai-learning-center 4c0cb514) on 2026-10-13.

Concept: Rate limiting and overload protection · also covers Reliability, Scaling, Production systems · Lesson 12 of 31 in System design · version 1

Every service has a finite capacity, and demand will sometimes exceed it through traffic spikes, misbehaving clients or lost capacity. Rate limiting controls how much any one caller may send, while load shedding and circuit breaking protect the service as a whole when total demand is too high. The goal is graceful degradation: keep serving the most important requests at full quality and reject the rest cheaply and early.

How it is done in practice

A typical setup layers the defenses. The edge gateway applies per-tenant rate and concurrency limits backed by a shared counter store. Sidecar proxies apply local token buckets and per-upstream circuit breaking limits. Services themselves shed load by criticality when utilization passes a threshold. Limits are rolled out in a shadow mode first, logging what would have been rejected, then enforced once the false positive rate is acceptable. Teams load test past saturation to confirm that goodput stays flat under overload, alert on rejection and overflow counters, and document for API consumers what a throttling response means and how to back off.

Simulate it

A token-bucket limiter admits at most its limit per second plus a burst, and turns the rest away before they reach the API. Failed in the readouts includes the requests the limiter refused.

1. At the start 150 requests a second pass untouched and the API is about 63% busy.
2. Raise the offered load to 600 with the limit at 200: goodput holds at 200 requests a second, p99 stays near 81 ms and the API stays at about 83% busy, while about 67% of requests are turned away.
3. Raise the limit to 400, above what the API can serve, and the limiter no longer protects it: the API saturates, p99 climbs to about 288 ms and goodput falls to about 239 requests a second.
4. Set the offered load to 190, the limit to 200 and drop the burst to 6: even below the limit about 6% of requests are turned away, because arrivals come in clumps that a tiny bucket cannot absorb.
5. Raise the burst to 50 and they all pass again.

Technologies

Diagram

flowchart LR
    C[Clients] --> G[Edge gateway]
    G -->|descriptors| R[Global rate limit service]
    R --> K[(Shared counter store)]
    G --> P[Sidecar proxy: local token bucket]
    P --> B{Circuit breaker limits ok?}
    B -->|no| F[Fail fast with overflow]
    B -->|yes| S[Service]
    S --> L{Utilization high?}
    L -->|yes, low criticality| X[Shed request]
    L -->|no or critical| W[Do the work]

flowchart LR
    C[Clients] --> G[Edge gateway]
    G -->|descriptors| R[Global rate limit service]
    R --> K[(Shared counter store)]
    G --> P[Sidecar proxy: local token bucket]
    P --> B{Circuit breaker limits ok?}
    B -->|no| F[Fail fast with overflow]
    B -->|yes| S[Service]
    S --> L{Utilization high?}
    L -->|yes, low criticality| X[Shed request]
    L -->|no or critical| W[Do the work]

Code

import time

class TokenBucket:
    def __init__(self, rate_per_s, burst):
        self.rate, self.burst = rate_per_s, burst
        self.tokens, self.last = burst, time.monotonic()

    def allow(self, cost=1):
        now = time.monotonic()
        self.tokens = min(self.burst, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= cost:
            self.tokens -= cost
            return True
        return False  # caller responds 429

bucket = TokenBucket(rate_per_s=100, burst=200)
print(bucket.allow())

Subtopics

Rate limiting algorithms

Token bucket and its relatives

The token bucket is the most common algorithm. Each caller has a bucket that refills at a steady rate up to a maximum size. A request takes a token, and if none are left it is rejected. The refill rate sets the sustained limit and the bucket size sets the allowed burst. Fixed window counters are simpler but allow double the limit across a window boundary. Sliding window logs or counters smooth that out at more memory or compute cost. Leaky bucket queues requests and drains them at a constant rate, which shapes traffic instead of rejecting it.

Limiting concurrency, not just rate

A rate limit counts requests per unit of time, but requests differ widely in cost. A concurrent request limiter caps how many requests a caller may have in flight at once. It handles expensive, slow endpoints well: a client that sends few but heavy requests is contained, and clients that retry on timeout cannot stack up work. Rejections should carry a clear status such as HTTP 429, with enough information for a well-behaved client to back off.

Local and global rate limiting

Two enforcement points

Local rate limiting is enforced inside each proxy or service instance using an in-memory token bucket. It is fast, has no external dependency and keeps working during partitions, but the limit applies per instance, so the effective fleet-wide limit scales with instance count. Global rate limiting enforces one shared limit across the fleet: each proxy calls a central rate limit service, which keeps counters in a shared store and answers allow or deny for a set of descriptors such as caller identity, route or header values.

Combining them

The global service is on the request path, so it adds latency and becomes a dependency that needs its own failure policy: fail open and risk overload, or fail closed and reject traffic. A common design uses both layers. A coarse local limit absorbs large bursts and cuts load on the global service, and the global limit then applies precise per-tenant quotas. Quota-based variants cut round trips further by having the central service hand each proxy an allocation for a period of time instead of a decision per request.

Load shedding and request criticality

Shedding by priority

Rate limits protect against individual callers. Load shedding protects against total demand exceeding capacity, even when every caller is within its limit. The server watches a utilization signal and, past a threshold, rejects requests before doing expensive work. Shedding should be ordered by importance. Requests are tagged with a criticality level, for example critical, sheddable and best-effort tiers, and the lowest tier is dropped first. One approach reserves a share of fleet capacity for critical traffic. Another sheds progressively as worker utilization rises: test traffic first, then non-critical methods, and critical ones last.

Choosing the signal and cost of rejection

Requests per second is a poor capacity measure because request cost varies. Measuring capacity in resource terms, such as CPU consumed or in-flight requests, tracks real load better. Rejection must be much cheaper than serving. Otherwise a server that spends most of its time saying no still collapses. That means shedding as early as possible, before parsing large payloads or taking locks. Criticality should propagate through the call chain so backends make consistent decisions about the same user request. Bounded queues with short timeouts stop a server from spending effort on requests whose callers have already given up.

Client-side adaptive throttling

Stopping load at the source

When a backend is heavily overloaded, even cheap rejections consume capacity. Client-side throttling lets clients drop requests locally before they reach the network. In adaptive throttling, each client tracks, over a recent window, how many requests it attempted and how many the backend accepted. When attempts grow beyond a multiple K of accepts, the client starts rejecting new requests locally, with probability rising in proportion to the gap. A K of 2 means the client lets through about twice what the backend accepts, which wastes some backend effort but tracks recovery quickly. This pairs with retry budgets so that an overloaded backend sees shrinking demand, not growing demand.

Circuit breaking and bulkheads at the proxy

Resource limits per upstream

In a mesh proxy, circuit breaking is a set of hard limits per upstream cluster. They cover maximum connections, maximum pending requests waiting for a connection, maximum in-flight requests and maximum concurrent retries. When a limit is hit, the proxy fails the request immediately instead of queueing it, and increments an overflow counter. This is a bulkhead: one slow upstream can consume only its own allotment of connections and queue slots, so it cannot starve calls to healthy upstreams. Limits can be set separately per routing priority, so high-priority traffic has its own pool.

Tuning

Limits set too high never trip and offer no protection. Limits set too low reject healthy traffic at peak. Start from observed peak concurrency per upstream plus headroom, then watch overflow counters and adjust. The concurrent retry limit, or a retry budget expressed as a fraction of active requests, is the guard against retry storms at the mesh layer. Outlier detection complements these limits by ejecting individual unhealthy hosts from the load balancing pool, while circuit breaking bounds the load on the cluster as a whole.

Sources

• docEnvoy global rate limiting (www.envoyproxy.io)
• docEnvoy local rate limiting (www.envoyproxy.io)
• docEnvoy circuit breaking (www.envoyproxy.io)
• articleScaling your API with rate limiters (Stripe) (stripe.com)
• bookHandling overload (SRE book) (sre.google)

Related lessons

• Timeouts, retries and idempotency
• Reliability, SLOs and failure modes
• Load balancing and service discovery
• Capacity planning and performance
• API design
• Queues, streams and delivery guarantees