Practice: Rate limiting, overload and resilience patterns
Generated by Claude through GigAI (karthik446/gigai-learning-center 4c0cb514) on 2026-10-13.
Main tool: Envoy · Envoy puts token bucket local limits, a global rate limit service, priority-aware circuit breakers and retry budgets in one proxy you can run in containers on a laptop, so every pattern in this lesson can be observed through its own stats. · Alternatives: NGINX offers simpler leaky bucket limits per instance without a first-class global limit service; Kong gateway packages rate limiting as plugins backed by Redis; Istio configures these same Envoy features through mesh resources instead of raw proxy config; Resilience4j and Sentinel put limiters, bulkheads and circuit breakers inside the application process rather than at the proxy; a managed cloud API gateway gives usage plans and throttling with no proxy to operate but far less control.
Every step links into the tool's own documentation; links were checked on 2026-10-13 · version 1
Setup (one time)
1. Install a container runtime with Compose, pull the official Envoy image, and read the terminology used throughout the path (downstream, upstream, listener, cluster, filter).
Every rate limiting and circuit breaking setting attaches to one of these objects, so the vocabulary has to be exact before you configure anything.
Explained in: Life of a Request — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: Running the Envoy image with its version flag prints a version, and you can define downstream, upstream, listener and cluster from memory.
2. Write a minimal static bootstrap file with one listener, an HTTP connection manager with the router filter, one cluster, and the admin interface enabled.
This single file is the base that every later step edits, and the admin interface is where you will read the stats that prove each pattern works.
Explained in: Life of a Request — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: Envoy validates the bootstrap file and starts without config errors.
3. Start the stack with Compose: one Envoy container in front of two small HTTP echo containers, one of which can be made slow or stopped on demand.
Overload patterns only show themselves when there is a real upstream that can be slowed or removed, so the test backends are part of the lab.
Explained in: Life of a Request — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: A request to the Envoy listener port returns the echo response and the admin stats page lists the upstream cluster.
Steps
1. Run a single request through the proxy and trace it in the access log and admin stats from listener to HTTP filter chain to router to cluster.
Rate limit filters reject in the HTTP filter chain before routing, while circuit breakers act later at the cluster, and knowing that order explains which protection fires first.
Explained in: Life of a Request — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: You can point at the place in the filter chain where a rate limit filter would sit and name the stats counters that moved for your one request.
2. Configure the local rate limit HTTP filter with a token bucket of small size, for example five tokens refilled once per second, then send a burst of fifty requests.
The token bucket is the rate limiting algorithm Envoy implements locally, and changing max tokens, tokens per fill and fill interval shows directly how burst size and sustained rate are separate knobs.
Explained in: Local rate limit — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: Roughly five requests of the burst succeed, the rest return status 429, and the local rate limit rate limited counter in admin stats matches the rejected count.
3. Configure local rate limit descriptors so that one route or one client header value gets its own smaller token bucket than the default.
Real limits are per caller or per endpoint, not one bucket for everything, and descriptors are how Envoy keys a bucket to request attributes.
Explained in: Local rate limit — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: Requests carrying the chosen header value are rejected at the lower limit while other requests still pass under the default bucket.
4. Start the reference rate limit service and a Redis container in the Compose stack, and add a cluster plus the global rate limit HTTP filter that calls the service over gRPC.
A global limit needs shared counters outside any one proxy, and this is the external service contract Envoy uses for that decision.
Explained in: Rate limit service — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: The rate limit service logs show a request per proxied call, and the rate limit filter ok counter in Envoy admin stats increases.
5. Write route level rate limit actions that compose a descriptor from the destination cluster and a client identity header, and add a matching limit in the rate limit service config.
The proxy only sends descriptors and the service owns the limits, so you need to see how actions build the key the service matches on.
Explained in: Rate limit — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: One client identity is rejected with status 429 after exceeding its limit while a second identity on the same route is still served.
6. Run two Envoy containers against the same upstream and compare behaviour with only the local limit, only the global limit, and then both together with the local limit set looser than the global one.
Local limits are cheap but multiply with the number of proxies, global limits are accurate but add a network call, and combining them lets the local bucket absorb floods before they reach the limit service.
Explained in: Local rate limiting — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: You have a small table showing total allowed requests per second in each of the three modes, and during a flood the rate limit service receives far fewer calls when the local limit is on.
7. Configure a retry policy on the route and a retry budget on the cluster thresholds, then slow one upstream and send steady load.
A proxy acting as a client must throttle itself when the backend struggles, and a budget that caps retries as a fraction of active requests is how Envoy stops retry storms from amplifying an overload.
Explained in: Circuit breakers (proto) — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: With the budget in place the upstream retry overflow counter increases and total upstream request volume stays near the incoming rate instead of multiplying.
8. Configure circuit breaker thresholds for max connections, max pending requests and max requests on two separate clusters, then slow the upstream of one cluster under load.
Per cluster limits are the bulkhead at the proxy: a slow dependency fills only its own pool and fails fast, so the other cluster keeps serving.
Explained in: Circuit breaking — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: Requests to the slow cluster get fast 503 responses with the overloaded header and its pending overflow counter rises, while the healthy cluster shows no errors.
9. Configure separate threshold entries for default and high routing priority on one cluster, mark a critical route as high priority, and overload the cluster.
Load shedding by criticality means the important traffic keeps its own capacity reservation when the default pool is exhausted, which is what graceful degradation looks like at the proxy.
Explained in: Circuit breakers (proto) — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: Under overload the default priority route is shed with 503 responses while the high priority route continues to return 200.
10. Reproduce the overload test as a scripted run with a pinned Envoy image tag: once with thresholds raised so high they never trip as a baseline, and once with your tuned thresholds.
Circuit breakers cannot be switched off, only set very high, and a repeatable baseline versus protected comparison is the evidence that your numbers help rather than hurt.
Explained in: Is there a way to disable circuit breaking? — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: One command runs both scenarios from a clean stack and writes latency and error counts for each, and a second run gives comparable numbers.
11. Enable the global rate limit filter in shadow mode using its runtime keys so limits are evaluated but not enforced, then raise enforcement gradually to full.
A wrong limit rejects real customers, so teams roll out limits by observing over limit counts first and enforcing only after the numbers look right.
Explained in: Rate limit — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: With enforcement off the over limit counter increases while clients still get 200 responses, and after raising enforcement the same traffic receives 429.
12. Deploy a production-like stack locally: several upstream replicas discovered through strict DNS, two Envoy replicas, local plus global limits, priority thresholds and the retry budget all enabled, then stop one upstream replica and the rate limit service in turn during load.
Capacity is lost in production through failed hosts and failed dependencies, and the full setup must degrade gracefully when either disappears, including a deliberate choice of whether the limit filter fails open or closed.
Explained in: Service discovery — envoy 1.40.0-dev-b8182b documentation (www.envoyproxy.io), checked 2026-10-13
Done when: After a replica is stopped Envoy drops it from the cluster membership and high priority traffic stays healthy, and with the rate limit service down requests follow the failure mode you configured while the local limit still caps the flood.