Caching

Generated by Claude through GigAI (karthik446/gigai-learning-center 4c0cb514) on 2026-10-13.

Concept: Caching · also covers Low latency, Scaling · Lesson 6 of 31 in System design · version 1

A cache trades freshness for speed and reduced load by keeping a copy of data closer to where it is read. The design work is not in adding the cache but in deciding how it is filled, how it goes stale, what gets evicted and what happens when it is empty or down. A cache that the system cannot survive without has become a hidden dependency.

How it is done in practice

A team adds a cache only after measuring that a dependency is too slow or too expensive, and writes down the allowed staleness for each cached object. The usual first step is cache-aside against a managed Redis cluster with a time to live plus jitter and delete on write. They watch hit ratio, latency, memory use and evictions on dashboards, and alert on sudden drops in hit ratio. Before launch they run a load test with the cache flushed to confirm the backend survives a cold start, and they wrap cache calls in short timeouts so the service degrades to direct reads instead of failing.

Simulate it

The database only ever sees the cache misses, so a good hit rate is the cheapest capacity there is. At the defaults it receives about 29 requests a second of the 200 offered and is about 22% busy.

1. Leave the controls alone and read the database utilisation: about 22%, steady.
2. Set the cache hit rate to 0.3 and wait ten seconds.
3. The database saturates at 100% busy and about 4% of requests start to fail.
4. Put the hit rate back to 0.85 and raise the offered load to 1200: the database saturates again, this time from volume.
5. At 1200 a second, a hit rate of 0.95 brings the database back to about 46% busy.

Technologies

Diagram

flowchart TD
    A[Request] --> B{Key in cache?}
    B -- hit --> C[Return cached value]
    B -- miss --> D[Read from database]
    D --> E[Set key with TTL plus jitter]
    E --> C
    W[Write request] --> X[Update database]
    X --> Y[Delete cache key]

flowchart TD
    A[Request] --> B{Key in cache?}
    B -- hit --> C[Return cached value]
    B -- miss --> D[Read from database]
    D --> E[Set key with TTL plus jitter]
    E --> C
    W[Write request] --> X[Update database]
    X --> Y[Delete cache key]

Code

import json, random

def get_user(r, db, user_id):
    key = f'user:{user_id}'
    cached = r.get(key)
    if cached is not None:
        return json.loads(cached)
    user = db.load_user(user_id)          # source of truth
    ttl = 300 + random.randint(0, 60)     # jitter against mass expiry
    r.set(key, json.dumps(user), ex=ttl)
    return user

def update_user(r, db, user_id, fields):
    db.save_user(user_id, fields)
    r.delete(f'user:{user_id}')           # invalidate, do not overwrite

# redis.conf for a pure cache
maxmemory 2gb
maxmemory-policy allkeys-lru

Subtopics

Cache-aside and other read and write patterns

Cache-aside

In cache-aside the application owns the logic: read the cache, and on a miss load from the source of truth, store the result with a time to live and return it. On a write, update the database and then delete the cached key, so the next read repopulates it. Deleting is safer than updating the cache in place, because two concurrent writers can otherwise leave the older value in the cache. Even so, a slow reader can write a stale value after an invalidation, which is why a time to live remains as a backstop.

Alternatives

Read-through moves the loading logic into the cache layer or a library, so callers only see a get. Write-through writes to the cache and the store together, keeping the cache warm at the cost of slower writes and caching data that may never be read. Write-behind acknowledges after writing the cache and flushes to the store asynchronously, which is fast but loses data if the cache node dies before the flush. Most services start with cache-aside because it fails safely: a cache outage degrades to slower reads.

Expiry, TTLs and invalidation

Bounding staleness

A time to live sets the maximum age of an entry and is the simplest staleness control. Pick it from the business tolerance for stale data, not from habit: seconds for prices or permissions, minutes or hours for profile data or rendered fragments. Add random jitter to the value so that entries created together do not all expire in the same instant. In Redis a key gets an expiry with EXPIRE or with SET and the EX option, and the remaining time can be read with TTL.

Explicit invalidation

When staleness must be shorter than any practical time to live, invalidate on change: delete the key when the source record is written, or publish change events that a consumer uses to evict keys. Versioned keys are another option: include a version or updated timestamp in the key so that new data lands under a new key and old entries simply age out. Invalidation is hard when one change affects many derived entries, such as lists and aggregates, so keep cached objects close to the shape of the source records where possible.

Eviction policies and memory sizing

What happens when memory is full

A cache has a memory limit, and the eviction policy decides which keys are removed when it is reached. Redis sets the limit with maxmemory and the behavior with maxmemory-policy. The noeviction policy rejects writes that need more memory. The allkeys policies consider every key, using approximated least recently used, least frequently used or random selection. The volatile policies consider only keys that have an expiry set, and volatile-ttl prefers keys closest to expiring. For a pure cache an allkeys policy is the usual choice.

Sizing

Size the cache from the working set, not the whole data set: access is usually skewed, so a small fraction of keys serves most requests. Estimate entries times average value size plus per key overhead, then validate with the hit ratio and eviction rate in production. A rising eviction rate with a falling hit ratio means the cache is too small or keys are too large. Do not mix durable data and cache entries in one instance with an allkeys policy, because eviction will silently delete the durable keys.

Stampedes, cold starts and cache as a dependency

Failure modes

A stampede happens when a hot key expires and many requests miss at once, all recomputing the same value and overloading the backend. A cold start is the same effect across all keys after a restart or flush. Negative lookups cause a third problem: requests for keys that do not exist always miss and always hit the database. The deeper risk is modal behavior: a system sized for a high hit ratio may not survive the load when the cache is empty, so a cache failure becomes a full outage.

Mitigations

Coalesce requests so only one caller per key recomputes while others wait or get the old value. Refresh hot entries before they expire, or serve stale data while a background refresh runs. Cache negative results with a short time to live. Use jitter on expiry. Put timeouts and a circuit breaker on cache calls so a slow cache does not stall every request. Finally, load test with the cache empty and make sure rate limiting protects the backend in that state.

Local, distributed and edge caches

Where the cache lives

An in-process cache is the fastest, with no network hop, but each instance has its own copy, so entries are inconsistent between instances, lost on every deploy and multiplied in memory by the instance count. A distributed cache such as Redis or Memcached is shared, survives application deploys and gives one place to invalidate, at the cost of a network round trip and a new service to operate. Edge caches and CDNs sit in front of the service and cache whole responses, controlled by HTTP cache headers. Many systems layer them: a small short lived local cache in front of a shared cache in front of the database.

Sources

• docRedis key eviction (redis.io)
• docRedis keyspace: expiry and TTL (redis.io)
• articleCaching challenges and strategies (Amazon Builders' Library) (builder.aws.com)
• docCache-Aside pattern (Azure Architecture Center) (learn.microsoft.com)

Related lessons

• Consistency models, transactions and consensus in plain words
• Rate limiting, overload and resilience patterns
• Capacity planning and performance
• Latency, throughput and back-of-envelope numbers
• Observability for designers
• Cost as a design constraint