Design exercise: file storage and sync

Generated by Claude (Fable 5.1) by hand, Scout learning center M4a on 2026-10-12, reviewed by a second reader (Claude Fable 5.1), Scout learning center M4b on 2026-10-12.

Concept: System design and architecture · also covers Storage systems, Distributed systems, Relational and non-relational data stores · Part C (Design exercises) of the system design module · design exercise · version 2

A classic design exercise: a folder-sync service at hundreds of petabytes with version history, sharing and resumable uploads. The reference answer splits files into content-addressed 4 MB blocks, keeps a strongly consistent metadata namespace per account, runs a commit protocol that reports missing blocks, stores hot blocks replicated and cold blocks erasure coded, and treats repair time, soft deletion and restore tests as part of durability.

Attempt the design from the prompt, the constraints and the questions before reading the reference answer.

The prompt

Design a file storage and sync service: users install a client that keeps a folder in sync across their devices and the web, share folders with other users, and can recover earlier versions of a file. Uploads must survive flaky connections and the service must never lose a file it has acknowledged.

Constraints and numbers

Functional

Upload, download and delete files and folders; sync changes to every device of the account within seconds; version history for 30 days (assumption); share a folder with other accounts; resumable uploads; web access.

Scale (assumption)

50 million users, 10 million active per day; an average of 5 GB stored per user, so 250 PB of logical data; 100 million file changes per day with an average file of 1 MB; 30 percent of uploaded bytes are duplicates of blocks already stored (assumption). Chosen for the exercise; derive the ingest rate, the physical storage per durability scheme and the metadata volume yourself.

Targets (assumption)

Durability designed so that losing a file requires several simultaneous disk or node failures, with a stated repair time; 99.9 percent availability for sync; a change visible on other devices within 10 s at p95; a 1 MB file uploads in under 2 s at p95 on a good connection.

Non-goals

Collaborative editing of file contents, search inside files, virus scanning, end-to-end encryption (server-side encryption at rest is assumed).

Clarifying questions worth asking

• What is the unit of storage? A block: files are split into fixed-size blocks of up to 4 MB, each named by its content hash, so identical blocks are stored once. Dropbox's public description of Magic Pocket uses exactly this unit.
• What happens when two devices edit the same file offline? The second commit sees a version mismatch and the client saves a conflicted copy beside the original; nothing is merged and nothing is lost.
• Is the file namespace strongly consistent? Yes, per account: a rename or a delete is linearizable within the account's metadata shard. Cross-account sharing is eventually consistent (seconds).
• How long are deleted files kept? 30 days in a soft-deleted state, then blocks are unreferenced and garbage collected (assumption).
• Hot or cold data? Most bytes are written once and rarely read; most reads are of recent files. Storage is tiered by age and access.
• Multi-region? Metadata lives in one region per account; blocks are replicated across regions for disaster survival. The reference answer stays single-region for the request paths.

Reference answer

Requirements and estimates

Derived numbers: changes and ingest

Changes: 100 million per day is about 1,160 per second average, 11,600 at peak (10x). Ingest: 100 million files times 1 MB is 100 TB per day, 1.2 GB per second average, 12 GB per second at peak; after 30 percent block deduplication, 70 TB per day of new physical blocks. Metadata: a file version record of about 500 bytes plus a block list of 32-byte hashes per 4 MB block; 100 million versions per day is 50 GB per day of metadata, 18 TB per year, sharded by account.

Derived numbers: durability arithmetic

Logical data is 250 PB. Three-way replication costs 3 bytes per byte, 750 PB of disk. An erasure code of 10 data fragments plus 4 parity fragments (assumption; the exact parameters are a choice) costs 1.4 bytes per byte, 350 PB, and survives any 4 lost fragments of a stripe; erasure coding of cold data is what both Dropbox's Magic Pocket write-up and the S3 write-up describe, with their own parameters. The difference, 400 PB of disk, is the business case for erasure coding cold data; hot data stays replicated because a replicated block is read from one disk, while an erasure-coded block needs several fragments reassembled.

The latency numbers

A 4 MB block from SSD is about 4 ms of sequential read (1 ms per MB from the latency reference numbers) and from disk about 80 ms plus a 10 ms seek; serving recent blocks from SSD and old blocks from disk is what makes both the download target and the storage cost work.

High-level design

Container view

The client does the hard work: it splits files into blocks, hashes them, and computes which blocks the server lacks. The metadata service owns the namespace (files, folders, versions, block lists, sharing) in a strongly consistent store sharded by account. The block service stores immutable, content-addressed blocks in a hot replicated tier and a cold erasure-coded tier, with repair and tiering workers behind it. A change feed per account drives a notification service that tells other devices to sync. The Google File System paper is the ancestor of this split: a small metadata master with chunk servers holding large chunks, three replicas by default.

flowchart LR
  C[Desktop and mobile clients: chunk, hash, delta] --> MS[Metadata service]
  MS --> MD[(Metadata DB: namespaces, files, versions, block lists; sharded by account)]
  C -->|PUT missing blocks| BS[Block service]
  BS --> HOT[(Hot block store: replicated, SSD)]
  BS --> COLD[(Cold block store: erasure coded, disk)]
  MS --> N[[change feed per account]]
  N --> NS[Notification service: long poll or push]
  NS --> C
  MS --> SH[Sharing and permissions]
  BS --> REP[Repair and tiering workers]

flowchart LR
  C[Desktop and mobile clients: chunk, hash, delta] --> MS[Metadata service]
  MS --> MD[(Metadata DB: namespaces, files, versions, block lists; sharded by account)]
  C -->|PUT missing blocks| BS[Block service]
  BS --> HOT[(Hot block store: replicated, SSD)]
  BS --> COLD[(Cold block store: erasure coded, disk)]
  MS --> N[[change feed per account]]
  N --> NS[Notification service: long poll or push]
  NS --> C
  MS --> SH[Sharing and permissions]
  BS --> REP[Repair and tiering workers]

Boxes are containers; cylinders are stores; double brackets are logs. Metadata and blocks are separate systems with separate consistency rules.

The data model

Entities

Account {account_id, quota, metadata_shard}. Node {account_id, node_id, parent_id, name, type (file or folder), current_version, deleted_at}. Version {node_id, version, size, block_hashes[], modified_at, device_id}. Block {hash (SHA-256 of content), size, tier, locations[], refcount}. Share {folder_node_id, member_account_id, role}. Cursor {account_id, device_id, last_change_seq}. ChangeFeed {account_id, seq, node_id, version}.

API

POST /commit {node_id, parent_version, block_hashes[]} returns 200 {version} or 409 {missing_blocks[]} or 412 (version conflict); PUT /blocks/{hash} with the block body (idempotent: the hash is the name); GET /blocks/{hash}; GET /changes?cursor= returns the account's changes since the cursor; a long-poll GET /notify?cursor= that returns when there is something new. Sharing endpoints change membership and write a change into every member's feed.

The core flow, step by step

A user saves a modified 10 MB file on a laptop; the phone gets it.

1. The client detects the change, splits the file into three 4 MB blocks (the last one smaller), hashes each, and compares with the previous version's block list: one block changed, so only one new hash. This is delta sync; the cost of a small edit is one block, not the file.
2. The client calls commit with the new block list and the parent version. The metadata service checks the parent version (the optimistic check that detects conflicts) and the block store for each hash; it answers 409 with the one missing block.
3. The client uploads the missing block by its hash. The upload is idempotent and resumable: an interrupted PUT is simply repeated, and a block that already exists (another user uploaded the same bytes) is acknowledged without transfer, which is where the deduplication comes from.
4. The client commits again; now every block exists, the service writes the new version and node record in one transaction on the account's shard, appends a change to the account's feed, and returns the version. The acknowledgement means the file is durable: blocks are replicated before the block service acknowledges a PUT, and the metadata write is synchronously replicated.
5. The phone's long poll returns; it fetches the change list, sees the new version of the file, and downloads only the block it does not already have.
6. Later, a tiering worker moves blocks not read for 30 days (assumption) into erasure-coded stripes; a garbage collector unreferences blocks whose versions passed the retention window and whose refcount is zero.

Scaling and the bottleneck

Storage scale and repair

Bytes are the bottleneck, and they scale horizontally: block servers are added by capacity, and placement spreads a block's replicas or fragments across failure domains (disk, rack, zone). Repair is the subtle part: when a disk dies, every stripe or replica set it held must be rebuilt from the survivors before another failure in the same set, so the design states a repair time (hours, not days) and sizes repair bandwidth for it; durability is a function of failure rate and repair time, not of replica count alone. The SRE book's data integrity chapter adds the layer most designs forget: replication protects against hardware loss, not against a bug or an operator deleting the wrong thing, so soft deletion, backups of metadata and regular restore tests are part of the design, and the restore is the real test.

Metadata consistency and sync

Metadata is small but hot and must be consistent: 11,600 commits per second at peak across shards keyed by account, each a transaction on one shard, with no cross-shard transactions because a file belongs to one account's namespace; sharing is modelled as a membership record plus feed fan-out, not as a cross-shard write. The change feed per account is a log with a monotonic sequence, which makes the sync cursor simple: a device asks for everything after its cursor and is never told the same change twice. Notification is long poll by default and push on mobile; both are hints, since the cursor fetch is what guarantees completeness.

Failure modes and what degrades

Each row names the failure, what the user sees, and the mitigation.

Trade-offs and the alternative not taken

The decisions a reviewer reads first.

What changes at 10x

At 2.5 exabytes the storage is a platform: geographic placement of stripes across regions, lifecycle policies that move data between tiers by age and access and delete by policy, continuous repair as steady-state load with a measured cost per TB per month, and heat management across disks so that popular blocks do not sit on a few spindles (the S3 write-up describes this). Metadata grows to petabytes and gets its own tiering (old versions to cheaper stores). Large accounts (an enterprise with millions of files) outgrow one shard and need a namespace split by folder subtree. Verification becomes a background job that reads every block on a schedule and compares hashes, because at that size silent corruption is a certainty, not a risk.

Self-check rubric

Read, not scored: find the level your answer reached.

L1, mid-level: applies

• Chunked, content-addressed blocks in an object store, with a separate metadata service that owns files, versions and block lists.
• A commit protocol that reports missing blocks and makes uploads idempotent and resumable.
• The ingest and storage arithmetic from the stated assumptions (100 TB per day, 250 PB logical).

L2, senior: chooses with numbers and defends

• Delta sync by block list diff, conflict detection by parent version with conflicted copies, and a cursor-driven change feed per account.
• Deduplication by hash and what it saves; the durability arithmetic of replication versus erasure coding with a chosen code.
• Repair time as part of durability, and failure-domain-aware placement.
• Soft deletion, metadata backups and restore tests as defence in depth beyond replication.

L3, staff: sets the constraint and judges

• Tiering and lifecycle as a cost model per TB per month, with heat management and continuous repair as steady-state load.
• Geographic placement and the consistency contract per data class (metadata per account, blocks everywhere, sharing eventual).
• Background verification of every block, and the namespace split for accounts that outgrow a shard.

The 45-minute interview frame

• 0:00 to 0:05 Requirements: the storage unit, offline conflicts, namespace consistency, retention, hot versus cold, regions. Say the numbers aloud: 11,600 commits per second at peak, 100 TB ingested per day, 250 PB logical, 350 PB on disk with a 10-plus-4 code versus 750 PB replicated.
• 0:05 to 0:10 Entities and API: Node, Version with a block list, Block by hash, Share, Cursor and ChangeFeed; the commit, block PUT, changes and notify endpoints.
• 0:10 to 0:20 High-level design with the diagram: clients that chunk and hash, the metadata service on account shards, the block service with hot and cold tiers, the change feed and notification path.
• 0:20 to 0:40 Two deep dives: the sync protocol (commit with missing blocks, idempotent block uploads, delta sync, version conflicts, the cursor), then durability (replication versus erasure coding arithmetic, placement, repair time, soft delete and restores).
• 0:40 to 0:45 Failure modes from the table, the 10x plan (lifecycle, geography, verification, namespace splits), and the L3 sentence: durability is a function of failure rate and repair time, and the restore test is the only proof.

The design-doc frame

• Context and scope. The client platforms, the existing object store if any, the retention and sharing requirements, and the incident or cost that motivates the design.
• Goals and non-goals. Never lose an acknowledged file, sync within 10 s, 30-day history, resumable uploads; no collaborative editing, no content search, no end-to-end encryption.
• Design. This answer fills it: the block and metadata split, the commit protocol, the tiers, the change feed.
• Alternatives considered. Whole-file storage, replication everywhere, an eventually consistent namespace, last-writer-wins conflicts, timer polling, with the trade-off table.
• Cross-cutting concerns. Security (per-account authorization on every node, sharing roles, encryption at rest), privacy (hash-based dedupe across accounts and what it reveals), observability (under-replicated blocks, repair backlog, commit latency, sync lag per device), cost per TB per month by tier, rollout by account cohort with the old store as fallback.
• Review questions the doc must answer. How long a block may be under-replicated after a disk failure and what bandwidth repair needs, and what two devices see after editing the same file offline.

Depth notes

• L1. Chunked upload, content-addressed blocks, a metadata service and an object store.
• L2. Sync protocol with deltas and conflicts, deduplication, resumable uploads, durability arithmetic (replication versus erasure coding).
• L3. A storage platform (tiering, lifecycle, geo placement, repair, cost per TB).

Sources

• blogDropbox Tech: Inside the Magic Pocket (dropbox.tech)
• paperThe Google File System (SOSP 2003) (static.googleusercontent.com)
• blogAll Things Distributed: Building and operating a pretty big storage system called S3 (www.allthingsdistributed.com)
• bookSite Reliability Engineering: Data integrity (sre.google)
• docLatency numbers every programmer should know (gist.github.com)

Related lessons

• Storage engines and indexes sd-storage-engines planned
• Replication and partitioning sd-replication-partitioning planned
• Consistency models, transactions and consensus in plain words sd-consistency-consensus planned
• Data modelling across stores sd-data-modelling planned
• Cost as a design constraint sd-cost planned