Design exercise: RAG service (retrieval-augmented generation)

Generated by Claude (Fable 5.1) by hand, Scout learning center M4a on 2026-10-12, reviewed by a second reader (Claude Fable 5.1), Scout learning center M4b on 2026-10-12.

Concept: System design and architecture · also covers API design, Relational and non-relational data stores, Scaling · Part C (Design exercises) of the system design module · design exercise · version 2

An AI-era design exercise: an internal question-answering service over a company's documents with citations, strict permissions and per-team evaluation. The reference answer sizes chunks, vectors and daily tokens from stated assumptions, uses hybrid retrieval with contextual chunks and a reranker, filters by permission before the model sees text, and treats the vector index as a rebuildable derived view. Converted by hand from the module outline's reference answer; every number and source is kept.

Attempt the design from the prompt, the constraints and the questions before reading the reference answer.

The prompt

Design an internal question-answering service over a company's documents (wikis, tickets, PDFs). A user asks a question and gets an answer with citations to the documents it came from. Several teams will put their own document sets on it.

Constraints and numbers

Functional

Ingest documents from several sources with incremental updates; answer a question with citations; respect document permissions (a user only gets answers from documents they may read); teams can evaluate answer quality on their own question sets.

Scale (assumption)

2 million documents, average 20 kB of text, growing 2 percent per month; 20,000 questions per day, peak 2 per second; 50 teams. Chosen for the exercise; derive the chunk count, the vector volume and the daily token volume yourself.

Targets (assumption)

First token of the answer within 2 s at p50 and 5 s at p95; retrieval under 300 ms at p95; new or changed documents searchable within 15 minutes; model spend bounded per team.

Non-goals

Training or fine-tuning models, agents that take actions, chat memory beyond one conversation.

Clarifying questions worth asking

• Which model provider, and can it change? Hosted provider now, must be swappable; the service calls it through a gateway (see the LLM gateway exercise).
• How strict are permissions? Strict: a retrieved chunk is filtered by the asker's access before it reaches the model; permission data comes from the source systems.
• What counts as good? Teams define question sets with expected sources; retrieval recall and answer faithfulness are measured.
• Languages? English first.
• Can documents be deleted? Yes; deletion must remove chunks and embeddings within the same 15 minutes.
• Budget? Stated per team per month in tokens.

Reference answer

Requirements and estimates

Derived numbers: documents, chunks, vectors

Text: 2 million times 20 kB is 40 GB. Chunks: at about 500 tokens (roughly 2 kB) per chunk, 10 chunks per document, 20 million chunks. Embeddings: at 1,024 dimensions of 4-byte floats, 4 kB per vector, 80 GB of vectors before compression; an HNSW index adds graph memory on top, so a single machine serving the whole index in memory is large; sharding by team (most teams are small) and quantization (the FAISS wiki) bring it down.

Derived numbers: the query path and tokens

Embed the question (one model call of a few hundred tokens), retrieve top 50 by vector plus top 50 by keyword, rerank to 8, build a prompt of about 6,000 tokens, generate about 400 tokens. At 20,000 questions per day that is roughly 120 million input tokens and 8 million output tokens per day across the fleet (assumption), which is the dominant cost and the reason for per-team budgets and prompt caching of the fixed instructions (the Claude prompt caching documentation).

Derived numbers: ingestion

2 percent per month of 2 million is 40,000 documents per month, about one per minute on average with bursts after bulk imports.

High-level design

Container view

Two pipelines share a document store. Ingestion turns document versions into contextualised chunks, vectors and keyword entries per collection. Query embeds the question, retrieves by vector and keyword in parallel, filters by permission before reranking, and streams a generation through the LLM gateway, logging everything an evaluation needs.

flowchart TB
  subgraph Ingestion
    SRC{{Wikis, tickets, PDFs}} --> CON[Connectors, incremental]
    CON --> QI[[document change log]]
    QI --> CH[Chunk + contextualise]
    CH --> EMB[Embedding calls via gateway]
    EMB --> VS[(Vector index per collection)]
    CH --> KW[(Keyword index)]
    CON --> META[(Documents, ACLs, versions)]
  end
  subgraph Query
    U[User] --> API[Ask API]
    API --> PERM[Permission filter]
    API --> RET[Retriever: vector + keyword]
    RET --> VS
    RET --> KW
    RET --> RR[Reranker]
    RR --> GEN[Prompt builder + generation via gateway]
    GEN --> GW{{LLM gateway}}
    GEN --> LOG[(Question log + evals)]
  end
  PERM --> META

flowchart TB
  subgraph Ingestion
    SRC{{Wikis, tickets, PDFs}} --> CON[Connectors, incremental]
    CON --> QI[[document change log]]
    QI --> CH[Chunk + contextualise]
    CH --> EMB[Embedding calls via gateway]
    EMB --> VS[(Vector index per collection)]
    CH --> KW[(Keyword index)]
    CON --> META[(Documents, ACLs, versions)]
  end
  subgraph Query
    U[User] --> API[Ask API]
    API --> PERM[Permission filter]
    API --> RET[Retriever: vector + keyword]
    RET --> VS
    RET --> KW
    RET --> RR[Reranker]
    RR --> GEN[Prompt builder + generation via gateway]
    GEN --> GW{{LLM gateway}}
    GEN --> LOG[(Question log + evals)]
  end
  PERM --> META

Boxes are containers; the two frames are the ingestion and query boundaries; hexagons are external systems; cylinders are stores; double brackets are logs.

The data model

Entities

Collection {id, team, source connectors, permission model}; Document {id, collection, source_uri, version, acl, updated_at}; Chunk {id, document, position, text, context_prefix, embedding, keyword index}; Question {id, user, collection(s), text, retrieved chunk ids, answer, citations, latency, tokens}; EvalSet {team, questions with expected document ids}.

API

POST /ask {question, collections[]} returns a stream of tokens, then {citations[]}; POST /collections/{id}/sync (connector-driven, idempotent per document version); POST /evals/{set}/run returns {recall@k, faithfulness, cost}.

The core flow, step by step

Ingestion path

Connectors emit document versions onto a change log (the queues and streams lesson; the outbox and CDC lesson when the source is a database). A worker chunks the text, prepends a short context sentence to each chunk (the contextual retrieval technique: the public write-up reports a 49 percent reduction in retrieval failures for contextual embeddings plus contextual BM25 and 67 percent with reranking added), embeds in batches, writes vectors to the collection's index and text to the keyword index, and records the document version. A new version replaces the old chunks atomically per document (write new, switch the version pointer, delete old); a deletion removes chunks and vectors and is verified by a periodic reconciliation between the metadata store and the indexes. Permissions are stored per document from the source system's ACL and refreshed by the connector.

Query path

The query path, in order:

1. Embed the question.
2. Run the vector query and the keyword query in parallel against the collections the user named.
3. Drop any chunk whose document the user may not read (filter before reranking, so the model never sees forbidden text).
4. Rerank the union to 8 chunks.
5. Build the prompt with fixed instructions first (cacheable), then the chunks with ids, then the question.
6. Stream the generation; extract citations as the chunk ids the model referenced and map them to document URIs.
7. Log everything needed for evaluation (retrieved ids, answer, tokens, latency) with the question text retained under the team's policy.

Scaling and the bottleneck

Retrieval quality as an engineering loop

The Azure guide structures a RAG build as chunking, enrichment, embedding, retrieval evaluation and end-to-end evaluation. The service makes that a product feature: each team keeps an evaluation set (questions with the documents that should be cited); a run reports recall at k for retrieval and a faithfulness score for the answer; a change to chunking, embedding model or reranker is tested against every team's set before rollout, the way a schema migration is tested. The L3 insight is that the vector index is a derived view that can be rebuilt; the source of truth is the document store with versions, so re-embedding with a new model is a backfill (the migrations lesson), done per collection with the old and new index served side by side until the evals pass.

Tenancy, permissions and cost

One index per collection (team) keeps blast radius and cost attribution simple and makes deletion of a team's data a drop of an index; a few very large collections get sharded indexes with scatter-gather across shards. Permission filtering happens on chunk metadata before reranking; for sources with fine-grained ACLs the filter is a join against the document ACL table, which is why document ids and ACL versions are carried on every chunk. Cost: per-team token budgets enforced at the gateway, prompt caching for the instruction prefix, a smaller model for the reranker, and a per-team dashboard of cost per answered question. Latency: retrieval under 300 ms is the index plus the keyword query plus the rerank; the generation dominates and streams, so the first token target is met by starting generation as soon as the prompt is built.

When RAG is not the answer

The staff-level question is whether the service is needed at all, and the answer is no in four common cases. When the need is style, format or a fixed vocabulary rather than facts that change, fine-tuning a model once is cheaper than retrieving on every question. When users want the documents themselves, a plain search index with good ranking serves them faster than a generated summary and never misquotes. When the question set is closed and small (a product FAQ, a policy with twenty answers), a lookup table or a static page answers exactly and costs nothing per question. When the knowledge is public and stable (a standard, a language, well-known history), the model already has it and retrieval adds latency and a citation that proves nothing. The test this service applies before a team onboards a collection: the answers must depend on documents that are private, permissioned or fresh; if none of those holds, the team is pointed at the cheaper thing.

Failure modes and what degrades

Each row names the failure, its effect on answers, and the mitigation.

Trade-offs and the alternative not taken

The decisions a reviewer reads first.

What changes at 10x

200 million chunks: quantized indexes, sharded by collection then by hash, index build separated from serving with immutable segments; retrieval becomes its own service (the vector search exercise). Multi-language: per-language embedding models and keyword analysers, selected per document. Conversation memory: a short per-conversation context stored by the ask API, not by the index.

Self-check rubric

Read, not scored: find the level your answer reached.

L1, mid-level: applies

• The ingestion pipeline (chunk, embed, index) and the query pipeline (retrieve, prompt, generate) with citations.
• A vector store, permission filtering somewhere, and a latency budget.

L2, senior: chooses with numbers and defends

• Hybrid retrieval and reranking with the reason; filtering before the model sees text; per-collection indexes.
• Incremental updates and deletions with reconciliation; an evaluation set per team.
• Token cost arithmetic and prompt caching; streaming for first-token latency; the embedding-model-change backfill.

L3, staff: sets the constraint and judges

• The vector index as a rebuildable derived view; evaluation as a release gate across teams.
• Provider abstraction and fallback; tenancy and cost attribution as the platform questions.
• What the service must refuse (answers without citations, questions over collections the user cannot name), and when the right answer is not RAG at all (a small, static FAQ; or search without generation).

The 45-minute interview frame

• 0:00 to 0:05 Requirements: the six clarifying questions, the non-goals, and the numbers aloud: 20 million chunks, 80 GB of vectors, 120 million input tokens per day (all assumptions), 2 questions per second at peak, 15-minute freshness.
• 0:05 to 0:10 Entities and API: Collection, Document, Chunk, Question, EvalSet; the streaming ask endpoint, the idempotent sync endpoint, the evaluation run.
• 0:10 to 0:20 High-level design with the diagram: the two pipelines, one index per collection, the permission filter before the reranker, the gateway in front of the model, the question log that feeds evaluation.
• 0:20 to 0:40 Two deep dives: retrieval quality as an evaluation loop (contextual chunks, hybrid retrieval, rerank, per-team eval sets as a release gate, re-embedding as a backfill), then permissions and tenancy (filter by ACL version before rerank, per-collection indexes, deletion, per-team budgets and cost per answer).
• 0:40 to 0:45 Failure modes from the table, the 10x plan, and the L3 sentence: the index is a derived view; the design is really about the document store, the ACLs and the evaluation loop.

The design-doc frame

• Context and scope. The document sources, the teams, the permission systems of record, and the model provider in use today.
• Goals and non-goals. Cited answers, strict permissions, 15-minute freshness, bounded spend per team; no fine-tuning, no agents, no long-term memory.
• Design. This answer fills it: both pipelines, the data model, the diagram and the query path.
• Alternatives considered. Index layout, permission enforcement, vector store, with the trade-off table.
• Cross-cutting concerns. Privacy of question logs, cost per team, rollout per collection, observability of retrieval recall and freshness.
• Review questions the doc must answer. The permission filter's correctness under ACL change, and the re-embedding plan when the embedding model changes.

Depth notes

• L1. An ingestion pipeline (chunk, embed, index) and a query path (retrieve, rerank, generate) with citations.
• L2. Hybrid retrieval, contextual chunking, freshness and re-indexing, per-tenant isolation of documents, evaluation harness, latency and token budgets.
• L3. A RAG platform for many teams (index ownership, access control at retrieval, evaluation gates in CI, cost per answer, provider abstraction).

Sources

• paperRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al.) (arxiv.org)
• docAzure Architecture Center: RAG solution design and evaluation guide (learn.microsoft.com)
• docGoogle Cloud Architecture Center: infrastructure for a RAG-capable generative AI application (docs.cloud.google.com)
• blogAnthropic: Introducing Contextual Retrieval (www.anthropic.com)
• repoFAISS wiki: index types and guidelines to choose an index (github.com)
• repopgvector: vector similarity search for Postgres (github.com)
• paperEfficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (HNSW) (arxiv.org)
• docClaude Platform docs: Prompt caching (platform.claude.com)
• docClaude Platform docs: Rate limits (platform.claude.com)
• docAWS Well-Architected: Generative AI Lens (docs.aws.amazon.com)

Related lessons

• Queues, streams and delivery guarantees sd-queues-streams planned
• Storage engines and indexes sd-storage-engines planned
• Data modelling across stores sd-data-modelling planned
• Caching sd-caching planned
• Rate limiting, overload and resilience patterns sd-rate-limiting-overload planned
• Observability for designers sd-observability planned
• Cost as a design constraint sd-cost planned
• Design exercise: LLM gateway
• Design exercise: vector search service