Serving open-weight LLMs with vLLM and SGLang

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: vLLM · also covers SGLang, Large language models · Step 22 of 35 in MLOps / ML platform engineer · version 1

Serving open-weight LLMs is a distinct discipline because generation is autoregressive: each request holds GPU memory for its whole lifetime and produces output token by token. Engines such as vLLM and SGLang exist to pack many concurrent sequences onto a GPU through continuous batching and careful management of the key-value cache. The platform engineer deploys these engines behind an OpenAI-compatible API, benchmarks them and tunes them for a latency and cost target.

How it is done in practice

A platform team packages each engine as a pinned container image, deploys one model per GPU pool, and fronts everything with a gateway that exposes a single OpenAI-compatible surface with per-tenant keys and token-based rate limits. Before a model is offered, it goes through a benchmark run on the target GPU with production-shaped prompts; the result sets replica limits and the autoscaling threshold. Engine upgrades are treated like model releases, with a canary and a comparison of latency percentiles and output quality on an evaluation set.

Technologies

Diagram

sequenceDiagram
  participant C as Client
  participant G as Gateway
  participant E as Engine scheduler
  participant K as KV cache blocks
  C->>G: chat completion with stream
  G->>E: route by model name
  E->>K: look up shared prefix
  E->>K: allocate blocks for prefill
  loop each decode step
    E->>E: admit waiting sequences into batch
    E-->>C: next token
  end
  E->>K: free or keep blocks for reuse

sequenceDiagram
  participant C as Client
  participant G as Gateway
  participant E as Engine scheduler
  participant K as KV cache blocks
  C->>G: chat completion with stream
  G->>E: route by model name
  E->>K: look up shared prefix
  E->>K: allocate blocks for prefill
  loop each decode step
    E->>E: admit waiting sequences into batch
    E-->>C: next token
  end
  E->>K: free or keep blocks for reuse

Code

# vLLM
vllm serve MODEL_ID --port 8000 \
  --max-model-len 8192 --gpu-memory-utilization 0.90

# SGLang
python -m sglang.launch_server --model-path MODEL_ID \
  --host 0.0.0.0 --port 30000

Subtopics

Why LLM serving differs from classic model serving

Autoregression and the KV cache

A classic model runs one forward pass per request. An LLM runs a prefill pass over the prompt and then one decode step per output token, reusing attention keys and values cached from earlier tokens. That cache grows with sequence length and dominates GPU memory after the weights. Requests have unknown and very different lengths, so static batching wastes capacity. Continuous batching fixes this: the scheduler admits new sequences into the running batch at every decode step and evicts finished ones, so the GPU stays full.

Metrics that replace plain latency

LLM endpoints are measured by time to first token, which reflects queueing plus prefill; inter-token latency, which reflects decode speed under the current batch; and total output tokens per second across all requests, which determines cost. These trade against each other: more concurrency raises throughput and worsens per-request token speed. Objectives are therefore written per metric and per workload, for example interactive chat versus offline summarisation.

PagedAttention and vLLM

The memory problem and the paging answer

Earlier systems reserved one contiguous slab of KV cache per request, sized for the maximum possible length. Most of it stayed unused, and free memory fragmented, which capped batch size. PagedAttention, introduced with vLLM, borrows virtual memory paging from operating systems: the cache is divided into fixed-size blocks allocated on demand and mapped through a block table, so blocks need not be contiguous. Waste is limited to the last partial block per sequence. Blocks can also be shared between sequences with copy on write, which helps parallel sampling and beam search. The paper reports throughput gains of roughly two to four times at the same latency against the systems it compared with.

Operating consequences

Because memory is pooled, the engine preallocates a fraction of GPU memory at start and admits requests until blocks run out, at which point requests queue or are preempted. Key settings are the memory fraction given to the engine, the maximum context length, and tensor parallelism across GPUs for models that do not fit on one device.

SGLang: prefix reuse and structured generation

What SGLang adds

SGLang is a serving framework for language and multimodal models with a fast runtime and a set of features aimed at agentic and structured workloads. Its runtime keeps the KV cache of earlier requests and reuses it automatically when a new prompt shares a prefix, which pays off for long system prompts, few-shot templates, multi-turn chat and tool loops. It also provides continuous batching, paged attention, constrained decoding for JSON or grammar-bound output, tensor parallelism and quantized model support.

Running it

The server is started with a launch command that takes a model path, host and port, and is installed with pip, from source, or as a container image. Server arguments control memory fraction, context length, parallelism, the chat template and API keys. It exposes OpenAI-compatible completion and chat endpoints plus a native generate endpoint, so the same client code works against either engine.

OpenAI-compatible endpoints and Kubernetes deployment

The API as the portability layer

Both engines speak the OpenAI wire format for chat completions, completions and streaming via server-sent events. Standardising on it lets a gateway route by model name to different engines, apply auth, quotas and token accounting, and swap runtimes without client changes. On Kubernetes each model is a Deployment or serving resource requesting whole GPUs, with weights pulled from object storage or a cache volume to keep start time down, a readiness probe that passes only after the model is loaded, and a generous termination grace period so that in-flight streams finish. Autoscaling keys on queue depth or running requests, not on CPU.

Benchmarking and tuning an LLM endpoint

A repeatable method

Benchmark with a prompt and output length distribution drawn from the real workload, since synthetic fixed lengths mislead. Sweep concurrency or arrival rate upward and record time to first token, inter-token latency and total tokens per second at each step; the operating point is the highest load at which the latency objectives still hold. Then change one variable at a time: context length cap, memory fraction, quantized versus full weights, number of GPUs per replica, and prefix caching with a shared system prompt. Record engine version, model revision and GPU type with every result, because releases shift numbers noticeably.

Sources

• repoSGLang repository (github.com)
• paperEfficient Memory Management for LLM Serving with PagedAttention (arxiv.org)
• docSGLang Documentation: Install SGLang (docs.sglang.io)
• docSGLang Documentation: Sending Requests (docs.sglang.io)
• docSGLang Documentation: Server Arguments (docs.sglang.io)

Related lessons

• GPU compute on Kubernetes with the NVIDIA stack
• Model serving architecture: inference APIs for batch and real-time demand
• Low-latency inference: TensorRT-LLM, quantization and edge deployment
• Autoscaling and cost efficiency: right-sizing, spot mix and GPU utilization
• LLMOps: evals, tracing, guardrails and AI-assisted development with Claude Code and Cursor
• Integrating hosted LLMs: AWS Bedrock, the Claude API and prompt engineering