Practice: Serving open-weight LLMs with vLLM and SGLang
Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.
Main tool: SGLang · SGLang puts continuous batching, automatic prefix cache reuse, grammar-constrained output, an OpenAI-compatible server and a routing gateway in one engine, so every idea in this lesson can be practised with a single tool. · Alternatives: vLLM (PagedAttention-based engine with the widest community adoption and a similar OpenAI-compatible server), Hugging Face Text Generation Inference (a packaged container server with fewer tuning knobs), NVIDIA Triton with TensorRT-LLM (compiled engines that are faster per model but heavier to build), llama.cpp or Ollama (CPU-friendly quantized serving for a laptop, not built for high-concurrency GPU batching).
Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1
Setup (one time)
1. Install SGLang into a fresh Python virtual environment using the package installer method in the installation page.
A clean pinned environment keeps the engine's heavy dependencies away from other projects and makes the later reproduce steps honest.
Explained in: Installation - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: Importing sglang in the virtual environment succeeds and prints a version number.
2. Start a model server with a small open-weight chat model from the Hugging Face Hub; full speed needs an NVIDIA GPU, so on a laptop without one pick the smallest supported model and expect slow output.
Every later step talks to this running server, and choosing a tiny model keeps download time and memory within laptop limits.
Explained in: Installation - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: The server log reports that it is ready and listening on the local port.
3. Install the SGLang Simulator from the SGLang repository as the no-GPU substitute for load experiments.
The simulator lets you study scheduling and batching behaviour under load on a laptop where a real GPU run is not possible.
Explained in: SGLang Simulator - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: The simulator server starts locally without asking for a GPU.
4. Install the SGLang Model Gateway using the Python package option, after checking its prerequisites.
The gateway is the routing layer used in the registration and production-like steps, so it has to be available before the path reaches them.
Explained in: SGLang Model Gateway - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: The gateway launch command prints its help text listing routing and worker options.
Steps
1. Run your first chat completion against the local server by following the quickstart request example.
A single successful round trip proves the model loaded, the tokenizer works and the HTTP server is reachable before anything else is layered on.
Explained in: Quickstart - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: A JSON response containing generated text and a token usage block is printed in the terminal.
2. Run the same request with streaming enabled and watch the tokens arrive one at a time.
This makes visible why LLM serving differs from classic model serving: generation is autoregressive, the request holds memory for its whole lifetime, and latency splits into time to first token and time per token.
Explained in: Tutorial: Sending a request - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: Output appears incrementally as separate chunks, and you can state the delay before the first chunk versus the gap between later chunks.
3. Query the server through the OpenAI Python client by pointing its base address at the local SGLang endpoint.
An OpenAI-compatible endpoint is what lets application teams swap a hosted model for a self-hosted one without changing client code, which is the contract a platform engineer has to keep.
Explained in: Tutorial: Sending a request - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: The unmodified client library returns a chat completion from the local model.
4. Read the Model Gateway architecture section and sketch how the control plane, data plane and workers relate to the single server you have been running.
Workers, routers and the split between registering capacity and routing requests are the core objects of an SGLang deployment, and the later steps assume this vocabulary.
Explained in: SGLang Model Gateway - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: You have a one-page diagram naming the worker, the router, the control plane and the data plane and what each one owns.
5. Configure the memory and scheduling arguments on a server restart, lowering the static memory fraction and the maximum number of running requests, then compare this design with the paged block allocation that PagedAttention introduced in vLLM.
Both engines exist to pack many concurrent sequences into a fixed key-value cache pool, and changing these arguments shows directly how cache capacity limits concurrency.
Explained in: Server Arguments - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: The startup log shows a different cache pool size for each setting, and you can explain in two sentences how vLLM pages the same memory.
6. Run several requests that share a long identical system prompt, note the cached token counts in the server log, then call the flush cache endpoint and send the request again.
Prefix reuse is the signature SGLang optimisation: a shared prefix is computed once and reused, so flushing the cache shows the cost that reuse was saving.
Explained in: SGLang Native APIs - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: The log shows cached tokens for the repeated prefix before the flush and none on the first request after it.
7. Write a JSON schema for a small record and send a request constrained to it through the OpenAI-compatible API, then compare the output with an unconstrained run of the same prompt saved to a file.
Structured generation guarantees parseable output by masking tokens during decoding, and keeping both outputs side by side shows what the constraint changes.
Explained in: Structured Outputs - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: The constrained response parses against the schema on every run, while the saved unconstrained response fails or drifts at least once.
8. Load a small embedding model on a second port and request embeddings in the compact base64 response format.
Platform teams serve embedding models next to chat models, and the compact encoding shrinks response payloads noticeably for high-volume retrieval traffic.
Explained in: OpenAI APIs - Embedding - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: The response carries a base64 string instead of a float list, and its byte size is smaller than the default response for the same input.
9. Start the gateway in co-launch mode so that it brings up the router and its workers together and registers them behind a single address.
Registering workers with a router is how a model is promoted from a lone process to an addressable service that can gain or lose capacity.
Explained in: SGLang Model Gateway - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: A request sent to the gateway address is answered, and the gateway log names the worker that handled it.
10. Run a concurrent benchmark workload (use the simulator when there is no GPU), then compare throughput and latency before and after adjusting the settings that raise token usage.
Benchmarking and tuning an endpoint means driving it to a target latency and cost, and token usage in the log is the first signal of whether the server is saturated or idle.
Explained in: Hyperparameter Tuning - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: You have a two-row table of throughput and latency for the default and tuned settings, with the token usage figure observed for each.
11. Enable deterministic inference on the server and run the same prompt several times at different levels of concurrent load.
Batch-dependent numerical differences make regression checks flaky, so a deterministic mode is what lets an automated pipeline compare outputs across builds.
Explained in: SGLang for RL Systems - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: Repeated runs of the same prompt return byte-identical output whatever other requests are in flight.
12. Read how llm-d provides a Kubernetes-native gateway for SGLang and map each piece onto a local cluster such as kind or minikube; real GPU nodes are out of reach on a laptop, so treat this as a design exercise.
Kubernetes deployment of an OpenAI-compatible endpoint adds inference-aware routing and scaling on top of the engine, and knowing which component does what comes before writing any manifest.
Explained in: llm-d - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: You can list the Kubernetes resources that would sit in front of your SGLang workers and what each one decides.
13. Deploy a production-like topology locally by launching two workers as separate processes and then starting the gateway separately in front of them over HTTP.
Separating router and worker lifecycles is how real fleets roll out, scale and replace capacity without dropping the public endpoint.
Explained in: SGLang Model Gateway - SGLang Documentation (docs.sglang.io), checked 2026-10-11
Done when: Requests to the gateway are spread across both workers, and stopping one worker leaves the endpoint still answering.