Practice: Low-latency inference: TensorRT-LLM, quantization and edge deployment

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Main tool: TensorRT-LLM · TensorRT-LLM puts the LLM runtime, the quantized precision choices, the serving endpoint and the latency benchmark behind one configuration file, so each speed versus accuracy trade in this lesson can be changed and measured in one place. · Alternatives: TensorRT compiles a single ONNX graph into a fixed engine for non-LLM models; ONNX Runtime is a portable runtime with CPU, GPU and edge execution providers but less LLM-specific tuning; Triton Inference Server is a multi-framework serving layer that hosts engines rather than optimising them; vLLM is a PyTorch LLM server that is easier to start but less tuned per GPU; llama.cpp runs quantized GGUF models on CPUs and small devices; Jetson and Inferentia are hardware targets with their own toolchains.

Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1

Setup (one time)

1. Check the install prerequisites on a Linux machine with a CUDA-capable GPU; the tool needs one, so on a laptop without it use a single consumer GPU workstation or a remote Linux box you already own as the closest substitute.
The runtime is built for GPU execution and has no CPU-only mode. Knowing the driver, Python and OS requirements first saves a failed install.
Explained in: Installation Guide — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: You have a written list showing your OS, Python version, driver version and GPU memory all meet the listed prerequisites.
2. Install TensorRT LLM using the pre-built release container so the whole toolchain comes pinned.
The container bundles matching versions of the runtime and its GPU libraries. It is also the same image the deployment guides use, so later steps match the documentation.
Explained in: Installation Guide — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: A shell opens inside the release container and the GPU is visible from inside it.
3. Run the sanity check from the installation guide inside the environment.
A passing sanity check proves the package imports and can reach the GPU before you spend time downloading model weights.
Explained in: Installation Guide — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: The sanity check completes without an error and prints generated output.

Steps

1. Run the quick start example of the LLM API with the smallest model that fits your GPU memory.
A first generation end to end confirms model download, loading and decoding all work. Everything later is a variation of this script.
Explained in: LLM API Introduction — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: The script prints generated text for your prompt and exits cleanly.
2. Read the section about TensorRT LLM to learn how the tool turns a model into a runtime tuned for a specific GPU.
The lesson describes compiled engines for GPUs, and you need to know what the current runtime optimises at load time and what it leaves flexible. That tells you which settings are fixed per device.
Explained in: Overview — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: You can explain in three sentences what is tuned for the target GPU and why the result is not portable to other hardware.
3. Write a YAML options file for the LLM API that sets batch size, token limits and the key-value cache memory fraction for your model.
The YAML options file is the core object you version and review: it holds every runtime decision outside the weights. Writing one by hand teaches what each knob controls.
Explained in: Deployment Guide for Qwen3 on TensorRT LLM - Blackwell & Hopper Hardware — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: The model loads with your options file and the startup log shows the values you set.
4. Run offline inference on a fixed set of prompts and save the outputs and wall-clock timings to a file as your full-precision baseline.
Every optimisation in this lesson trades accuracy or flexibility for speed. Without a saved baseline you cannot tell what a later change cost.
Explained in: Quick Start Guide — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: A results file on disk holds the prompts, the generated text and the timing for the baseline run.
5. Load an FP8 quantized checkpoint of the same model, rerun the fixed prompts and compare memory use, timing and outputs against the baseline; where FP4 needs newer data-center GPUs, read that section and stay on FP8 locally.
Quantization lowers the numeric precision of weights and activations to cut memory and compute. Comparing against the baseline shows the real saving and the first signs of output drift.
Explained in: Overview — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: A side by side table shows GPU memory, timing and output differences between the full-precision and FP8 runs.
6. Start an OpenAI-compatible endpoint for the quantized model with the trtllm-serve command and your YAML options file.
Latency targets are set on a served endpoint, not on an offline script. Serving locally lets you measure what a client would see.
Explained in: Quick Start Guide — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: The server log reports it is listening and a chat completion request returns generated text.
7. Query the health check endpoint of the running server and script it as a readiness probe.
Model loading takes minutes, so a deploy needs a signal that separates a starting server from a broken one. The same probe is reused by any orchestrator later.
Explained in: Deployment Guide for Qwen3.8-Flash-Next on TensorRT LLM - Blackwell Hardware — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: The health request returns a success status only after the model has finished loading.
8. Run the documented accuracy evaluation against the served quantized model and against the full-precision model.
Accuracy control means measuring the quality loss from quantization on a benchmark, not judging by eye. The two scores decide whether the lower precision is acceptable.
Explained in: Deployment Guide for GPT-OSS on TensorRT-LLM - Blackwell Hardware — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: You have two evaluation scores recorded and a stated threshold for the accuracy drop you will accept.
9. Run the performance benchmark against the server at several concurrency levels and compare time to first token, per-token latency and throughput.
Right-sizing compute means finding the smallest configuration that still meets the latency target. The curve of latency against concurrency shows where a GPU is saturated or wasted.
Explained in: Deployment Guide for GLM-5 on TensorRT LLM - Blackwell Hardware — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: A table lists time to first token, inter-token latency and token throughput per concurrency level, with the highest level that still meets your latency target marked.
10. Configure an edge-like profile by lowering the key-value cache memory fraction, batch size and sequence length to fit a small memory budget on one GPU, then rerun the benchmark; a real embedded device is not covered, so a memory-capped single GPU is the local substitute.
Edge deployment is defined by a tight memory and power budget. Capping the cache and batch shows how the runtime behaves when the device, not the traffic, is the limit.
Explained in: Deployment Guide for Qwen3 on TensorRT LLM - Blackwell & Hopper Hardware — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: The server starts within your chosen memory cap and the benchmark shows the latency and throughput cost of the smaller budget.
11. Reproduce the deployment from a preconfigured recipe inside the pinned release container and commit the recipe file next to your benchmark results.
A recipe plus a pinned container image makes the measured numbers repeatable by a colleague or a CI job. It removes hand-tuned settings that live only in a shell history.
Explained in: Model Recipes — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: A fresh container started from the recipe alone reaches the same benchmark numbers within normal run to run variation.
12. Reproduce the documented latency-optimised deployment at reduced scale by applying its precision, parallelism and CUDA graph choices to your small model; the full setup needs several data-center GPUs, so one GPU with the same options is the local substitute.
The production write-up shows how precision, parallelism and system-level options are combined for a strict latency target. Applying the same choices yourself links each option to a measured effect.
Explained in: Pushing Latency Boundaries: Optimizing DeepSeek-R1 Performance on NVIDIA B200 GPUs — TensorRT LLM (nvidia.github.io), checked 2026-10-11
Done when: Your final configuration file lists each production option you applied, with a benchmark row showing its effect on latency against the earlier runs.

Back to Low-latency inference: TensorRT-LLM, quantization and edge deployment