Model serving

model-serving · Machine learning, LLMs and agents · responsibility

Also known as: model serving, serving infrastructure, serving, serving llms.

Narrower concepts

• SGLang
• TensorRT-LLM
• vLLM

The lesson for this

• Model serving architecture: inference APIs for batch and real-time demand

Pathways that reach it

• MLOps / ML platform engineer