Practice: System design for multi-tenant ML infrastructure: isolation, quotas and backpressure

Generated by Claude through GigAI 0.1.11.10 on 2026-10-12.

Main tool: Kubernetes · Kubernetes ships the isolation, quota, priority and network primitives that every multi-tenant ML platform is built from, so practising them directly shows what the higher-level tools add. · Alternatives: Kueue and Volcano add job queueing and fair-share batch scheduling on top of the cluster, vCluster gives each tenant a virtual control plane instead of a namespace, gVisor sandboxes untrusted containers at the runtime layer, Kyverno and OPA Gatekeeper enforce tenant policy at admission, and Envoy applies rate limits and load shedding at the request layer.

Every step links into the tool's own documentation; links were checked on 2026-10-12 · version 1

Setup (one time)

1. Install a container runtime, kubectl and a local multi-node cluster tool such as kind or minikube, and start a cluster with one control plane node and two workers, using a network plugin that enforces NetworkPolicy.
NetworkPolicy objects are silently ignored when the network plugin does not implement them, so tenant network isolation cannot be practised on a default plugin that lacks support.
Explained in: Network Policies | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: kubectl lists three Ready nodes and the network plugin pods are Running in the system namespace.
2. Confirm that the ResourceQuota admission controller is enabled on the API server of your local cluster.
Quota is enforced at admission time, so if the plugin is off every quota you create later is recorded but never enforced.
Explained in: Resource Quotas | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: The API server admission plugin list, or its default set, includes ResourceQuota.
3. List the initial namespaces of the cluster and set a namespace preference in your kubectl context so you can switch between tenants quickly.
Every later step is scoped to a tenant namespace, and a wrong default namespace is the most common cause of confusing results.
Explained in: Namespaces | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: kubectl shows the initial namespaces and your current context reports the namespace you set.

Steps

1. Create two tenant namespaces, team-a and team-b, plus a shared platform namespace, and label each with a tenant name, following the namespace per tenant model.
The namespace is the base unit of control plane isolation, and comparing it with the virtual control plane model tells you when namespaces alone are not enough.
Explained in: Multi-tenancy | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: kubectl lists the three namespaces with their tenant labels, and you can state one case where a virtual control plane per tenant would be needed instead.
2. Create a ResourceQuota in each tenant namespace that caps total CPU and memory requests and limits, with a larger share for team-a than for team-b.
A compute quota is how a platform turns scarce shared capacity into a predictable per-tenant share.
Explained in: Resource Quotas | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: Describing the quota in each namespace shows the hard values you set and zero used.
3. Run a pod without resource requests in team-a, then a deployment sized to exceed the team-b quota, and query the quota usage after each attempt.
Seeing admission reject the request is the first form of backpressure a tenant meets, and reading used against hard is how you debug it.
Explained in: Resource Quotas | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: The pod without requests is rejected with a quota error, the oversized deployment stalls below its replica count, and the quota shows used close to hard.
4. Create three PriorityClass objects for low, medium and high priority work and a quota per tenant scoped to each priority class, so only a small share of each tenant budget can run at high priority.
Priority-scoped quota stops one tenant from marking every training job as urgent and starving the others during contention.
Explained in: Resource Quotas | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: A high priority pod beyond the scoped quota is rejected while a low priority pod of the same size in the same namespace is admitted.
5. Apply a default deny policy for all ingress and egress in both tenant namespaces, then add a policy that allows traffic only from pods in the same namespace and from the platform namespace.
Namespaces do not isolate network traffic by themselves, so data plane isolation between tenants needs an explicit deny by default.
Explained in: Network Policies | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: A request from a team-a pod to a team-b pod times out, while a request between two team-a pods succeeds.
6. Configure node isolation by tainting and labelling one worker for team-a only, and run a team-a pod with the matching toleration and node selector next to a team-b pod without them.
Dedicated nodes are the stronger and more expensive isolation layer, used for noisy or sensitive workloads such as accelerator training.
Explained in: Multi-tenancy | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: The team-a pod lands on the dedicated worker and the team-b pod is never scheduled there.
7. Create a quota on an extended resource name in team-b to stand in for accelerators, since a laptop has no GPU, and compare the sum of all tenant quotas with the allocatable capacity of your nodes.
Quota is independent of real capacity, so the gap between promised quota and actual nodes is the overcommit trade-off you attribute and charge back per tenant.
Explained in: Resource Quotas | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: You have a small table per tenant of quota hard, quota used and share of node allocatable, and can say whether the cluster is overcommitted.
8. Create a Service with the same name in team-a and team-b, then query its short name and its fully qualified name from a pod in each namespace and compare the answers.
DNS is shared across tenants, so you need to see exactly what a tenant can discover by name even when network policy blocks the traffic.
Explained in: DNS for Services and Pods | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: The short name resolves to the local tenant Service in each namespace, and the fully qualified name of the other tenant resolves but the connection is blocked.
9. Deploy a shared model serving Service in the platform namespace with a readiness probe, make one replica fail readiness to simulate overload, and query the EndpointSlice conditions while it happens.
Removing an overloaded backend from rotation through the ready and serving conditions is the built-in load shedding signal on a shared service.
Explained in: EndpointSlices | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: The EndpointSlice shows the failing replica as not ready while the other replicas keep receiving requests from both tenants.
10. Read the API Priority and Fairness section and note how flow control keeps one tenant with a busy controller or a job storm from exhausting the API server for everyone.
Backpressure on the shared control plane is a different mechanism from quota on workloads, and a design review will ask about both.
Explained in: Multi-tenancy | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: You can explain in two sentences how a flood of API requests from one tenant is queued or rejected without blocking the others.
11. Write a pinned cluster configuration and an admission configuration that limits the high priority class to namespaces holding a matching quota, keep them with all tenant manifests in one repository, then delete the cluster and reproduce it from those files.
Tenant onboarding has to be repeatable from version control, and limiting priority classes by default closes the gap where a new namespace has no quota yet.
Explained in: Resource Quotas | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: A freshly recreated cluster rejects a high priority pod in a namespace without a matching quota, with no manual step after applying the repository.
12. Configure a production-like entry point by installing an ingress controller and creating one Ingress with name based virtual hosting that routes a host per tenant to that tenant's serving Service, with quotas, policies and priority classes all in force.
A real shared platform exposes many tenants behind one edge, and this is where isolation, quota and overload behaviour have to hold together under traffic.
Explained in: Ingress | Kubernetes (kubernetes.io), checked 2026-10-12
Done when: Requests with the team-a host reach only team-a pods, requests with the team-b host reach only team-b pods, and scaling one tenant past its quota does not change the response of the other.

Back to System design for multi-tenant ML infrastructure: isolation, quotas and backpressure