Practice: Workload scheduling, capacity management and GPU cluster operations

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Main tool: Kubernetes · Every batch system in this lesson either extends or is contrasted with the Kubernetes scheduler and Job API, so practising on plain Kubernetes shows exactly which gaps the batch systems fill. · Alternatives: Kueue adds job queues, team quotas and fair sharing on top of the default scheduler; Volcano replaces or supplements the scheduler with its own gang-scheduling batch scheduler; Slurm is a standalone HPC scheduler with partitions and fair-share accounting that runs outside Kubernetes; Kubernetes alone gives you the scheduler, Jobs, priorities and the newer pod group API that all of these build on or compete with.

Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1

Setup (one time)

1. Check the prerequisites for the minikube tutorial and install a container runtime, minikube and kubectl on your laptop.
Everything in this path runs on a local cluster, with no paid cloud and no GPU. CPU and memory requests stand in for GPU requests throughout, because the scheduler treats all of them as countable resources.
Explained in: Using Minikube to Create a Cluster | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: The minikube and kubectl version commands both print a version on your laptop.
2. Start a local minikube cluster and confirm its node is Ready.
A small cluster with a known, limited capacity makes contention easy to create. Contention is the whole subject of scheduling and capacity management.
Explained in: Using Minikube to Create a Cluster | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: Listing the nodes shows one node in the Ready state.
3. Read the cluster overview and note which components form the control plane and which run on the nodes.
The scheduler is a control plane component that only decides placement, while the kubelet on the node actually starts the pod. Knowing that split explains every later observation.
Explained in: Using Minikube to Create a Cluster | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: You can name the control plane component that assigns pods to nodes and the node component that runs them.

Steps

1. Run a single pod with a small CPU request and inspect its events to find the moment the scheduler assigned it to a node.
This is the smallest observable unit of scheduling: one pod, one decision. Every batch system in the lesson is built around this one-pod-at-a-time behaviour.
Explained in: Kubernetes Scheduler | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: Describing the pod shows a Scheduled event that names the node it was bound to.
2. Create a pod that requests more CPU than your node has and read the scheduling failure message.
The scheduler first filters out nodes that cannot fit the pod and then scores the rest. Seeing a filter failure teaches you how to read the reason a workload is stuck, which is the daily debugging task on a full cluster.
Explained in: Kubernetes Scheduler | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: The pod stays Pending and its events show a FailedScheduling message that states insufficient CPU.
3. Run a Job whose parallelism asks for more total CPU than the cluster has and watch which of its pods start.
This shows where kube-scheduler falls short for batch: it places pods one by one, so a job is partly started and holds capacity while the rest waits. There is no queue, no team budget and no all-or-nothing start.
Explained in: Job | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: Some pods of the Job are Running while others of the same Job are Pending, and the Job is not complete.
4. Create a low and a high PriorityClass, fill the cluster with a low priority Job, then submit a high priority Job and watch the preemption.
Priority and preemption are the only ordering tools the default scheduler has. They decide who wins but say nothing about quotas or fairness between teams.
Explained in: PriorityClass | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: Low priority pods are evicted and the high priority pods are Running in their place.
5. Create a Job with its suspend field set to true, confirm that no pods exist, then patch suspend to false to admit it.
This is the mechanism Kueue uses: it holds every job suspended in a queue and resumes it only when the team quota and fair sharing rules allow. Doing it by hand makes clear that Kueue is an admission layer in front of the scheduler, not a replacement for it.
Explained in: Job | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: The suspended Job has zero pods, and pods appear only after you set suspend to false.
6. Compare the status of the Jobs you ran so far, reading their conditions, start time and completion time to work out how long each one waited and ran.
Capacity management depends on measured wait time and run time per job. The Job status is the record that queueing systems and dashboards read these numbers from.
Explained in: Job | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: You have a small table of your Jobs with their start time, completion time and final condition.
7. Read how gang scheduling works in Kubernetes and note what the minimum count of a pod group means.
Distributed training needs all workers or none, which is the problem Volcano solves with its own pod group and minimum member setting. The native feature follows the same idea, so this reading gives you the vocabulary for both.
Explained in: Gang Scheduling | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: You can explain in one sentence why a gang of four workers must not start with only three placed.
8. Create a pod group with a gang scheduling policy whose minimum count exceeds what the cluster can fit, after enabling the feature gates the documentation names on a fresh minikube profile.
This reproduces the partly started Job from the earlier step and shows the fix: the group waits as a whole instead of holding capacity. The feature is still gated, so on real clusters teams usually get this behaviour from Volcano or Kueue.
Explained in: PodGroup | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: All pods of the group stay Pending together, and all start together once you lower the requests or the minimum count.
9. Run an Indexed Job with a fixed number of completions and limited parallelism, as the Kubernetes counterpart of a Slurm job array.
Slurm users think in batch submissions, arrays and partitions with a central queue. Mapping an array onto completions and parallelism shows what carries over and what does not, since Slurm also orders the queue and accounts for usage while a plain Job does neither.
Explained in: Job | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: The Job finishes with one succeeded pod per index and never runs more pods at once than the parallelism you set.
10. Create a CronJob that submits your batch Job on a schedule with a concurrency policy that forbids overlapping runs.
Recurring workloads such as nightly retraining are a steady source of load that capacity planning must account for. Forbidding overlap stops a slow run from piling up behind itself.
Explained in: CronJob | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: The CronJob creates a new Job at each scheduled time and skips a run while the previous one is still active.
11. Create a cluster with kubeadm on local virtual machines and add a worker node, then repeat the oversized Job test from a saved set of manifests. If your laptop cannot run two virtual machines, add a second node to minikube as the closest substitute.
Scheduling behaviour only becomes realistic with more than one node, and a scripted cluster build makes the experiments repeatable. This is the same tool many on-premises GPU clusters are bootstrapped with.
Explained in: Creating a cluster with kubeadm | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: Listing the nodes shows a control plane node and a worker node both Ready, and pods of your Job land on more than one node.
12. Create a ClusterRole that allows a team to create and read Jobs but not to create PriorityClasses, bind it to a test user, and check what that user may do.
On a shared cluster, anyone who can mint a high priority class can jump the queue and evict other teams. Restricting that right is the first governance control for a multi-tenant batch platform.
Explained in: ClusterRole | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: An access check for the test user answers yes for creating Jobs and no for creating PriorityClasses.
13. Configure a cluster add-on pod of your own as critical with a system priority class, fill the cluster with batch Jobs, and confirm the add-on is never evicted.
On a production cluster under constant batch pressure, the components that keep the cluster working must outrank every training job. This is the last safeguard before you hand the cluster to many teams.
Explained in: Guaranteed Scheduling For Critical Add-On Pods | Kubernetes (kubernetes.io), checked 2026-10-11
Done when: With the cluster full and batch pods Pending, the critical pod is still Running and shows the system priority class.

Back to Workload scheduling, capacity management and GPU cluster operations