Practice: Distributed training across multiple GPU nodes
Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.
Main tool: Kubeflow · Kubeflow makes the distributed training job a declarative Kubernetes resource, so the platform work of this lesson (launching the worker group, wiring it together, recovering it) is something you can inspect and automate on a local cluster. · Alternatives: Ray Train with KubeRay scales a training loop from Python on a Ray cluster instead of declaring a Kubernetes resource, a raw JobSet or MPI launcher gives the worker group with no training runtime on top, and PyTorch Lightning or Hugging Face Transformers handle the in-process distribution strategy but leave launching and recovery to the platform.
Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1
Setup (one time)
1. Read the installation methods and choose a standalone install of the Trainer and Pipelines components on a local kind or minikube cluster, with no cloud account and no GPU.
Kubeflow is a set of subprojects, and a laptop cannot comfortably hold the full distribution. Picking only the pieces this lesson needs keeps the cluster small enough to run.
Explained in: Installing Kubeflow | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: You have written down which install method and which subprojects you will put on the local cluster.
2. Install the Kubeflow Trainer control plane on the local cluster by following the Trainer documentation entry page.
The Trainer controller is what turns a training job resource into a group of worker pods. Nothing in the later steps runs without it.
Explained in: Kubeflow Trainer | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The Trainer controller pod reports Running in the local cluster.
3. Install the Kubeflow Pipelines SDK in a fresh Python virtual environment and point it at the local cluster.
The path drives training jobs from pipeline steps, so the SDK and its connection to the cluster have to work first.
Explained in: Building Python function-based components | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: Importing the SDK in Python succeeds and it can reach the local Pipelines endpoint.
Steps
1. Run a first Python function-based component that only prints its inputs, inside a one-step pipeline on the local cluster.
This proves the whole chain from laptop to cluster pod before any training code is involved. Later failures are then about training, not plumbing.
Explained in: Building Python function-based components | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The run shows as succeeded and the pod log contains the printed inputs.
2. Read what Kubeflow Trainer is and note which parts belong to the training framework and which to the platform.
In data-parallel training the framework synchronises gradients while the platform launches the workers and connects them. Knowing the split tells you where to look when a job hangs.
Explained in: Overview | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: You can list, from memory, what the Trainer does for a job and what it leaves to PyTorch.
3. Write a small PyTorch DistributedDataParallel training function for two CPU workers, using the Trainer introduction to see how workers receive their rank and peers; CPU workers stand in for GPU nodes here.
Data-parallel training means every worker holds a model copy, trains on its own shard and averages gradients each step. Writing it for two CPU workers shows the mechanism without a GPU.
Explained in: Overview | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The function runs locally in two processes and both print the same loss after each step.
4. Create a training job as a Kubernetes resource from a pipeline step, using the resource operation to submit a two-worker job manifest for your function.
This is the core object model of Kubeflow Trainer: a distributed job is a declared resource the controller reconciles. Submitting it from a pipeline makes it one repeatable step.
Explained in: Manipulate Kubernetes Resources as Part of a Pipeline | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: Listing the training job resource shows it created, and two worker pods start together and finish.
5. Create a persistent volume claim from the pipeline and mount it into the workers as the checkpoint directory.
Worker pods are disposable, so a checkpoint written to pod storage dies with the pod. A shared volume is what makes recovery possible at all.
Explained in: Manipulate Kubernetes Resources as Part of a Pipeline | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: After a run, checkpoint files are visible on the volume from a separate pod.
6. Run the job again with the volume attached, delete one worker pod mid-training, and confirm the restarted job resumes from the last checkpoint.
Fault tolerance is only real once you have watched a failure and a resume. One lost worker stalls the whole group, so the restart behaviour matters more than in single-node training.
Explained in: Manipulate Kubernetes Resources as Part of a Pipeline | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The log of the restarted worker reports loading a checkpoint and continues from a later step than zero.
7. Create a volume snapshot of the checkpoint volume after two runs with different worker counts, and compare the saved metrics of both.
Snapshots keep each run's state so you can compare results instead of overwriting them. Comparing one worker against two shows what scaling out really bought you.
Explained in: Manipulate Kubernetes Resources as Part of a Pipeline | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: Two snapshots exist and you have a side-by-side note of final loss and wall time for both runs.
8. Package a Ray Train version of the same training loop as a lightweight component, declaring ray in its packages to install, and run it with two local CPU workers.
Ray Train scales the loop from inside Python rather than through a job resource. Running both styles side by side makes the trade-off concrete.
Explained in: Lightweight Python Components | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The component run succeeds and its log shows two Ray Train workers reporting metrics.
9. Configure a fixed base image for the training components instead of installing packages at start-up.
All workers must run identical framework versions or they fail to synchronise. A pinned image also removes install time from every worker start.
Explained in: Building Python function-based components | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The worker pods show your pinned image tag and start without a package install phase in the log.
10. Compare the two approaches against the reasons the Trainer overview gives for using it, and record which one you would give to which team.
A platform engineer has to recommend one path per use case. The overview gives the criteria the Trainer was designed around.
Explained in: Overview | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: You have a short written comparison covering launch model, failure recovery and who owns the job definition.
11. Configure the Kubernetes platform-specific settings of the pipeline, such as resource requests, volumes and node selection, in the pipeline definition itself; where a GPU request is expected, use a CPU request locally.
Settings applied by hand on the cluster are lost on the next run. Keeping them in the pipeline definition makes the job reproducible by anyone and by CI.
Explained in: Kubernetes Platform-specific Features | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: A fresh run from the definition alone produces worker pods with the requests and mounts you declared.
12. Compile the pipeline to a pinned definition file, commit it, and reproduce the full run from that file on a clean local cluster.
The compiled file is the artifact an automated job can submit without your laptop environment. Reproducing it on a clean cluster is the closest laptop test of how it will behave elsewhere.
Explained in: Compile a Pipeline | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The run launched from the committed file on a fresh cluster finishes with the same worker count and a checkpoint on the volume.