Practice: ML-native orchestrators: Kubeflow Pipelines, Flyte, Prefect and Metaflow
Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.
Main tool: Kubeflow · Kubeflow Pipelines shows every idea of the lesson in one place on a local cluster: containerized steps, typed inputs and outputs, tracked artifacts and a graph you can inspect. · Alternatives: Flyte is also Kubernetes-native but enforces strongly typed tasks and versioned workflows with its own control plane; Prefect is a Python-first framework that runs flows on a laptop and scales out through work pools; Metaflow is a Python-first framework that moves local steps to remote compute with decorators; Argo Workflows is the general-purpose Kubernetes workflow engine with no ML artifact or lineage layer of its own.
Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1
Setup (one time)
1. Read how Kubeflow is split into subprojects and confirm that Kubeflow Pipelines is the only one this path needs.
Kubeflow is a family of projects, not one binary. Knowing which piece does orchestration stops you from installing the whole platform on a laptop.
Explained in: Introduction | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: You can name the Kubeflow subproject that runs pipelines and two subprojects you will not install.
2. Choose the standalone installation method for Kubeflow Pipelines and deploy it on a local Kubernetes cluster such as kind or minikube, with no cloud account and no GPU.
The installation methods differ a lot in footprint. The standalone subproject install is the one that fits in laptop memory.
Explained in: Installing Kubeflow | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: All Kubeflow Pipelines pods in the local cluster report a running state.
3. Install the Kubeflow Pipelines Python SDK into a fresh virtual environment.
The SDK is the authoring interface: you write components and pipelines in Python and it compiles and submits them.
Explained in: Interfaces | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: Importing the SDK in a Python session works and prints a version number.
4. Start a port forward to the Kubeflow Pipelines user interface and open it in a browser.
The user interface is where you read graphs, logs and artifacts for every run in the rest of this path.
Explained in: Interfaces | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The pipelines dashboard loads locally and shows an empty or sample list of runs.
Steps
1. Run the getting started hello world pipeline against your local cluster from the SDK.
One end to end run proves that the SDK, the API server and the cluster are talking to each other before you write anything of your own.
Explained in: Getting started | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The run appears in the user interface with a succeeded status and the greeting string as its output.
2. Read why Kubeflow Pipelines exists and list what it adds on top of a plain task scheduler.
This is the lesson's first subtopic: ML-native orchestration means containerized steps, typed data passing and tracked artifacts, not just a schedule that happens to run ML code.
Explained in: Overview | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: You have a short written list of at least three things the pipeline platform gives you that a cron style scheduler does not.
3. Write two Python-based components, one that prepares a small dataset and one that trains a tiny CPU model on it.
The component is the unit of the object model. Each one becomes its own container at run time, so writing two makes the isolation between steps visible.
Explained in: Component | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: Both components compile without errors and each shows as a separate task when used in a pipeline.
4. Write a pipeline that takes input parameters and wires the output of the first component into the second, then run it.
A pipeline declaration is what turns separate components into a dependency graph. Data passing between tasks is what defines the execution order.
Explained in: Pipeline | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The run graph in the user interface shows two connected nodes and the parameter values you supplied.
5. Create a second version of the pipeline that adds control logic: a condition on a metric and a loop over two parameter values.
Real ML pipelines branch and fan out. Seeing how control logic changes the graph prepares you to read production pipelines.
Explained in: Pipeline | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The run graph shows the loop iterations as separate tasks and the conditional branch as taken or skipped.
6. Configure a pipeline root for your pipeline so its artifacts land under a location you chose in the local object store.
The pipeline root is where versioned artifacts persist. Controlling it is how a platform team separates projects and environments.
Explained in: Pipeline Root | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: After a new run, the output artifacts are listed under the root path you configured rather than the default one.
7. Write an explicit input and output specification with a dataset artifact and a model artifact for your training component, then map each type to how Flyte would declare the same typed task.
Typed interfaces are the main idea Kubeflow Pipelines and Flyte share. Doing the mapping by hand shows where Flyte is stricter and where the two are equivalent.
Explained in: Component | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The run shows typed dataset and model artifacts on the task, and you have a short table mapping each one to its Flyte counterpart.
8. Run the Python function of one component directly on your laptop without the cluster, the way a Prefect flow or a Metaflow step starts locally.
Prefect and Metaflow are Python-first and local by default. Trying the same loop in Kubeflow Pipelines shows how much, or how little, local iteration the Kubernetes-native tool gives you.
Explained in: Component | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The function returns its result in a local Python session with no pod created, and you have noted what the local call skips compared with a cluster run.
9. Package one component as a containerized component with a pinned base image and pinned dependencies, built and loaded into the local cluster.
A pinned image is what makes a step reproducible across laptops, CI and clusters. It is also the artifact a team actually promotes.
Explained in: Component | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The task pod for that component runs your own image tag and the run succeeds twice with identical outputs.
10. Compare your standalone Kubeflow Pipelines install with the Kubeflow community distribution and decide which one a team in your situation should operate.
Choosing an orchestrator is mostly about how much platform the team wants to run. The step from one subproject to the full distribution is that decision made concrete.
Explained in: Installing Kubeflow | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: You have a one paragraph decision naming who authors pipelines, where compute lives and which install you would promote to a shared cluster.
11. Enable the MLflow integration against a local MLflow tracking server and rerun your pipeline so its parameters and metrics are recorded there.
Teams usually combine orchestrators and trackers rather than pick one. This integration shows Kubeflow Pipelines running the graph while MLflow holds the experiment record.
Explained in: Overview | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The local MLflow console shows a run for your pipeline with the same parameters and metrics as the Kubeflow Pipelines user interface.
12. Create a run of your compiled pipeline through the REST API from a small script, the way a CI job would trigger it.
Automation does not go through the user interface. Triggering runs over the API is how pipelines are wired into CI and into other orchestrators.
Explained in: Pipelines API Reference (v2beta1) | Kubeflow (www.kubeflow.org), checked 2026-10-11
Done when: The script returns a run identifier and that run appears in the user interface without you clicking anything.
Back to ML-native orchestrators: Kubeflow Pipelines, Flyte, Prefect and Metaflow