ML-native orchestrators: Kubeflow Pipelines, Flyte, Prefect and Metaflow

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: Kubeflow · also covers Flyte, Prefect, Metaflow · Step 9 of 35 in MLOps / ML platform engineer · version 1

Practice this

On this page

ML-native orchestrators treat a pipeline as a graph of containerized, typed steps that produce versioned artifacts, rather than as scheduled tasks that happen to run ML code. Kubeflow Pipelines and Flyte are Kubernetes-native platforms; Prefect and Metaflow are Python-first frameworks that start on a laptop and scale out to remote compute. Choosing between them is mostly a question of who authors pipelines, where compute lives and how much platform the team wants to operate.

Read more

How it is done in practice

A platform team usually picks one ML-native orchestrator and wraps it in a golden path: a project template, base images with pinned ML libraries, a CI job that builds step images and registers or deploys the pipeline, and defaults for resources, retries and artifact locations. Data scientists develop a pipeline locally on a sample, then run it remotely with full data and GPUs by changing configuration instead of code. Runs are parameterized and triggered by a schedule, an upstream data pipeline or a merge. The final steps evaluate the model against a gate and register it. The team watches cache hit rates, queue wait times for GPU steps and failed-run causes, and keeps tenants isolated with namespaces, quotas and per-team storage prefixes.

Technologies

Kubeflow PipelinesKubeflowFlytePrefectMetaflowArgo WorkflowsKubernetesRayMLflow

Diagram

flowchart TB
  Dev[Pipeline code in Python] --> CI[CI: build images, compile or register]
  CI --> CP[Orchestrator control plane]
  CP --> K8s[Kubernetes or batch compute]
  K8s --> A[Step: prepare data]
  K8s --> B[Step: train on GPU]
  K8s --> C[Step: evaluate]
  A --> Store[(Artifact store)]
  B --> Store
  C --> Store
  CP --> Meta[(Run metadata and lineage)]
  C --> Reg[Model registry]
flowchart TB
  Dev[Pipeline code in Python] --> CI[CI: build images, compile or register]
  CI --> CP[Orchestrator control plane]
  CP --> K8s[Kubernetes or batch compute]
  K8s --> A[Step: prepare data]
  K8s --> B[Step: train on GPU]
  K8s --> C[Step: evaluate]
  A --> Store[(Artifact store)]
  B --> Store
  C --> Store
  CP --> Meta[(Run metadata and lineage)]
  C --> Reg[Model registry]

Code

python
from metaflow import FlowSpec, step
class TrainFlow(FlowSpec):
    @step
    def start(self):
        self.alphas = [0.1, 1.0, 10.0]
        self.next(self.train, foreach="alphas")
    @step
    def train(self):
        self.alpha = self.input
        self.score = 1.0 / (1.0 + self.alpha)  # stand-in for a real fit
        self.next(self.join)
    @step
    def join(self, inputs):
        self.best = max(inputs, key=lambda i: i.score).alpha
        self.next(self.end)
    @step
    def end(self):
        print("best alpha", self.best)
if __name__ == "__main__":
    TrainFlow()

Subtopics

What ML-native orchestration adds

Artifacts, not just task state

A general scheduler records whether a task succeeded. An ML-native orchestrator also records what each step consumed and produced: datasets, models, metrics, with types and storage locations. That enables three things. Lineage: for any model, find the exact data, code version and parameters that produced it. Caching: if a step's inputs and code are unchanged, reuse the previous output instead of recomputing, which matters when a step is hours of GPU time. Reproducibility: each step runs in a pinned container image with declared resources, so a run can be repeated later or on another cluster.

Workload shape

ML pipelines are often run on demand with parameters (an experiment, a retrain for one segment) rather than only on a clock. They fan out widely for hyperparameter sweeps or per-partition training, need per-step resources such as GPUs and large memory, and are authored by data scientists who want plain Python and a fast local loop. These orchestrators optimize for those needs.

Kubeflow Pipelines

Model

Kubeflow Pipelines (KFP) is a platform for building and deploying portable, scalable ML workflows on Kubernetes, and one component of the wider Kubeflow project, which also covers notebooks, training operators, tuning and model serving. With the KFP Python SDK, a component is a function or container with typed inputs and outputs, and a pipeline is a function that wires components together. The SDK compiles the pipeline to an intermediate representation in YAML, which is submitted to a KFP backend. Each component executes as its own container, and the backend stores runs, experiments, parameters and artifacts so runs can be compared in the UI.

Operating it

The backend runs in the cluster with an API server, a workflow engine (Argo Workflows in the open source distribution), a metadata store and an object store for artifacts. Step caching, recurring runs, and per-step resource and node placement settings are built in. The cost is operational: the platform team owns upgrades, multi-user isolation through namespaces and profiles, and the image build path for components.

Flyte

Model

Flyte is an open source, Kubernetes-native workflow orchestrator for data and ML. Tasks are strongly typed Python functions, each run in a container, and workflows compose tasks into a graph that is checked at registration time, so many wiring errors surface before anything runs. Registered tasks and workflows are immutable and versioned, and executions are grouped by project and domain (for example development, staging, production), giving a multi-tenant structure out of the box. Launch plans bind a workflow to default inputs, schedules and notifications.

Why teams pick it

Flyte passes typed data between tasks through object storage and tracks it, supports caching keyed on inputs and a cache version, dynamic workflows and map tasks for large fan-out, per-task resource requests including GPUs, and retries that can distinguish system failures such as spot preemption from user errors. Plugins hand a task to Spark, Ray or distributed training operators. It suits organizations that want strong contracts and a central platform serving many teams.

Prefect and Metaflow

Prefect

Prefect turns ordinary Python into workflows with a flow decorator on the entry function and a task decorator on units of work. Control flow is native Python, so loops and conditionals work as written and the graph is discovered at runtime rather than declared up front. Tasks get retries, caching and concurrency controls, and state is reported to a Prefect server or the hosted cloud service. To run on a schedule or in response to events, a flow is packaged as a deployment and executed by workers pulling from a work pool backed by infrastructure such as Kubernetes or containers.

Metaflow

A Metaflow flow is a class deriving from FlowSpec; methods marked as steps are nodes, and each step calls self.next to name its successor. Any value assigned to self becomes a persisted, versioned artifact available in later steps and after the run through the client API. Branches and foreach fan out, and a join step merges results. The same code runs locally or with steps dispatched to cloud batch compute or Kubernetes through decorators that request resources, and flows can be deployed to a production scheduler.

Choosing and combining orchestrators

Decision factors

Start with who writes pipelines. Data scientists iterating on models favor Metaflow or Prefect for the local loop and low ceremony; a central platform with many tenants favors Flyte or KFP for typed interfaces, isolation and governance. Next, where compute lives: KFP and Flyte assume Kubernetes, Metaflow fits cloud batch services as well, and Prefect is infrastructure agnostic. Then consider operating cost, since KFP and Flyte are multi-service control planes that need an owner. Finally, check integration with the model registry, experiment tracker and feature store already in place.

Coexistence

Many organizations run two layers: a general scheduler such as Airflow for data pipelines and cross-system coordination, and an ML-native tool for training and evaluation graphs, with the scheduler triggering ML pipeline runs through an API. Lower-level engines such as Argo Workflows sit underneath several of these tools and are sometimes used directly by platform teams for container-native DAGs.

Sources

Related lessons

Practice this
Orchestrating ML pipelines with AirflowDurable workflows and event-driven services with Temporal