Workflow orchestration and durable execution

Turning steps into repeatable, retryable pipelines: Airflow for ML and for data (with Dagster), the ML-native orchestrators, and Temporal.

Workflow orchestration is the layer that turns individual steps such as ingestion, feature computation, training, evaluation and deployment into repeatable, observable, retryable pipelines. This module covers the general-purpose scheduler most teams start with (Airflow), used for ML pipelines and for data pipelines alongside Dagster, the ML-native orchestrators built around containers and artifacts, and durable execution with Temporal for long-running, event-driven processes.

Lessons (4)

• Orchestrating ML pipelines with Airflow · Practice

  Airflow models a pipeline as a DAG of tasks defined in Python, and a scheduler decides when each task runs based on a schedule, dependencies and state.

• ML-native orchestrators: Kubeflow Pipelines, Flyte, Prefect and Metaflow · Practice

  ML-native orchestrators treat a pipeline as a graph of containerized, typed steps that produce versioned artifacts, rather than as scheduled tasks that happen to run ML code.

• Durable workflows and event-driven services with Temporal · Practice

  Temporal provides durable execution: workflow code runs as if failures do not exist, because every step is recorded in an event history and the workflow can be rebuilt by replay after a crash.

• Data pipeline orchestration with Airflow and Dagster · Practice

  An orchestrator schedules pipeline steps, runs them in dependency order, retries failures and records what happened.

Changes

• Version 1, 2026-10-14: re-cut from the role pathway groups into a topic module (site 2026.10.2); no lesson changed