Practice: Orchestrating ML pipelines with Airflow
Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.
Main tool: Airflow · Airflow is the tool this lesson is about, and its DAG, scheduler and executor model can be practised end to end on a laptop before the same DAG files move to a Kubernetes-backed deployment. · Alternatives: Dagster models pipelines as data assets with built-in lineage, Prefect runs plain Python flows with a lighter scheduler, Argo Workflows and Kubeflow Pipelines run every step as a container natively on Kubernetes, and managed Airflow services (Amazon MWAA, Google Cloud Composer, Astronomer) run the same DAGs without you operating the scheduler and metadata database.
Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1
Setup (one time)
1. Check the prerequisites: a supported Python version, a fresh virtual environment and Airflow installed with a pinned version and its matching constraints file.
A pinned install keeps the scheduler and provider versions reproducible, so the same environment can be rebuilt later in CI.
Explained in: Airflow 101: Building Your First Workflow — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: The airflow version command prints the pinned version from inside the virtual environment.
2. Create a dags folder in your Airflow home and confirm it is the folder Airflow loads DAG files from.
Every later step drops a Python file into this folder, so you need to know where the DAG processor looks and how it picks files up.
Explained in: Dags — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: The configured dags folder path is printed by the Airflow config command and matches the folder you created.
3. Start the local Airflow stack (API server, scheduler and DAG processor) and open the UI in a browser.
You need a running scheduler and UI to watch DAG runs, task states and logs for the rest of the path.
Explained in: Pythonic Dags with the TaskFlow API — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: The Airflow UI loads on localhost and you can log in and see the DAG list.
4. Install ruff in the same environment and enable the Airflow lint rules for your dags folder.
Linting catches deprecated imports and authoring mistakes before the scheduler ever parses a file, and it is the first check you will later run in CI.
Explained in: Best Practices — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: Running ruff against the dags folder finishes and reports either no findings or a list of Airflow rule findings.
Steps
1. Run the tutorial DAG: copy the example pipeline definition into your dags folder, check that it parses, then test one task and one full DAG run from the command line.
This is the shortest route to seeing a DAG file become tasks the scheduler knows about, without waiting for a schedule.
Explained in: Airflow 101: Building Your First Workflow — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: The tutorial DAG appears in the DAG list and the command-line test run prints the rendered task output with a success state.
2. Write a small ML pipeline with the TaskFlow API: tasks that extract a sample dataset, train a tiny scikit-learn model on CPU and evaluate it, passing only file paths and metrics between them.
TaskFlow is the idiomatic way to author Python tasks, and passing small references rather than data teaches the control-plane habit the lesson describes.
Explained in: Pythonic Dags with the TaskFlow API — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: The DAG shows three tasks in the graph view, a manual run succeeds, and the evaluation metric is visible as an XCom value.
3. Write a second version of the pipeline with classic operators and declare the dependencies explicitly, including a fan-out to two training tasks and a fan-in to one evaluation task.
Reading and wiring explicit upstream and downstream relationships is the core of the DAG model and what you will see in most existing codebases.
Explained in: Dags — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: The graph view shows the fan-out and fan-in exactly as you declared them and the tasks run in that order.
4. Configure a daily schedule with a start date a week in the past, trigger runs and log the data interval start and end inside a task.
A run covers a window of data, not the moment it starts, and every partitioned ML job depends on reading that window correctly.
Explained in: Dag Runs — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: The task log prints a data interval start and end one day apart, and the run's logical date matches the interval rather than the wall-clock time.
5. Enable catchup on the DAG, unpause it and watch the scheduler create runs for the missed intervals, then turn catchup off and compare.
Catchup decides whether a deploy silently launches a week of training jobs, so you should see both behaviours before relying on the default.
Explained in: Dag Runs — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: With catchup on, one DAG run per missed day appears in the grid view; with it off, only the latest interval runs.
6. Run a backfill for a three-day range from the command line, first as a dry run and then for real with a concurrency limit.
Backfill is how you recompute features or retrain for historical dates on purpose, and the dry run shows what will be created before anything executes.
Explained in: Backfill — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: The dry run lists the three dates it would run, and the real backfill produces three completed DAG runs visible in the UI.
7. Write the training task so it is idempotent: it writes its output to a path keyed by the data interval and overwrites rather than appends, then clear and re-run the same task twice.
Retries, backfills and manual clears all re-execute tasks, so a task that produces different results on a second run corrupts downstream models.
Explained in: Best Practices — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: After two re-runs of the same interval the output folder contains exactly one artifact for that date with identical content.
8. Configure retries, a retry delay and an execution timeout on a task, then make it fail on purpose on the first attempt and succeed on the second.
Transient failures in remote training or storage calls are normal, and you need to see how attempts, the up-for-retry state and timeouts behave.
Explained in: Tasks — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: The task instance shows two attempts in the UI, the first marked failed and up for retry and the second successful, each with its own log.
9. Configure a per-task executor configuration that requests a custom image and resource limits for the training task; this needs the Kubernetes executor, so use a local kind or minikube cluster in place of a cloud cluster.
Executor configuration is how one DAG gives a heavy task its own pod shape while light tasks stay cheap, which is the heart of choosing an executor for ML workloads.
Explained in: Tasks — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: Listing pods in the local cluster during the run shows a worker pod for the training task with the image and limits you set.
10. Run the training step in an isolated container with the Docker operator on your laptop, then switch the same step to the Kubernetes pod operator against the local cluster.
Launching the heavy work in its own image keeps conflicting ML dependencies out of the Airflow workers and makes Airflow the control plane, not the compute.
Explained in: Best Practices — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: The Airflow task log streams the container output and the task succeeds while no ML library is installed in the Airflow environment itself.
11. Write a DAG loader test and one unit test with pytest that import every DAG, assert there are no import errors and check the task count and dependencies of your pipeline, then run them alongside ruff as a single CI job.
A broken DAG file takes down scheduling for that pipeline, so parse and structure checks must run automatically on every change in a pinned environment.
Explained in: Best Practices — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: The pytest run passes locally, and deliberately breaking an import in a DAG file makes the loader test fail.
12. Create a staging environment that mirrors production: a separate Airflow deployment on a PostgreSQL metadata database with its own connections and variables, deploy the same DAG files to it and run the pipeline there end to end.
Staging is where you catch differences in connections, images and executor behaviour before a DAG reaches production, and it is the last gate after tests pass.
Explained in: Best Practices — Airflow 3.3.2 Documentation (airflow.apache.org), checked 2026-10-11
Done when: The pipeline completes in the staging deployment using staging connections, and the run history is stored in the PostgreSQL database rather than the local default.