Data pipeline orchestration with Airflow and Dagster

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: Airflow · also covers Dagster, Data pipelines · Step 25 of 34 in AI forward deployed engineer · version 1

Practice this

On this page

An orchestrator schedules pipeline steps, runs them in dependency order, retries failures and records what happened. Airflow models a pipeline as a DAG of tasks and is the tool most customers already run; Dagster models it as a graph of data assets and is common in newer stacks. Knowing both lets a forward deployed engineer extend whatever the customer has, without introducing another system.

Read more

How it is done in practice

A typical customer pipeline has an ingestion step landing raw files or tables, a dbt build triggered when ingestion succeeds, then Python steps for entity resolution, embedding or model scoring, and finally a publish step with quality checks. The team develops locally against a small sample, tests task logic as plain functions, and ships DAG or asset code through pull requests into the customer's deployment. Each step is idempotent per partition so on-call engineers can clear and rerun safely. Failures alert a shared channel with a runbook link, and the handover includes the schedule, owners, expected runtimes and the procedure for backfilling after a fix.

Technologies

Apache AirflowDagsterPrefectCeleryKubernetesdbtKafkaAmazon SQSGoogle Pub/SubParquet

Diagram

flowchart TB
  DF[DAG files in Python] --> DP[DAG processor]
  DP --> DB[(Metadata database)]
  SCH[Scheduler] --> DB
  SCH --> EX[Executor]
  EX --> W1[Worker]
  EX --> W2[Worker]
  W1 --> DB
  W2 --> DB
  UI[Web UI and API] --> DB
  W1 --> WH[Warehouse or lakehouse]
  W2 --> WH
flowchart TB
  DF[DAG files in Python] --> DP[DAG processor]
  DP --> DB[(Metadata database)]
  SCH[Scheduler] --> DB
  SCH --> EX[Executor]
  EX --> W1[Worker]
  EX --> W2[Worker]
  W1 --> DB
  W2 --> DB
  UI[Web UI and API] --> DB
  W1 --> WH[Warehouse or lakehouse]
  W2 --> WH

Code

python
from datetime import datetime, timedelta
from airflow.sdk import dag, task

@dag(schedule='@daily', start_date=datetime(2026, 1, 1), catchup=False,
     default_args={'retries': 2, 'retry_delay': timedelta(minutes=5)})
def customer_orders():
    @task
    def extract(data_interval_start=None) -> str:
        day = data_interval_start.strftime('%Y-%m-%d')
        return f'landing/orders/dt={day}/orders.parquet'

    @task
    def load(path: str) -> None:
        print(f'overwrite partition from {path}')

    load(extract())

customer_orders()
python
import dagster as dg
import pandas as pd

@dg.asset
def raw_orders() -> pd.DataFrame:
    return pd.read_parquet('landing/orders.parquet')

@dg.asset
def clean_orders(raw_orders: pd.DataFrame) -> pd.DataFrame:
    return raw_orders.dropna(subset=['order_id']).drop_duplicates('order_id')

defs = dg.Definitions(assets=[raw_orders, clean_orders])

Subtopics

Airflow architecture and core concepts

Components

An Airflow installation has a scheduler that triggers scheduled workflows and submits tasks to an executor, a DAG processor that parses DAG files, a web interface and API for inspecting and triggering runs, and a metadata database that stores the state of every DAG run and task instance. The executor decides where tasks run: locally in small setups, or on remote workers through Celery or Kubernetes in production. An optional triggerer handles deferred tasks that wait on external events without occupying a worker. DAG files are plain Python that all components must be able to read.

DAGs, tasks and operators

A DAG declares tasks and their dependencies plus a schedule. A task is an instance of an operator, a template for a unit of work such as running a bash command, a Python function or a SQL statement; sensors wait for a condition. Airflow orchestrates; heavy computation should run in the warehouse or a separate compute service, not in the worker process.

Authoring DAGs: idempotency, retries and backfills

Design each task to be rerun

Orchestrators retry and backfill, so a task must give the same result when run twice for the same interval. Parameterize work by the run's logical date and data interval, never by the wall clock. Write with overwrite-partition or merge semantics, not blind appends. Set retries and retry delay in default arguments, add timeouts, and keep tasks small enough that a retry is cheap. Pass only small values between tasks through XCom; put real data in storage and pass its location. Keep top-level DAG file code light, because the file is parsed repeatedly. Decide deliberately whether catchup should create runs for past intervals, and use backfill to reprocess history after a logic fix.

Dagster and the asset model

Declare what should exist

Dagster's central abstraction is the asset: an object in persistent storage, such as a table, file or trained model, defined in code by a function that produces it. Decorating a function as an asset declares the thing, and naming other assets as its inputs declares dependencies, so the lineage graph comes from the code itself and no separate wiring is needed. Dagster then materializes assets, records metadata about each materialization, and shows which assets are stale relative to their upstream. Partitions, schedules, sensors and asset checks attach to assets.

Consequences for development

Because assets are ordinary Python functions with resources injected, they can be unit tested and run locally with a development server before deployment. The asset graph also answers the questions customers ask most: where does this table come from, and when was it last refreshed. Integrations can load dbt models as assets, so SQL transformations and Python steps share one lineage view.

Choosing an orchestrator and operating it

Default to what the customer runs

The deciding factor on an engagement is rarely features. If the customer has Airflow with an on-call rotation, add a DAG there, even if you would pick differently on a blank slate. Task-centric Airflow suits heterogeneous jobs across many systems and has the largest ecosystem of provider packages and managed offerings. Asset-centric Dagster suits teams that think in tables and want lineage, local testing and data quality checks built in. Prefect is a third common option with a Python-first flow model.

Operating concerns

Whichever tool is used, plan for secrets held in connections or a secrets backend, never in code; alerting on failure and on missed service-level targets; concurrency limits so backfills do not overload the warehouse; and deployment of pipeline code through CI. Event-driven triggers from queues such as Kafka, SQS or Pub/Sub complement schedules when data arrives irregularly, and columnar files such as Parquet are the usual interchange format between steps.

Sources

Related lessons

Practice this
ELT transformations with dbtDeploying into customer clouds: AWS, Google Cloud and Azure