Data platforms and pipelines

The data layer under ML and AI work: Spark, lakehouse and stores, feature stores, cleaning and linking messy records, Snowflake and Databricks, dbt.

This module covers the data layer under ML and AI work. It starts with large-scale compute with Spark, the lakehouse, warehouse and operational stores that hold the data, and the feature store that serves the same values to training and inference. It then turns to the state of a customer's data: cleaning and linking messy records, working inside Snowflake and Databricks accounts you do not own, and building tested transformations with dbt.

Lessons (6)

• Large-scale data pipelines with Spark and PySpark · Practice

  Apache Spark is the default engine for building training sets and feature tables that do not fit on one machine.

• Storage for ML: lakehouse, warehouse and operational data stores · Practice

  ML systems use several stores, each chosen for one access pattern: object storage and lakehouse tables for bulk training data, a warehouse for analytical SQL, a relational database for metadata and transactional state, and a key-value store for low-latency lookups at inference.

• Feature stores and feature pipelines: offline/online consistency · Practice

  A feature store is the system that defines features once and serves the same values to training and to online inference.

• Cleaning messy customer data: data mapping and entity resolution · Practice

  Customer data arrives as overlapping exports from CRMs, ERPs and spreadsheets, with inconsistent formats, missing values and the same real-world entity recorded many times.

• Working in customer data platforms: Snowflake and Databricks · Practice

  Most enterprise customers already keep their analytical data in Snowflake or Databricks, so AI work starts inside an account someone else owns and governs.

• ELT transformations with dbt · Practice

  In ELT, raw data is loaded into the warehouse first and transformed there with SQL.

Changes

• Version 1, 2026-10-14: re-cut from the role pathway groups into a topic module (site 2026.10.2); no lesson changed