Practice: Large-scale data pipelines with Spark and PySpark
Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.
Main tool: Spark / PySpark · Spark with PySpark lets you practise the whole lesson on a laptop with one engine: lazy DataFrame plans, shuffles and skew, window and Arrow-based feature work, Structured Streaming, and submission to a real cluster manager. · Alternatives: Apache Flink is streaming-first with lower latency but a weaker batch and Python story; Dask and Ray Data are Python-native and lighter to operate but have a less mature SQL optimizer and shuffle; DuckDB and Polars are faster on a single machine but do not scale out; managed Spark services run the same engine and trade cluster operations for cost and lock-in.
Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1
Setup (one time)
1. Install a supported Java runtime and Python, then install PySpark with pip into a fresh virtual environment.
Every later step runs against this local install, and a clean environment keeps the Spark version explicit and repeatable.
Explained in: Getting Started — PySpark 4.2.0 documentation (spark.apache.org), checked 2026-10-11
Done when: Importing pyspark in the virtual environment's Python and printing its version shows the release you installed.
2. Download the matching prebuilt Spark distribution and unpack it so the standalone launch scripts and spark-submit are available.
The pip package is enough for local mode, but the cluster steps need the scripts and image tooling shipped in the full distribution.
Explained in: Spark Standalone Mode - Spark 4.2.0 Documentation (spark.apache.org), checked 2026-10-11
Done when: Running spark-submit with the version flag from the unpacked folder prints the same version as the pip install.
3. Install a container runtime and a local Kubernetes cluster such as minikube or kind, sized to the stated minimum CPU and memory, with kubectl pointed at it.
The final steps submit a job to Kubernetes, and the prerequisites section lists what the cluster and your account must allow.
Explained in: Running Spark on Kubernetes - Spark 4.2.0 Documentation (spark.apache.org), checked 2026-10-11
Done when: kubectl lists one Ready node and reports that you are allowed to create pods.
Steps
1. Run a PySpark script that creates a SparkSession in local mode and shows a small DataFrame.
The SparkSession is the entry point to everything else, and a first run proves that the Java and Python sides can talk to each other.
Explained in: Getting Started - Spark 4.2.0 Documentation (spark.apache.org), checked 2026-10-11
Done when: The script prints a table of rows and exits without errors.
2. Write a chain of select, filter and groupBy transformations on a larger generated DataFrame, call explain on it, and only then trigger an action such as count.
Transformations only build a plan and nothing runs until an action, which is the lazy model behind every cost and correctness question you will debug later.
Explained in: Getting Started - Spark 4.2.0 Documentation (spark.apache.org), checked 2026-10-11
Done when: The plan prints immediately, and a job appears in the Spark UI only after the action runs.
3. Start a standalone master and one worker on your laptop, then run the same script with its master URL pointing at that cluster.
With a real cluster manager the driver and the executors are separate processes, so you can see which one plans and which one runs tasks.
Explained in: Spark Standalone Mode - Spark 4.2.0 Documentation (spark.apache.org), checked 2026-10-11
Done when: The master web UI shows one live worker and your application with an executor assigned to it.
4. Create a per-user event table and compute rolling and ranked features over it with window functions, in both the DataFrame API and Spark SQL.
Window functions are the workhorse of feature tables, and each partition-by clause adds a shuffle you should learn to spot in the plan.
Explained in: Spark SQL, Built-in Functions (spark.apache.org), checked 2026-10-11
Done when: Each row carries a rank and a rolling aggregate for its user, and both versions return identical results.
5. Enable Arrow and write a pandas UDF that derives features from a free-text column, then time it against a plain row-at-a-time Python UDF.
Unstructured data work in PySpark is dominated by the cost of moving rows between the JVM and Python, and Arrow batches are how you cut it.
Explained in: PySpark Usage Guide for Pandas with Apache Arrow - Spark 4.2.0 Documentation (spark.apache.org), checked 2026-10-11
Done when: Both UDFs give the same output and the Arrow-based one is measurably faster on the same data.
6. Compare the physical plan and runtime of a join between a large and a small table with and without a broadcast hint.
Choosing between a broadcast join and a sort-merge join decides whether the large side is shuffled at all, which is the biggest single lever on job cost.
Explained in: Performance Tuning - Spark 4.2.0 Documentation (spark.apache.org), checked 2026-10-11
Done when: The hinted plan shows a broadcast hash join with no shuffle of the large table, and the unhinted run with auto broadcast disabled shows a sort-merge join.
7. Create a join where one key holds most of the rows, run it with adaptive query execution off, then enable the skew join optimization and run it again.
Skewed keys produce one very long task that stalls the whole stage, and you need to recognise that pattern and know the built-in remedy.
Explained in: Performance Tuning - Spark 4.2.0 Documentation (spark.apache.org), checked 2026-10-11
Done when: In the Spark UI the first run shows one task far longer than the rest, and the second run shows the skewed partition split into several tasks.
8. Run a Structured Streaming query that aggregates events over event-time windows with a watermark and a checkpoint directory, using the built-in rate or file source as the local substitute for a Kafka topic.
Near-real-time features reuse the same DataFrame code, but add state, late data and restart recovery that batch jobs never face.
Explained in: Structured Streaming Programming Guide - Spark 4.2.0 Documentation (spark.apache.org), checked 2026-10-11
Done when: Windowed counts update on every trigger, and after you stop and restart the query it resumes from the checkpoint without recounting old data.
9. Read the notes for upgrading to your pinned Spark version and check each behaviour change against the features your jobs use before promoting the version.
Promoting a Spark version across environments is a platform decision, and silent changes in defaults are what break a rerun months later.
Explained in: Migration Guide: SQL, Datasets and DataFrame - Spark 4.2.0 Documentation (spark.apache.org), checked 2026-10-11
Done when: You have a short list of the changes that affect your job, each with the config or code adjustment it needs.
10. Package your job by building a PySpark container image with the image tool shipped in the Spark distribution and loading it into your local cluster.
On Kubernetes the image is the deployable unit, so the Spark version, Python dependencies and job code are frozen together in it.
Explained in: Running Spark on Kubernetes - Spark 4.2.0 Documentation (spark.apache.org), checked 2026-10-11
Done when: The image is listed in the local cluster's image store under the tag you chose.
11. Write pytest tests with a shared SparkSession fixture that assert DataFrame equality for your feature transformations, and run them from a clean checkout of the pinned environment.
Tests that run on a local session are what let CI catch a broken transformation before it reaches a shared cluster.
Explained in: Testing PySpark — PySpark 4.2.0 documentation (spark.apache.org), checked 2026-10-11
Done when: The suite passes from a fresh environment, and fails with a readable row diff when you deliberately break a transformation.
12. Deploy the job to your local Kubernetes cluster in cluster mode, with a dedicated namespace and service account and explicit executor counts and resource requests, then read the driver logs.
This is the shape of a production submission on a shared cluster: the driver runs as a pod, asks for executor pods, and is bound by RBAC and quotas.
Explained in: Running Spark on Kubernetes - Spark 4.2.0 Documentation (spark.apache.org), checked 2026-10-11
Done when: A driver pod and the requested executor pods appear in the namespace, the driver pod ends in Completed status, and its logs show the job result.