Practice: Python for platform engineering: packaging, environments and testing
Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.
Main tool: Python · The interpreter with its built-in venv module and bundled pip is the lowest common denominator on every laptop, CI runner and cluster image, so hygiene learned here carries over to every higher-level tool. · Alternatives: uv (one fast binary for environments, locking and builds, with a real cross-platform lock file), Poetry (its own resolver, lock file and build backend driven from pyproject.toml), conda (also manages non-Python binaries and the interpreter itself), pip-tools (compiles a fully pinned requirements file from loose inputs), versus plain Python with venv and pip, which ships with the interpreter and is the baseline all the others build on.
Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1
Setup (one time)
1. Install a current Python 3 release and confirm which interpreter version your shell launches, keeping any older versions side by side.
Platform work often targets several Python versions at once, so you need to know exactly which interpreter each command uses.
Explained in: Installing Python modules — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: Asking the interpreter for its version prints the release you intend to use, and you can launch a second version by its explicit name if one is installed.
2. Check that pip is available for that interpreter by running it as a module, and bootstrap it if it is missing.
Some system installs ship Python without pip, and every later step depends on it.
Explained in: Installing Python modules — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: Running pip as a module of your chosen interpreter prints a pip version and the path of the Python it belongs to.
3. Create an empty project directory for a small pipeline library and decide never to install into the system Python from here on.
Installing into the system interpreter can break operating system tools and makes results impossible to reproduce elsewhere.
Explained in: Installing Python modules — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: You have an empty project folder and can explain why the system site-packages is off limits for this exercise.
Steps
1. Create a virtual environment inside the project directory with the venv module and activate it.
This is the first run of the isolation mechanism that every training job, CI job and container image relies on.
Explained in: 12. Virtual Environments and Packages — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: Your shell prompt shows the environment name and the interpreter path now points inside the project directory.
2. Inspect the environment you created: open its configuration file, list its directories, and compare the prefix and base prefix values reported by the interpreter.
Knowing that a venv is a small directory pointing at a base interpreter explains why environments are cheap to rebuild and should not be copied or moved.
Explained in: venv — Creation of virtual environments — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: You can point at the config file that names the base interpreter and show that the prefix and base prefix differ only while the environment is in use.
3. Install two or three libraries with pip, one of them at an exact older version, then upgrade it and list and show what is installed.
Installing, pinning a version, upgrading and inspecting are the core operations behind every dependency decision a platform team makes.
Explained in: 12. Virtual Environments and Packages — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: The pip list output shows the exact version you asked for first and the newer version after the upgrade.
4. Run pip freeze to write every installed version into a requirements file, then create a second environment, install from that file and compare the two freeze outputs.
A fully pinned file is the simplest lock, and comparing two environments built from it is how you prove a dependency set is reproducible. Dedicated lock tools add hashes and cross-platform resolution on top of this idea.
Explained in: 12. Virtual Environments and Packages — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: The freeze output of the second environment is identical line for line to the requirements file from the first.
5. Write a small importable library with a pyproject.toml that declares its name, version, dependencies and build backend, then install the project into the environment with pip.
Shared pipeline code should be an installable distribution with declared dependencies, not a folder added to the import path by hand.
Explained in: Installing Python modules — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: The pip list output shows your library by name and version, and importing it works from a directory outside the project.
6. Write unit tests for the library with the unittest module, add one doctest to a function docstring, and run the whole suite from the command line.
Platform and pipeline code is run unattended by other teams, so its behavior has to be checked by tests that anyone can run with one command.
Explained in: 10. Brief tour of the standard library — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: The test run reports all tests passing, and breaking one function on purpose makes the run fail with a clear message.
7. Write a command-line entry point for the library whose options are declared with types, allowed choices and required flags, and feed it a bad value.
Validating configuration at the boundary turns a wrong setting into an immediate, readable error instead of a failure hours into a job. Libraries such as Pydantic extend the same idea to nested config files.
Explained in: 10. Brief tour of the standard library — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: A valid invocation runs, and an invalid value or a missing required option exits with a usage error before any work starts.
8. Read the glossary entries for annotation, type hint and variable annotation, then add type hints to the public functions of your library.
Type hints are the contract that static checkers and linters such as mypy and ruff enforce, and the glossary defines exactly what they are and that the interpreter does not enforce them at run time.
Explained in: Glossary — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: Every public function in your library has annotated parameters and return values, and you can state what does and does not check them.
9. Run the test suite again with warnings turned into errors and with development mode enabled.
Strict interpreter flags act as a built-in lint pass, surfacing deprecations and unclosed resources before a dependency upgrade turns them into outages.
Explained in: 1. Command line and environment — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: The suite passes under both flags, or it fails on a specific warning that you then fix and rerun to green.
10. Configure logging in the library so that pipeline steps emit leveled messages instead of print calls, and run the entry point at two different levels.
Consistent logging is what lets the same code be debugged on a laptop and observed quietly on a cluster.
Explained in: 11. Brief tour of the standard library — part II — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: Running at the default level shows only warnings and errors, and running at a verbose level also shows the informational and debug lines.
11. Reproduce the project from scratch: delete the environment, create a new one with the options that clear the target and upgrade the core installer, install from the pinned file, install the library and run the tests.
A clean rebuild from the files in version control is exactly what a CI job does, and it is the only real proof that nothing depends on leftover laptop state.
Explained in: venv — Creation of virtual environments — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: Starting from a directory with no environment, a short fixed list of commands ends with the full test suite passing.
12. Read how Python derives the locale encoding, then check which encoding your laptop reports and which one a minimal container image would report.
A different default text encoding is a classic reason for a job that reads files correctly on a laptop to fail in CI or on a cluster.
Explained in: Unicode Objects and Codecs — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: You can name the encoding your interpreter uses by default and explain why a bare container may choose a different one.
13. Run the rebuild and the test suite the way a production job would: inside a local container built from a pinned Python image, with environment variables set to force UTF-8 mode, fix the hash seed and skip writing bytecode, and with the interpreter ignoring user site packages.
Pinning the interpreter, the dependencies and the interpreter settings together is what makes a training job behave the same on a laptop, in CI and on a cluster. No paid cloud is needed, since a local container stands in for the CI runner.
Explained in: 1. Command line and environment — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: Two consecutive container runs from a clean checkout produce the same freeze output and the same passing test report.
Back to Python for platform engineering: packaging, environments and testing