Python for platform engineering: packaging, environments and testing

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: Python · also covers Software engineering · Step 2 of 35 in MLOps / ML platform engineer · version 1

Practice this

On this page

Python is the common language of ML teams, so platform tooling, pipeline components and SDKs are mostly written in it. The platform concern is not syntax but engineering hygiene: isolated environments, reproducible dependencies, properly packaged libraries and tested code. Getting these right is what makes a training job behave the same on a laptop, in CI and on a cluster.

Read more

How it is done in practice

Teams standardise on a project template: a src layout, a pyproject.toml with pinned tool settings, a lock file, a test folder and a container build that installs from the lock. Shared platform code lives in a small number of internal packages published to a private index by CI on tagged commits. Data scientists create environments with one command, and CI rebuilds the environment from scratch on every change so that drift between laptops and the cluster shows up as a failed build instead of a failed training run.

Technologies

venvpipuvPoetrysetuptoolshatchlingbuildtwinepytestmypyruffPydantic

Code

toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "platform-sdk"
version = "0.3.0"
requires-python = ">=3.10"
dependencies = ["pydantic>=2", "requests>=2.31"]

[project.scripts]
platform = "platform_sdk.cli:main"
bash
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python -m pip install -e .
python -m pytest -q
python -m build   # writes sdist and wheel to dist/

Subtopics

Virtual environments and isolation

How a virtual environment works

A virtual environment is a self-contained directory tree holding a particular Python version plus its own set of installed packages. It is created with the standard venv module, for example by running python -m venv .venv, and activated by a script that puts its bin directory first on the shell path. Packages installed with pip then go into that environment only. This solves the conflict where one application needs version 1 of a library and another needs version 2. The environment directory is disposable and should be excluded from version control; the list of requirements is what gets committed.

Why it matters more in ML

ML stacks have heavy native dependencies: numerical libraries, GPU runtimes and compiled extensions that are tied to specific driver and Python versions. Two projects on the same machine frequently cannot share one interpreter. In production the container image is the real isolation boundary, yet a virtual environment inside the image is still common because it keeps application packages apart from system Python. The platform rule is one environment per project, created from a declared file, never built up by hand.

Dependency pinning and lock files

Declared versus resolved dependencies

There are two lists to keep. The declared list states what the project needs directly, with version ranges. The resolved list, or lock file, records the exact version of every direct and transitive package that was actually installed. The simplest form is pip freeze writing a requirements.txt that pip install -r can replay. Dedicated tools produce lock files with hashes so that an install fails if a package was tampered with or changed. Training and serving should install from the same lock, since a different numerical library version between them is a real cause of training-serving skew.

Packaging libraries with pyproject.toml

Project layout and build

A distributable package has a pyproject.toml that names a build backend in its build-system table and holds metadata in the project table: name, version, required Python version, dependencies and entry points. A common layout puts importable code under a src directory, with tests beside it. Running the build frontend produces two artifacts: a source distribution and a wheel, which is a prebuilt archive pip can install without running build steps. Artifacts are uploaded to an index with an upload tool, and a test index exists for rehearsing a release before publishing for real.

Internal platform SDKs

Platform teams ship shared code as versioned packages instead of copying modules between repositories: a client for the feature store, pipeline step helpers, logging and tracing setup. These go to a private index, and consumers pin a version. Semantic versioning signals breaking changes, and console entry points in the project metadata turn the package into a command line tool that data scientists can run. Keeping the SDK thin, with few dependencies, avoids forcing version conflicts onto every training environment that installs it.

Testing platform and pipeline code

Test layers

Platform Python code is tested in layers. Unit tests cover pure functions such as feature transforms and config parsing, using a test runner with fixtures and parametrised cases. Contract tests check that a pipeline step accepts and emits the expected schema. Integration tests run a miniature pipeline on a tiny dataset to prove the steps connect, which is cheap insurance against failures that would otherwise show up hours into a real run. CI runs these across the supported Python versions inside a fresh environment built from the lock file, so the test also verifies that the package installs cleanly.

Typing, linting and validated configuration

Static checks as a platform default

Type hints plus a static type checker catch a large class of errors before a job is ever scheduled, which matters when a failure costs GPU hours. A linter and formatter remove style debate and run as pre-commit hooks and in CI. Configuration, a known debt source in ML systems, is parsed into typed models at startup so a wrong learning rate type or a missing path fails fast with a clear message. Project templates bake all of this in, so every new repository starts with the same tool settings in its pyproject.toml.

Sources

Related lessons

Practice this
MLOps and the ML lifecycle: from experiment to production systemKubernetes for ML workloads: containers, deployments and jobs