Practice: Experiment tracking with MLflow
Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.
Main tool: MLflow · MLflow is open source, runs fully on a laptop, and exposes the same client API, backend store and artifact store split that a platform engineer has to operate in production. · Alternatives: Weights and Biases is a hosted tracker with richer dashboards but a paid tier, Neptune and Comet are similar hosted services, ClearML and Aim are open-source and self-hosted like MLflow (ClearML adds orchestration, Aim focuses on fast run comparison), TensorBoard only visualises metrics and has no registry, and DVC tracks experiments in Git instead of on a server.
Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1
Setup (one time)
1. Create a fresh Python virtual environment and install MLflow into it with pip.
A clean environment keeps the MLflow version pinned and separate from other projects. Every later step uses this one install for the client, the UI and the server.
Explained in: Tracking Experiments with Local Database | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: Running the mlflow command with the version flag prints a version number.
2. Point MLflow at a local SQLite database by setting the tracking URI as the tutorial shows.
A database backend is the default path for current MLflow and behaves like the production store. It avoids the legacy file store from the first run onward.
Explained in: Tracking Experiments with Local Database | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: A SQLite database file appears in your working directory after the first logged run.
3. Install Docker and Docker Compose and confirm that containers start on your laptop.
The last steps run PostgreSQL and an S3-compatible object store in containers. This replaces paid cloud services with a local substitute.
Explained in: Remote Experiment Tracking with MLflow Tracking Server | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: Both the docker and the compose version commands print a version.
Steps
1. Run the quickstart training script with autologging turned on, then open the tracking UI and find the run.
This is the shortest route to a real run. It shows what MLflow captures without any logging code.
Explained in: MLflow Tracking Quickstart | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The UI lists one run with parameters, metrics and a model that you did not log by hand.
2. Write a second script that creates a named experiment, starts a run and logs params, metrics, tags and one artifact file explicitly.
Explicit logging makes the data model concrete: experiments hold runs, and runs hold params, metrics, tags and artifacts. You need it whenever autologging does not cover a library.
Explained in: MLflow Tracking APIs | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The new experiment shows a run whose params, metrics and artifact match exactly what your script logged.
3. Start a tracking server with an explicit relational backend store URI and log a run to it.
The backend store holds run metadata such as params, metrics and tags. Choosing and configuring it is the first decision when operating the server.
Explained in: Backend Stores | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The server starts against your database, and the run is still listed after you restart the server.
4. Configure a separate artifact store location for the server and log a model file to it.
Artifacts are stored apart from metadata because they are large files. Seeing the two stores side by side explains the server architecture.
Explained in: Artifact Stores | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The artifact files appear under the location you configured, and the run page shows that artifact URI.
5. Configure autologging for one library only, change its options such as model logging, and combine it with one explicit metric.
A platform team makes autologging the default but must control what it records. Mixing it with explicit calls is the usual pattern in real training code.
Explained in: Automatic Logging with MLflow Tracking | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: A new run contains the autologged values plus your explicit metric, and the option you turned off is absent.
6. Run the training script several times with different hyperparameters, then query the runs programmatically with a filter and an ordering on one metric.
Comparison is the purpose of tracking. A programmatic query is what automation uses to pick the best run.
Explained in: ML Experiment Tracking | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The query returns a table of runs sorted by your metric, and the top row matches the best run shown in the UI.
7. Register the model from the best run in the Model Registry with the Python API.
Registration turns a run artifact into a named, versioned model. That version can be traced back to the run that produced it.
Explained in: ML Model Registry | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The Models page shows your registered model at version 1 with a link back to the source run.
8. Read the deployment section of the ML overview and note which local serving options exist for a registered model.
Tracking feeds serving, so you should know where a tracked model goes next. This page is the entry point to the serving guides.
Explained in: MLflow AI Engineering Platform | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: You can name the local serving route you would use for your registered model and which guide covers it.
9. Log the training dataset as a tracked input of a run, with its source and digest.
A result is reproducible only when the run records the data as well as the code and params. Dataset tracking stores that lineage on the run.
Explained in: ML Dataset Tracking | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The run page shows the dataset name, digest and source next to its params and metrics.
10. Enable system metrics logging through the environment variable and rerun the training script without changing its code.
Environment variables are how a platform makes logging the default for every job and CI run. System metrics also record the resource context of each run.
Explained in: System Metrics | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: The run shows a system metrics section with CPU and memory charts although the script was not edited.
11. Create the compose file from the remote tracking tutorial that defines PostgreSQL and an S3-compatible MinIO store, and start the containers.
This is a local substitute for a managed database and cloud object storage. It gives the same backend and artifact store split as production.
Explained in: Remote Experiment Tracking with MLflow Tracking Server | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: Both containers report as running, and a bucket for artifacts exists in the MinIO console.
12. Launch the tracking server against the PostgreSQL backend and the MinIO artifact store, then log a run to it from a separate client shell.
A shared server that proxies artifacts and keeps metadata in a real database is the setup a team relies on. Logging from a separate client proves the full path works.
Explained in: Remote Experiment Tracking with MLflow Tracking Server | MLflow AI Platform (mlflow.org), checked 2026-10-11
Done when: A run logged from the client appears in the server UI, with metadata in PostgreSQL and artifact files in the MinIO bucket.