Practice: Python for integrations, services and data transformation
Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.
Main tool: Python · Python lets you practise the whole lesson in one language on a laptop, from a throwaway parsing script to typed boundaries, concurrent I/O, a small network service and a pinned environment, using only the interpreter and its standard library. · Alternatives: TypeScript on Node.js shares one language with web front ends but has a thinner data and AI library ecosystem; Go ships a single static binary and has strong concurrency but is slower to write for throwaway data reshaping; Java or Kotlin fit customers standardised on the JVM at the cost of heavier project setup; shell scripts with command-line JSON tools suit one-off glue but do not grow into typed services.
Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1
Setup (one time)
1. Install a current Python 3 release and start the interpreter from a terminal to confirm which version runs.
Every later step assumes you know exactly which interpreter is on your path. Customer machines often have several, so checking this first becomes a habit.
Explained in: 2. Using the Python Interpreter — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: The interpreter starts, prints its version banner and exits cleanly when you leave it.
2. Create a project folder and a virtual environment inside it, then activate the environment.
An isolated environment keeps this project's dependencies separate from the system Python, which is the basis of running the same way in someone else's environment.
Explained in: 12. Virtual Environments and Packages — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: Your shell prompt shows the environment name and the interpreter that runs is the one inside the project folder.
3. Install one third-party package into the active environment with pip, for example Pydantic, and list what is installed.
You need to see how packages land in the environment before you pin them later. Pydantic is the library a team would use for the typed models you first build by hand in this path.
Explained in: 12. Virtual Environments and Packages — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: The pip list output shows the package and its version inside the virtual environment only.
Steps
1. Run the interactive interpreter and experiment with strings and lists: slice, index, concatenate and mutate a list of sample customer field values.
Strings and lists are the raw material of every integration script. The interactive prompt is the fastest way to probe unfamiliar customer data.
Explained in: 3. An Informal Introduction to Python — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: You can predict the result of a slice or a list mutation before pressing enter and the interpreter agrees.
2. Write a first script file that reads a file path from its command-line arguments and prints each line of that file with its line number.
Field scripts are driven by arguments, not edited constants. Passing the input path in is what lets the same script run against any customer export.
Explained in: 2. Using the Python Interpreter — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: Running the script against two different files prints each file's numbered lines without changing the script.
3. Write a reshaping script that turns a list of flat records into a dictionary keyed by customer id, using dictionaries and comprehensions to rename, drop and group fields.
Parsing and reshaping data is the first subtopic of this lesson. Dictionaries and comprehensions are the core object model you will use for it every day.
Explained in: 5. Data Structures — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: The script prints a dictionary whose keys are customer ids and whose values hold only the renamed fields you kept.
4. Load a JSON file of deliberately messy records, validate each one, and write the clean records and the rejected records to two separate JSON files.
Real customer data has missing and malformed fields. A script that separates good rows from bad ones, instead of crashing on the first bad row, is what survives contact with it.
Explained in: 7. Input and output — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: Two output files exist, the record counts in them add up to the input count, and the script finishes without a traceback.
5. Write a validation function that raises a clear exception for each kind of bad record, and handle those exceptions in the loading loop so each rejection is recorded with its reason.
Explicit exceptions at the boundary turn silent data corruption into a readable report. They are the mechanism that validation libraries build on.
Explained in: 8. Errors and Exceptions — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: The rejected file lists a specific reason for each bad record, and a record with a missing id gives a different reason from one with a bad date.
6. Write type annotations on every function that crosses a boundary (the loader, the validator and the writer) and inspect the annotations the interpreter stores for each one.
Typed boundaries are the second subtopic. Annotations are the standard-library foundation that Pydantic models and a checker such as mypy read, so practise them plainly first.
Explained in: 4. More Control Flow Tools — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: Printing a function's annotations shows the parameter and return types you declared for all three boundary functions.
7. Create a module from the transformation code that can be imported or executed as a script, run it twice on the same input and compare the two saved outputs.
A module that runs both ways can be reused by a service and still run by hand. Comparing persisted outputs across runs proves the transformation is deterministic before you trust it on customer data.
Explained in: 6. Modules — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: Importing the module runs nothing, executing it writes the output files, and the outputs of the two runs are identical.
8. Package the code as a directory with separate submodules for loading, validating and writing, and import between them with intra-package references.
Packaging is the last subtopic. A real package layout is what makes the code installable and importable the same way on any machine.
Explained in: 6. Modules — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: You can import the package from the project root and call the validator through its package path from a fresh interpreter session.
9. Write coroutines that simulate slow I/O calls for each record, schedule them as tasks, and run the program with the asyncio runner to compare the elapsed time against a sequential loop.
Concurrent I/O is the third subtopic. Enrichment calls to customer systems are I/O bound, so running them as tasks is where most of the speedup in integration work comes from.
Explained in: A conceptual overview of asyncio — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: The concurrent version finishes in roughly the time of the single slowest call while the sequential version takes about the sum of all calls.
10. Configure a bounded worker pool with an asyncio queue so that only a fixed number of records are processed at once, then change the limit and rerun.
Unbounded concurrency gets you rate limited or overloads a customer system. A queue with a fixed number of workers is the standard way to cap it.
Explained in: High-level API Index — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: Log output shows that the number of in-flight records never exceeds the limit, and total time changes when you change the limit.
11. Start a small local network service using asyncio streams that accepts a request, runs your validation on the payload and writes back a response, then query it from a second terminal with the streams client example.
This covers the small HTTP services subtopic using Python alone. Serving over raw streams shows what a framework such as FastAPI running on Uvicorn handles for you once you add it.
Explained in: Streams — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: The client terminal prints a response that reflects the validation result while the server terminal logs the connection.
12. Reproduce the project in a production-like way: freeze the installed packages to a requirements file, create a brand new virtual environment, install from that file and run the service and the transformation there. If Docker is available locally, repeat the same install inside a container as the closest substitute for a customer host.
A pinned environment rebuilt from scratch is the proof that the work runs the same way inside someone else's environment. It is the same sequence a CI job or a container build would automate.
Explained in: 12. Virtual Environments and Packages — Python 3.15.0 documentation (docs.python.org), checked 2026-10-11
Done when: The fresh environment lists exactly the pinned versions, and both the transformation and the service give the same results as in the original environment.
Back to Python for integrations, services and data transformation