Practice: Autoscaling and cost efficiency: right-sizing, spot mix and GPU utilization
Generated by Claude through GigAI 0.1.11.10 on 2026-10-12.
Main tool: AWS · AWS is where the lesson's node provisioning, Spot purchasing and cost attribution decisions are made, so practising against its own purchasing and cost management guidance ties every pod-level scaling choice to the bill it produces. · Alternatives: Google Cloud (GKE cluster autoscaler and node auto-provisioning with Spot VMs, billing export to BigQuery), Azure (AKS with KEDA as a managed add-on and Spot node pools), a self-managed Kubernetes cluster with Cluster Autoscaler and OpenCost or Kubecost (no cloud lock-in, but you run the capacity and cost tooling yourself).
Every step links into the tool's own documentation; links were checked on 2026-10-12 · version 1
Setup (one time)
1. Read the introduction of the Cost Optimization Pillar and note the practice areas you will apply to an ML inference and training workload.
The pillar gives the vocabulary used in every later step: matching supply to demand, cost-effective resources and expenditure awareness.
Explained in: Cost Optimization Pillar - AWS Well-Architected Framework - Cost Optimization Pillar (docs.aws.amazon.com), checked 2026-10-12
Done when: You have a one-page note listing the pillar's practice areas with one ML example next to each.
2. Sign in to the AWS Management Console with a free-tier account and a non-root user; nothing in this path needs paid capacity.
You need console access to see the cost and purchasing screens that the guides describe, even though the workloads themselves run on your laptop.
Explained in: What is the AWS Management Console? - AWS Management Console (docs.aws.amazon.com), checked 2026-10-12
Done when: The console home page opens for your non-root user and shows the Region you selected.
3. Create programmatic credentials for that user and configure a named profile for the command line interface on your laptop.
Later steps validate capacity and cost definitions from files, which needs credentials that are scoped and not the root account.
Explained in: AWS General Reference - AWS General Reference (docs.aws.amazon.com), checked 2026-10-12
Done when: A caller identity check from your terminal returns your user and account number.
4. Start a local Kubernetes cluster with kind or minikube as the stand-in for a managed EKS cluster, after reading how the managed orchestration options differ.
A managed cluster and its nodes are paid, so a local cluster is the closest substitute for practising pod autoscaling while you keep the AWS capacity model in mind.
Explained in: Choosing an AWS container service - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: The local cluster lists at least one Ready node and you can name the managed option it stands in for.
Steps
1. Write a map of the AWS cost management capabilities, grouping each service under see, plan, optimize or control for an ML platform.
This is the object model of cost work on AWS: you cannot attribute or reduce GPU and node spend until you know which service answers which question.
Explained in: Choosing an AWS cost management strategy - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: You have a table that places each cost management service in one group with the ML question it answers.
2. Compare the EC2 purchase offerings for three workloads: an always-on inference service, a bursty batch scoring job and a long training run.
On-Demand, Spot and commitment-based pricing behave differently under interruption and burst, and the right mix is what makes autoscaling cheap rather than just elastic.
Explained in: Choosing a purchasing option for Amazon EC2 - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: Each of the three workloads has a chosen offering and one sentence on the interruption risk you accept.
3. Write right-sizing criteria for your workload: target CPU, memory and GPU utilization, the request-to-usage ratio you accept, and the review interval.
Right-sizing comes before autoscaling because scaling an oversized pod or node only multiplies the waste; no GPU is needed, use CPU-only requests as the substitute on the laptop.
Explained in: Choosing an AWS cost management strategy - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: A written criteria sheet exists with numeric utilization targets for CPU, memory and GPU.
4. Create a Horizontal Pod Autoscaler for a small inference deployment on the local cluster, drive load at it, and record replica counts against CPU utilization.
The Horizontal Pod Autoscaler is the first scaling layer on any EKS-style cluster, and watching it react shows why correct resource requests from the previous step matter.
Explained in: Choosing an AWS container service - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: Replica count rises under load and falls back after it stops, and you have the timeline written down.
5. Configure a KEDA scaler on the local cluster that scales a batch scoring worker on queue depth, using a local queue as the substitute for a managed queue service, and scale it to zero when the queue is empty.
Event-driven scaling follows backlog instead of CPU, which fits bursty ML jobs, and scale to zero is where most of the idle cost disappears.
Explained in: Choosing an AWS application integration service - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: The worker runs zero replicas with an empty queue and several replicas after you enqueue a batch of messages.
6. Write a Karpenter node pool definition that mixes Spot and On-Demand capacity across several instance families; real provisioning needs paid capacity, so validate the file with a client-side dry run on the local cluster instead.
Node provisioning is the second scaling layer, and a hybrid Spot and On-Demand mix is how deadlines stay safe while most node hours are bought at the Spot price.
Explained in: Choosing a purchasing option for Amazon EC2 - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: The node pool file passes schema validation and states which workloads may land on Spot and which must not.
7. Configure the workload for Spot interruption: add a disruption budget, graceful termination and checkpointing, then drain a local node to simulate a reclaim.
Spot savings only hold when an interruption costs seconds rather than a lost training run, so the handling has to be tested before it is trusted.
Explained in: Choosing a purchasing option for Amazon EC2 - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: After the simulated drain the service stays within its replica floor and the batch job resumes from its checkpoint.
8. Create a cost attribution scheme: define tags and matching Kubernetes labels for team, model and environment, apply the labels on the local cluster, and choose the cost service that will report on them.
Cost per team and per model is impossible to reconstruct after the fact, so the tagging contract has to exist before the spend does.
Explained in: Choosing an AWS cost management strategy - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: Every pod in your test namespace carries the three labels and your note names the cost service that reports on each.
9. Write the queries for GPU utilization and cost per thousand inferences against a sample usage export loaded into a local SQL engine, after choosing which analytics service would hold the real data.
Idle GPU time is the largest avoidable cost in ML serving, and a unit-cost number is what lets you judge whether an autoscaling change helped; a laptop has no GPU, so use sample metric rows.
Explained in: Choosing an AWS analytics service - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: Your queries return a utilization percentage and a cost per thousand inferences from the sample data.
10. Write a container task and service definition that splits capacity between Fargate and Fargate Spot through capacity providers, and keep it in version control; do not deploy it, since running it is paid.
Declaring the Spot mix as a reviewed file is what makes the capacity decision reproducible instead of a console setting someone changed once.
Explained in: Architect for AWS Fargate for Amazon ECS - Amazon Elastic Container Service (docs.aws.amazon.com), checked 2026-10-12
Done when: The definition is committed with an explicit base and weight for each capacity provider and a reviewer can read the Spot share from it.
11. Configure usage metric alarms and a monthly budget as code alongside the capacity files, and run a validation of all of them in a local continuous integration job.
Automated checks catch a capacity or quota change before it reaches the bill, which is the control half of cost practice.
Explained in: Architect for AWS Fargate for Amazon ECS - Amazon Elastic Container Service (docs.aws.amazon.com), checked 2026-10-12
Done when: The local pipeline run fails when you break a capacity file on purpose and passes when you restore it.
12. Create a pinned capacity policy document for a production-like setup that combines your pod autoscaling targets, node pool mix, interruption handling, tags and the commitment level for the steady baseline.
A working engineer hands over one reviewed policy that says what scales, what is committed and what rides on Spot, so the whole path ends as a decision others can reproduce.
Explained in: Choosing a purchasing option for Amazon EC2 - AWS Decision Guides (docs.aws.amazon.com), checked 2026-10-12
Done when: The policy is committed next to the validated files and states baseline, burst and Spot shares with the latency objective each must meet.
Back to Autoscaling and cost efficiency: right-sizing, spot mix and GPU utilization