AWS and EKS foundations for an ML platform

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: AWS · also covers AWS EKS, Cloud infrastructure · Step 4 of 35 in MLOps / ML platform engineer · version 1

Amazon EKS is the managed Kubernetes service many ML platforms are built on: AWS runs the control plane and the team owns the workloads, the node strategy and the integrations. The foundations to get right are identity, networking, compute choice and storage, because ML workloads are heavy users of all four. The Well-Architected Machine Learning Lens gives a review framework for the resulting design.

How it is done in practice

Teams define the whole foundation in infrastructure as code: VPC and subnets, the EKS cluster, add-on versions, node pools or autoscaler configuration, IAM roles and the service account mappings. A new ML team is onboarded by a module that creates a namespace, a service account, a scoped IAM role and a bucket prefix. Cluster upgrades are rehearsed in a non-production cluster first. The EKS best practices guide is used as the baseline for security, reliability, networking and scalability settings, and the ML Lens is applied when a new workload type goes to design review.

Technologies

Diagram

flowchart TB
  subgraph AWS_managed[AWS managed]
    CP[EKS control plane]
  end
  subgraph VPC[Customer VPC private subnets]
    NG1[CPU node pool]
    NG2[GPU node pool]
    SA[Service account] --> POD[Training or serving pod]
    POD --> NG2
  end
  CP --> NG1
  CP --> NG2
  SA -->|assumes| ROLE[IAM role]
  ROLE --> S3[(S3 datasets and models)]
  POD -->|VPC endpoint| S3
  POD -->|pulls image| ECR[(ECR)]
  KA[Node autoscaler] -->|launches instances| NG2

flowchart TB
  subgraph AWS_managed[AWS managed]
    CP[EKS control plane]
  end
  subgraph VPC[Customer VPC private subnets]
    NG1[CPU node pool]
    NG2[GPU node pool]
    SA[Service account] --> POD[Training or serving pod]
    POD --> NG2
  end
  CP --> NG1
  CP --> NG2
  SA -->|assumes| ROLE[IAM role]
  ROLE --> S3[(S3 datasets and models)]
  POD -->|VPC endpoint| S3
  POD -->|pulls image| ECR[(ECR)]
  KA[Node autoscaler] -->|launches instances| NG2

Code

# Map a Kubernetes service account to an IAM role with EKS Pod Identity
aws eks create-pod-identity-association \
  --cluster-name ml-platform \
  --namespace team-a \
  --service-account trainer \
  --role-arn arn:aws:iam::ACCOUNT_ID:role/team-a-trainer

# Verify from inside a pod that uses the service account
aws sts get-caller-identity

Subtopics

EKS architecture and the responsibility split

What AWS manages and what the team manages

EKS runs the Kubernetes control plane for you across multiple availability zones and handles its scaling, patching and availability. The data plane is your choice: managed node groups of EC2 instances, self-managed nodes, Fargate for serverless pods, or modes in which AWS also automates node provisioning. Cluster capabilities come as add-ons such as the VPC CNI, CoreDNS, kube-proxy and storage drivers, each with its own version to keep compatible. Upstream Kubernetes conformance means standard tooling and operators work unchanged. Version upgrades remain a team responsibility and need a rehearsed process, since Kubernetes versions leave standard support on a fixed schedule.

IAM and identity for pods

IAM building blocks

IAM controls who is authenticated and what they are authorised to do. The pieces are principals (users, roles, federated identities), policies written as JSON documents that allow or deny actions on resources, and roles that are assumed to obtain temporary credentials. Roles with short-lived credentials are preferred over long-term access keys. Least privilege means granting only the actions and resources a task needs, for example read access to one bucket prefix for a training job.

Giving a pod an AWS identity

Pods should not inherit the broad role of the node they run on. Instead a Kubernetes service account is mapped to an IAM role, either through the cluster OIDC provider with a trust policy (IAM roles for service accounts) or through EKS Pod Identity associations. The AWS SDK inside the pod picks up temporary credentials automatically. On an ML platform this gives per-team and per-workload scoping: a training job can read its dataset and write its checkpoints, an inference service can only read model artifacts, and every access is attributable in audit logs.

VPC networking for EKS

Pod networking and address planning

With the default VPC CNI each pod receives a real IP address from the VPC subnets, which makes pods directly routable but means large clusters consume addresses quickly. Subnet sizing therefore has to account for peak pod count, not node count, and options such as prefix delegation or secondary address ranges exist for when space runs short. Nodes normally sit in private subnets across several zones, with load balancers in front for ingress. VPC endpoints for object storage and the container registry keep heavy data and image traffic on the AWS network and avoid NAT gateway charges, which can be significant for dataset downloads.

Compute and node strategy for ML

Choosing how nodes appear

ML clusters mix cheap general-purpose nodes for control and pipeline pods with accelerated instances for training and inference. Managed node groups give fixed pools per instance type backed by auto scaling groups. A node autoscaler such as Karpenter instead watches pending pods and launches instances that fit them directly, which suits bursty and heterogeneous GPU demand, and it can consolidate underused nodes. GPU nodes use accelerated machine images with drivers preinstalled and carry taints so only GPU workloads land there. Scaling GPU pools to zero when idle is one of the largest cost levers.

Spot, zones and storage

Spot capacity is far cheaper but can be reclaimed at short notice, so it fits checkpointed training and batch inference, not latency-critical serving. Capacity for specific GPU types is uneven across zones, and a job with a zonal persistent volume is pinned to that zone, which can leave it unschedulable. Common storage choices are object storage for datasets and model artifacts, block volumes for per-pod scratch, and shared file systems for datasets many pods read at once. Image pull time for large images is managed with registry caching and prewarmed nodes.

The Well-Architected Machine Learning Lens

A review framework for ML workloads

The Machine Learning Lens extends the AWS Well-Architected Framework to ML workloads. It walks through the ML lifecycle phases, from business goal identification and problem framing through data processing, model development, deployment and monitoring, and for each phase lists best practices under the framework pillars: operational excellence, security, reliability, performance efficiency, cost optimisation and sustainability. It is cloud guidance, not a product, and is used as a checklist in design reviews. Typical findings concern missing lineage tracking, no rollback plan for a model, unscoped data access, and training resources that are not right-sized or not shut down.

Sources

• docWhat is Amazon EKS? (docs.aws.amazon.com)
• guideAmazon EKS Best Practices Guide (docs.aws.amazon.com)
• guideMachine Learning Lens - AWS Well-Architected Framework (docs.aws.amazon.com)
• docWhat is IAM? (docs.aws.amazon.com)

Related lessons

• Kubernetes for ML workloads: containers, deployments and jobs
• Reproducible environments: GitOps with Argo CD and infrastructure as code with Terraform
• Security, identity and access, and compliance for ML platforms
• Autoscaling and cost efficiency: right-sizing, spot mix and GPU utilization
• GPU compute on Kubernetes with the NVIDIA stack
• Managed deployment and serving with Amazon SageMaker
• Storage for ML: lakehouse, warehouse and operational data stores
• System design for multi-tenant ML infrastructure: isolation, quotas and backpressure