Practice: Managed deployment and serving with Amazon SageMaker

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Main tool: Amazon SageMaker · Amazon SageMaker is the tool the lesson is about, and its model, endpoint configuration and endpoint objects, its four inference options and its Application Auto Scaling integration can only be learned by creating them yourself. · Alternatives: KServe and Seldon Core serve models on a Kubernetes cluster the team runs itself, BentoML packages a model as a container the team can host anywhere, and Vertex AI and Azure Machine Learning offer comparable managed endpoints tied to other clouds, while SageMaker has AWS run the instances, load balancing and health checks.

Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1

Setup (one time)

1. Create or pick an AWS account and an IAM execution role with SageMaker and S3 access; this is paid cloud, so plan to use the smallest CPU instance type and to delete resources after each session.
Every hosting call runs under an execution role that must be able to read the model artifact and pull the container image. Sorting out permissions first avoids confusing failures later.
Explained in: Deploy models for real-time inference - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: You can name the role ARN and the S3 bucket you will use, and a caller identity check from your laptop returns your account.
2. Install the SageMaker Python SDK, boto3 and the AWS CLI in a fresh virtual environment on your laptop and configure a default region.
The whole path is driven from the laptop through these three clients. A clean environment keeps the versions known.
Explained in: Deploy models for real-time inference - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: Importing the SDK and creating a SageMaker client in a Python session works without a credentials or region error.
3. Prepare a small CPU model artifact in S3 and choose the inference container image that will serve it.
Managed hosting needs exactly two things from you: a model artifact and a container. Having both ready makes the later steps about the platform and not about the model.
Explained in: Deploy models for real-time inference - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: The model archive is listed in your bucket and you have written down the image URI you will deploy.
4. Check the auto scaling prerequisites and confirm the service-linked role that Application Auto Scaling uses for SageMaker endpoints.
Scaling calls fail when the caller lacks the needed permissions or the service-linked role cannot be created. It is cheaper to find that out before an endpoint is running.
Explained in: Auto scaling prerequisites - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: Your identity has the listed permissions and you can state which service-linked role will manage scaling for your endpoints.

Steps

1. Run a first deployment of your model to a real-time endpoint with the SageMaker Python SDK on one small CPU instance; this is paid cloud, and the closest local substitute is the SDK local mode, which runs the same container in Docker.
One end-to-end deployment shows what AWS does for you: provisioning, health checks and a stable HTTPS endpoint. Everything after this refines that first run.
Explained in: Deploy the model to Amazon EC2 - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: The endpoint status reads InService in the console or in a describe call.
2. Run an inference request against the endpoint from your laptop with the boto3 runtime client.
Invoking through the runtime API, not the SDK predictor, is how real client services call an endpoint. It also shows the content type and payload the container expects.
Explained in: Invoke models for real-time inference - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: You get a prediction back and can print the response body and the name of the variant that served it.
3. Create the same deployment again as three explicit objects with the AWS CLI: a model, an endpoint configuration and an endpoint.
The SDK hides the object model. Building it by hand shows that the model binds a container image to an artifact and a role, the endpoint configuration lists production variants, and the endpoint is the running instance of one configuration.
Explained in: Deploy models for real-time inference - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: Describe calls return all three objects and you can point at where the image, the artifact location and the instance type are each recorded.
4. Read the hosting options page and write a short decision note that assigns real-time, serverless, asynchronous and batch to four workloads you know.
Choosing the inference option is the main platform decision in this lesson. It depends on latency, payload size, traffic shape and whether the instance may sit idle.
Explained in: Hosting options - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: Your note names one option per workload with the deciding constraint, such as payload size, cold start tolerance or offline scoring.
5. Write two target tracking policy configurations as JSON files, both on the predefined invocations per instance metric but with different target values, and compare them side by side.
Keeping scaling policies as files makes them reviewable and repeatable. Comparing two targets forces you to reason about how many requests one instance should carry.
Explained in: Define a scaling policy - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: Two policy files exist in your project folder and you can explain which one scales out earlier and why.
6. Register the production variant of your endpoint as a scalable target with a minimum and maximum instance count using the AWS CLI.
Application Auto Scaling only acts on resources that are registered with it. The minimum and maximum set the hard bounds for cost and capacity.
Explained in: Register a model - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: A describe scalable targets call lists your variant with the minimum and maximum you set.
7. Apply one of your policy files to the registered variant, then send a burst of requests from a loop on your laptop.
Applying the policy creates the CloudWatch alarms that drive scaling. Generating load is the only way to see the policy act.
Explained in: Apply a scaling policy - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: The policy and its CloudWatch alarms are listed, and the scaling activity history shows a scale-out or the alarm changes state under load.
8. Configure a second endpoint configuration with two production variants and explicit traffic weights, then update the endpoint to it.
Variants and weights are the basic mechanism for shifting traffic to a new model version gradually. An update swaps configurations without taking the endpoint down.
Explained in: Deploy models for real-time inference - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: Repeated invocations return responses from both variants in roughly the proportion of the weights you set.
9. Run a rollback by updating the endpoint back to the previous endpoint configuration, then list and tidy up the endpoints you no longer need.
Because endpoint configurations are immutable, the earlier one is still there and rollback is a single update. Practising it once makes it routine when a release goes wrong.
Explained in: Endpoints - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: The endpoint describes as InService on the original configuration and every invocation is served by the original variant.
10. Write a deployment script that suspends scaling activities on the variant, performs the endpoint update, then resumes scaling.
Scaling activity in the middle of an update can interfere with the rollout. Scripting the suspend, update and resume sequence makes each release repeatable from a pipeline.
Explained in: Temporarily turn off scaling policies - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: Running the script twice gives the same result, and the scalable target shows scaling suspended during the update and resumed after it.
11. Run a batch transform job over a small S3 dataset against the candidate model to evaluate the new variant offline before it takes live traffic.
Batch transform scores a whole dataset with no persistent endpoint. That makes it the offline option and a safe way to test a variant before shifting production traffic to it.
Explained in: Batch transform for inference with Amazon SageMaker AI - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: The job status is Completed and the output files in S3 hold one prediction per input record.
12. Configure a production-like endpoint that scales to zero instances when idle and scales back out on the first request; this uses inference components and is paid cloud, so keep the maximum at one small instance.
Scale to zero combines the object model, the scalable target and the scaling policies from the earlier steps into one cost-aware setup, close to how low-traffic production endpoints are run.
Explained in: Scale an endpoint to zero instances - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: With no traffic the endpoint reports zero instances, and after a new request the instance count returns to one and the request is served.

Back to Managed deployment and serving with Amazon SageMaker