Practice: Staged rollouts: shadow, canary, champion/challenger and rollback
Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.
Main tool: Amazon SageMaker · Amazon SageMaker offers shadow tests, blue/green traffic shifting with canary, linear and all-at-once modes, rolling updates and alarm-driven automatic rollback as managed endpoint features, so every strategy in this lesson can be practised against one endpoint. · Alternatives: Argo Rollouts drives canary and blue/green steps on Kubernetes with metric analysis from Prometheus; Istio provides the traffic splitting and mirroring underneath such rollouts; KServe gives a canary traffic percentage per inference service on your own cluster; Seldon Core adds shadow and experiment routing on Kubernetes; Amazon CloudWatch is not a rollout tool but supplies the alarms that gate a SageMaker rollout.
Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1
Setup (one time)
1. Read the pricing section and plan every exercise on the smallest CPU instance type, since SageMaker endpoints need a paid AWS account and have no laptop mode; the closest local substitute is drafting and validating each request file with the AWS CLI skeleton option before sending it.
Endpoints bill for as long as they exist, and blue/green updates briefly run two fleets. Knowing the cost model up front lets you keep the practice cheap and delete resources on time.
Explained in: What is Amazon SageMaker AI? - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: You have written down the instance type you will use, a spending limit, and the list of resources to delete after each session.
2. Configure AWS security credentials on the laptop for a non-root identity allowed to manage SageMaker endpoints, CloudWatch alarms and EventBridge rules.
Every later step calls the SageMaker API from your laptop. Scoped credentials are also what a deployment pipeline would use.
Explained in: AWS General Reference - AWS General Reference (docs.aws.amazon.com), checked 2026-10-11
Done when: An AWS CLI identity check from the laptop returns your non-root user or role.
3. Create a baseline real-time endpoint from a small CPU model and one endpoint configuration; this is the champion that every rollout will update.
Deployment guardrails only apply to updates of an existing endpoint that is in service. You need a stable starting fleet to shift traffic away from.
Explained in: CreateEndpoint - Amazon SageMaker (docs.aws.amazon.com), checked 2026-10-11
Done when: The endpoint shows the status InService and answers a test invocation.
4. Create the canary prerequisites: a second endpoint configuration for the new model version and at least one CloudWatch alarm on the endpoint, such as error count or latency.
The alarm is the automated check that decides whether a rollout continues or rolls back. Without it a staged rollout is only a slow deployment.
Explained in: Use canary traffic shifting - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: Two endpoint configurations exist and the alarm is listed in the OK state.
Steps
1. Query the baseline endpoint with DescribeEndpoint and identify the production variants, the endpoint configuration name, the status and the last deployment configuration fields.
These response fields are the object model you will watch during every rollout. Reading them on a quiet endpoint makes the later changes easy to spot.
Explained in: DescribeEndpoint - Amazon SageMaker (docs.aws.amazon.com), checked 2026-10-11
Done when: You can point to the variant name, its current weight and instance count, and the endpoint status in the response.
2. Create an EventBridge rule that matches the model package state change event, so that approving a new model version is the recorded trigger for starting a rollout.
Promotion and rollout should be linked: a version gets approved first and only then exposed to traffic. The event gives you an auditable hand-off between the two.
Explained in: Events that Amazon SageMaker AI sends to Amazon EventBridge - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: Changing the approval status of a model package produces a matching event at the rule target, such as a log group.
3. Create a shadow test in the console that adds the new model as a shadow variant and copies a percentage of live requests to it, on a paid CPU endpoint as there is no laptop mode.
A shadow variant sees real inputs but its responses are never returned to callers. It is the lowest-risk way to check latency and errors under production conditions.
Explained in: Create a shadow test - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: The shadow test is running and its dashboard shows latency and error metrics for the production and shadow variants side by side, while callers still receive only production responses.
4. Configure an all-at-once blue/green update to the new endpoint configuration with a baking period and your alarm in the auto rollback configuration.
This is the simplest blue/green form: a full new fleet, one traffic switch, then a watch period before the old fleet is removed. It teaches the baking period and termination wait before you add stepwise shifting.
Explained in: Use all at once traffic shifting - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: The endpoint goes through Updating to InService on the new configuration, and the old fleet is terminated only after the baking period ends.
5. Run a canary traffic shifting update from the CLI that sends a small canary of capacity to the green fleet, waits, then shifts the remainder.
Canary limits the blast radius of a bad version to a small share of traffic. It is the managed counterpart of a canary release with a Kubernetes rollout controller.
Explained in: Use canary traffic shifting - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: DescribeEndpoint shows the canary settings in the deployment configuration, and the endpoint finishes InService on the new configuration after the wait interval.
6. Configure a linear traffic shifting update with a fixed step size and wait interval, then compare how long it takes against the canary run.
Linear shifting gives more checkpoints than canary at the cost of a longer rollout. Timing both lets you choose a mode per model with evidence.
Explained in: Use linear traffic shifting - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: You have noted the total rollout time and number of traffic steps for the linear and the canary run.
7. Configure a rolling deployment with a rolling batch size and a rollback batch size on an endpoint with at least two instances.
Rolling updates replace capacity in batches instead of doubling the fleet. That matters when quota or cost rules out a full blue/green fleet.
Explained in: Use rolling deployments - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: The update completes batch by batch and the endpoint returns to InService without the instance count ever doubling.
8. Reproduce a failed rollout by forcing your alarm into the alarm state during a baking period and watching the automatic rollback.
A rollback you have never triggered is not a rollback plan. Rehearsing shows how long recovery takes and what the endpoint reports while it happens.
Explained in: Use rolling deployments - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: The endpoint returns to InService on the previous configuration, and you have recorded the time from alarm to full recovery.
9. Write the canary update as a script using the SageMaker Python SDK in a pinned environment, taking the endpoint name and new configuration name as inputs.
A rollout typed by hand cannot be reviewed or repeated. A pinned, parameterised script is what a pipeline job runs for every release.
Explained in: Automation and Environments (docs.aws.amazon.com), checked 2026-10-11
Done when: Running the script twice from a clean environment performs the same canary update both times with no manual edits.
10. Run a champion/challenger experiment by configuring two production variants with traffic weights, then compare their per-variant metrics alongside a business outcome you log yourself.
Shadow and canary answer whether the new version is safe; an online experiment answers whether it is better. Splitting real responses between variants is the only way to measure outcomes.
Explained in: Validation of models in production - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: Per-variant invocation counts match the configured weights, and you can name the winning variant from your logged outcome.
11. Create an EventBridge rule on the endpoint deployment state change event and route it to a notification target.
On-call engineers need to learn about a rollback from an alert, not by polling the endpoint. The event also gives a timeline for post-incident review.
Explained in: Events that Amazon SageMaker AI sends to Amazon EventBridge - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: A rehearsed rollback produces a notification that shows the deployment state transitions.
12. Deploy a full production-like release by chaining the pieces: approval event, shadow test, scripted canary update gated by alarms with auto rollback, and deployment state notifications; then delete all endpoints.
Each strategy covers a different risk, and in production they run as one sequence with automated gates. Running the chain end to end shows where the hand-offs can fail.
Explained in: Deployment guardrails for updating models in production - Amazon SageMaker AI (docs.aws.amazon.com), checked 2026-10-11
Done when: One approved model version reaches full traffic with no manual step after approval, a second deliberately bad version is rolled back automatically, and no endpoints remain afterwards.
Back to Staged rollouts: shadow, canary, champion/challenger and rollback