AWS 377: AWS CDK Pipelines and environment promotion
Why this lesson matters
CDK Pipelines can synthesize, publish assets, update themselves, and deploy stages across accounts and Regions. That convenience creates a powerful supply-chain and trust path. Safe promotion requires deterministic source, bootstrap trust, environment-specific assemblies, ordered waves, tests, approvals, change visibility, and recovery.
Pipeline model
| Step | Input/output | Evidence |
|---|---|---|
| Source | Reviewed commit | Revision and trigger |
| Synth | Cloud assembly | Tool lock, tests, manifest/template hashes |
| Self mutation | Pipeline definition | Old/new pipeline versions and change review |
| Asset publication | File/image hashes | S3 version or ECR digest per environment |
| Prepare/deploy | CloudFormation change | Change set, stack events, role session |
| Wave/stage tests | Runtime endpoints | Test results, alarms, release identity |
| Approval | Exact execution/risk | Approver, expiry, evidence link |
A CDK Stage groups stacks. A pipeline stage/wave sequences deployment. Stacks inside a wave can deploy concurrently when dependencies allow; waves are sequential. Model dependencies deliberately and avoid cross-stack exports that make independent rollback impossible.
Self mutation lets a source change update the pipeline before later stages continue. It is useful for pipeline-as-code but means a pull request can alter its own controls. Protect source review, synth action, pipeline role, bootstrap trust, and mutation permissions. High-control environments may use a separately governed pipeline or explicit update process.
Cross-account and environment trust
Every target account/Region needs compatible modern bootstrap resources. The bootstrap template trusts the approved pipeline account/principal for deployment and asset publication; execution policies bound CloudFormation authority. Record qualifier, bootstrap version, KMS policy, role trust, permissions boundary, artifact bucket, and environment owner.
Do not pass environment-specific secrets through CDK context. Synthesize deterministic configuration references and resolve secrets at runtime. Assets may be replicated/published per environment, but prove that their content hash or image digest corresponds to the approved revision.
Separate development, test, and production accounts. A typical flow is source/synth, security-policy gates, development deployment/test, preproduction deployment/integration/load/recovery tests, approval, production canary, bake, then wider Regions. Promotion means the same source and artifacts with approved environment configuration, not rebuilding arbitrary code.
Environment configuration must be versioned and reviewed beside the release manifest. Validate account and Region before every deployment, reject wildcard production targets, and expose release identity from each workload. If one Region fails after another succeeds, freeze expansion, preserve both states, assess customer/data compatibility, and choose regional rollback or forward repair explicitly. Pipeline status alone cannot decide that recovery.
Tests, approvals, and rollback
Pre-deployment steps validate template, policy, quota/capacity, current health, and change set. Post-deployment steps verify exact release and user outcomes. Manual approval should identify execution, source/assembly hashes, environments, replacements/deletions/IAM changes, test evidence, alarms, rollback limits, and expiry.
CloudFormation rollback handles stack operations, not external writes or all pipeline changes. Keep known-good source, pipeline definition, assets, bootstrap compatibility, and data migration plan. If self mutation breaks the pipeline, recovery may require deploying a known-good pipeline stack through a controlled break-glass path. Test this path.
Read-only evidence and design lab
aws codepipeline get-pipeline --name PIPELINE --region ap-south-1
aws codepipeline list-pipeline-executions --pipeline-name PIPELINE --max-results 10 --region ap-south-1
aws cloudformation describe-stacks --stack-name CDKToolkit --region ap-south-1
aws cloudformation list-change-sets --stack-name TARGET_STACK --region ap-south-1
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole --region ap-south-1Design a pipeline account plus dev, test, and production accounts in two Regions. Define bootstrap trust and execution boundaries; one synth; two pre-production waves; approval; production canary Region; bake; second Region; post-tests; notification; and rollback decision. State concurrency and failure behavior for every stack.
Review a supplied assembly and execution. Match commit, template/asset hashes, pipeline version, assumed roles, change sets, stack events, tests, approval, release identity, and alarms. Diagnose 16 cases: stale bootstrap, qualifier mismatch, trust denial, execution policy denial, asset KMS denial, synth nondeterminism, context changed, self-mutation loop, removed control, dependency deadlock, concurrent stack failure, test timeout, expired approval, replacement surprise, first-Region regression, and rollback asset missing.
Cost, cleanup, and acceptance
Price CodePipeline, CodeBuild, S3/KMS, ECR, logs, notifications, test resources, duplicate environments, NAT/data transfer, target resources, and retained assets. This lesson creates nothing. Live cleanup must preserve audit evidence and remove pipeline, action resources, assets, bootstrap only when no stacks/pipelines depend on it, and target test resources.
Submit trust map, stage/wave graph, deterministic promotion manifest, control-change policy, test/approval contracts, change-set review, regional rollout, recovery procedure, sixteen failures, cost, and cleanup. Pass requires bounded cross-account trust, one traceable revision, reviewed self mutation, runtime gates, and tested pipeline plus workload recovery.