AWS 416: Block a noncompliant release and remediate it through the pipeline
Why this lesson matters
A trustworthy pipeline must reject a deliberately bad release before production, explain exactly why, preserve evidence, require correction in source, rebuild once, promote the corrected immutable artifact and verify runtime compliance. A bypass that makes the pipeline green is not remediation.
Scenario and controls
The fictional service ships as an ECR image plus CloudFormation. Candidate A contains a critical fixable package, an S3 bucket without required retention/encryption controls, an overbroad deployment permission and a secret-like test string. Production currently runs digest P under alias/traffic controls.
The pipeline stages are source, deterministic build/SBOM/provenance, unit/integration tests, IaC lint/plan/policy, secret scan, image scan, sign, candidate deploy, runtime/security tests, approval, production deployment, bake/alarms and evidence closure. Each gate consumes an immutable digest and emits a signed or protected machine-readable result.
| Gate | Candidate A result | Required correction/proof |
|---|---|---|
| Secret scan | Block | Remove value, rotate if real, rescan history |
| IaC policy | Block | Add controls and prove change set |
| IAM analysis | Block | Narrow actions/resources/PassRole |
| Inspector/image policy | Block | Update package/base, rebuild/rescan |
| Signing/provenance | Not reached | Sign only corrected digest |
| Runtime canary | Not reached | User/security tests on corrected bytes |
Evidence-driven execution
First predict which stage blocks and why. Run the supplied Candidate A reports through the policy evaluator. Record execution/action IDs, source commit, artifact digest, tool/rule/database version, finding ID, severity/context, policy decision and timestamps. Assert production digest P and deployment API history did not change.
Triage each failure as true positive, tool/error/coverage problem or possible exception. An exception needs owner, rationale, scope, compensation and expiry and cannot be created by the same role that failed the gate. In this lab every seeded defect is corrected, not waived or suppressed.
Correct source in a new reviewed commit: remove and investigate the secret-like value; update vulnerable dependency/base; add S3 versioning/encryption/retention design; narrow deployment role and PassRole. Do not modify generated reports, scan output, candidate registry object or deployed resources by hand. Rebuild Candidate B once and bind all evidence to its new digest.
Rerun every gate, including those that passed before, because dependency and IaC changes can create new failures. Prove successful scan coverage rather than only zero findings. Sign Candidate B after policy passes, copy/promote the same digest, deploy to isolated candidate, and test expected/denied operations.
Approval, deployment and recovery
The approval package includes A failure evidence, source diff, B digest/provenance/SBOM/signature, complete gate results, candidate tests, production health, rollout/rollback and data compatibility. The approver cannot modify pipeline/source/artifact. Revalidate evidence and current health immediately before release.
Deploy B progressively. Verify runtime image/task/pod digest equals approved B; verify user SLI, logs/alarms, secret absence, IAM denials and S3 controls through a bake period. If outcome fails, shift traffic to P or apply the planned recovery while preserving B evidence. Do not rebuild P by tag. Reconcile queues/data before closure.
~~~bash sha256sum supplied/candidate-a/manifest.json supplied/candidate-b/manifest.json aws ecr describe-images --repository-name app --image-ids imageDigest=sha256:DIGEST aws cloudformation describe-change-set --stack-name APP --change-set-name CANDIDATE_B aws accessanalyzer validate-policy --policy-type IDENTITY_POLICY --policy-document file://deploy-policy.json aws codepipeline get-pipeline-execution --pipeline-name PIPELINE --pipeline-execution-id ID ~~~
Commands are adapted to supplied evidence by default. An optional isolated account implementation must set a budget, use nonproduction names, protect logs and perform two-pass cleanup. Never weaken organization controls to finish the lab.
Failure injects and assessment
The assessor injects: scanner database unavailable, scan completes with unsupported coverage, policy result belongs to A while artifact is B, signing role is overbroad, destination promotion changes digest, approval expires, stage condition is overridden, canary passes but production alarm fires, rollback target was lifecycle-deleted, and secret appears in a failed build log. For each, stop safely, preserve evidence and choose retry, reject, escalate, compensate or recover.
Diagnose 24 cases: mutable tag, source/artifact mismatch, stale SBOM, unsigned report, scan failure called clean, unsupported package ignored, severity-only exception, suppression hides seed, IaC plan differs, policy edited after result, broad PassRole remains, secret rotated but history public, test bypass, gate order signs bad artifact, builder can approve, approval package mutable, stale approval, artifact rebuilt per environment, destination digest mismatch, runtime uses tag, alarm missing data, rollback incompatible, evidence deleted, and production mutation occurred during blocked run.
Metrics, cleanup and acceptance
Measure gate coverage, block correctness, false-positive/exception rate, time to explain/fix, evidence freshness, artifact promotion identity, denied-action proof, deployment/bake result and recurrence. Costs include build/scanning, ECR/S3/KMS, pipeline actions, candidate runtime, logs and retained evidence.
Submit both execution dossiers, immutable manifests, source correction, policy/findings analysis, role separation, approval, runtime/negative tests, rollback result, cleanup evidence and 24 diagnoses. Pass requires Candidate A blocked before production, no bypass, Candidate B rebuilt once and promoted by digest, production verified, and a complete audit chain.