AWS 415: Manual approvals, separation of duties, emergency change, and audit trace
Why this lesson matters
Automation should remove repetitive waiting, but some high-impact or ambiguous changes need accountable human judgment. A click is not meaningful approval unless the approver is independent, sees immutable evidence, understands risk, acts before expiry and cannot silently replace the artifact afterward.
Risk-based authority model
Define proposer, reviewer, policy/security approver, business owner, deployer and post-deployment verifier. Separation of duties depends on risk: a routine reversible nonproduction change may be automated, while production IAM, data migration, cryptographic, network perimeter or irreversible changes need independent approval. Avoid requiring humans to approve facts that machines can validate.
| Role | Decision/evidence | Must not control alone |
|---|---|---|
| Proposer | Intent, source change and rollback | Final high-risk approval |
| Automated gates | Tests, policy, scan and provenance | Business-risk acceptance |
| Approver | Exact digest, impact, timing and evidence | Replace artifact/policy after approval |
| Deployer | Execute approved immutable release | Broaden own deployment authority |
| Verifier | User/security outcome and bake | Hide failed outcome |
| Auditor | Reconstruct identity/timeline | Routine mutation of evidence |
The approval package binds pipeline/execution, source revision, artifact digest, infrastructure/configuration plan, test/scan/policy results, affected accounts/Regions/resources, customer/data impact, dependency/compatibility, deployment strategy, rollback limits, maintenance window, current health and exception IDs. Generate a checksum or immutable reference so the approved package cannot drift.
Manual gates and conditions
A CodePipeline manual approval action pauses execution until an authorized identity approves or rejects it. Optional SNS notification, review URL and comments guide the approver. Rejection or no response within seven days fails the action. Treat that seven-day limit as an upper service behavior, not permission to use stale evidence; define a shorter risk-based expiry and revalidate health/artifact/policy before deployment.
Restrict PutApprovalResult to exact pipelines/stages/actions and approved roles. Federation/MFA, short sessions, CloudTrail and meaningful comments establish accountability. Notifications are not authorization. The review URL must not leak secrets or permit evidence replacement, and the approver must inspect the execution's exact revision.
Stage conditions can gate entry, react to failure or verify success using supported rules such as alarms, deployment windows or Lambda/commands. Conditions can sometimes be overridden, so restrict, log and review override authority. A manual approval does not replace alarm-based rollback, and an alarm cannot accept business/data risk.
Emergency change and break glass
An emergency process is a faster governed path, not no control. Define objective activation criteria, incident/change ID, authorized emergency roles, MFA/federation recovery, short session, minimal permissions, dual control where feasible, communication, continuous logging, automatic expiry and post-use credential/session revocation. Test access before an emergency without exercising destructive permissions.
When normal identity or pipeline control planes fail, emergency access needs an independently available path and protected credentials. It must not depend entirely on the failed provider. Alert immediately on assumption and every sensitive action. Preserve what changed, why normal process was unsafe, affected resources, commands/API request IDs, verification and rollback status.
Within a fixed period, conduct retrospective review, reproduce the change in source/IaC, run normal gates, reconcile drift, remove temporary access and create prevention actions. An emergency console fix left outside desired state will be overwritten or become hidden configuration debt.
Audit trace and workshop
~~~bash aws codepipeline get-pipeline-state --name PIPELINE aws codepipeline list-pipeline-executions --pipeline-name PIPELINE aws codepipeline get-pipeline-execution --pipeline-name PIPELINE --pipeline-execution-id ID aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=PutApprovalResult aws sts get-caller-identity ~~~
Build a trace joining source review, build/test IDs, artifact digest, policy results, approval identity/comment/time, role sessions, deployment APIs, runtime version, alarms, verification and ticket closure. CloudTrail proves API identity/session context, not that the human read or understood evidence, so preserve the review package and attestation.
Given ten changes, classify automated, single-approval, dual-approval or emergency paths. Design IAM and evidence for a production database migration and security-group repair. Reconstruct two supplied executions, detecting stale evidence, self-approval and artifact substitution.
Test 22 failures: proposer self-approves, shared approver role, broad approval permission, notification treated authorization, review URL mutable, digest absent, evidence stale, seven-day timeout misunderstood, approval copied across execution, artifact replaced after approval, stage condition override broad, alarm missing data, rollback impossible, verifier equals deployer, comment empty, emergency criteria vague, break glass depends on failed IdP, session too long, use not alerted, temporary policy retained, console fix not codified, and audit trace cannot bind runtime bytes.
Cost and acceptance
Price pipeline actions, SNS, evidence/log storage, KMS and engineering delay; approval design should reduce risk without creating habitual rubber stamps. This lesson creates nothing. Submit authority matrix, risk tiers, immutable package, IAM design, normal/emergency flows, two audit reconstructions and all diagnoses. Pass requires independent identity, exact artifact binding, expiry/revalidation, tested break glass and complete post-change verification.