AWS 407: Secrets Manager rotation and secure pipeline consumption
Why this lesson matters
A secret is not secure merely because its value is encrypted. Architects must control who creates, reads, rotates, replicates and deletes it; coordinate credentials with the target system; prevent disclosure in pipelines and logs; and prove applications adopt a new version without an outage.
Secret, version and key model
Secrets Manager stores metadata plus encrypted secret versions. AWSCURRENT identifies the default retrieval version, AWSPENDING the candidate during rotation, and AWSPREVIOUS the prior version after successful rotation. Labels are movable pointers, not an ordered history. Retrieve by ARN and expected stage/version when deterministic rollback or testing requires it.
Use a structured value only for fields the consumer requires, such as username, password, host, port and database. Never store unrelated environments or multiple owners in one value merely to reduce secret count. Resource policies authorize cross-account readers; identity policies authorize callers; the KMS key policy/grants authorize decryption. All applicable layers must allow the request.
| Control | Owner | Required evidence |
|---|---|---|
| Secret/resource policy | Security and application | Narrow principals, actions and accounts |
| KMS key policy | Key owner | Intended decrypt path and denial test |
| Rotation schedule/window | Credential owner | Frequency, UTC window and collision analysis |
| Target credential change | Database/service owner | Candidate works and old behavior is understood |
| Consumer cache | Application owner | Refresh/retry without logging value |
| Audit/alert | Operations | Retrieval, policy, rotation and deletion events |
Rotation state machine
Managed rotation is available for supported services; other targets use a Lambda rotation function. The standard four steps are createSecret, setSecret, testSecret and finishSecret. Each step can be retried, so the function must validate ClientRequestToken and stages, be idempotent, use cryptographically strong values, restrict network/IAM access and never log secret material.
createSecret creates or reuses the AWSPENDING version. setSecret writes the candidate credential to the target after verifying it is modifying the expected resource and identity. testSecret authenticates and performs a minimally sufficient authorization test. finishSecret moves AWSCURRENT; Secrets Manager labels the old current version AWSPREVIOUS. A stranded AWSPENDING version is evidence of incomplete rotation, not permission to force labels blindly.
Single-user rotation changes the same target identity and can create a brief incompatibility for stale consumers. Alternating-users rotation switches between two identities to preserve overlap but needs elevated rotation credentials and synchronized permissions. Choose using target capabilities, propagation behavior, connection lifetime, privilege risk and recovery requirements. Test target backups, replicas and connection pools.
Schedules use UTC rate or cron expressions and a rotation window; supported schedules can be as frequent as four hours. Avoid overlapping database maintenance, deployments or replica lag. Monitor duration, failures, age and consumer authentication errors. Do not declare success from the rotation API alone.
Secure consumption
Applications should use workload roles and supported caching, refreshing before cache expiry and after authentication failure with bounded jitter. Never place values in environment dumps, command arguments, build output, test reports, images, artifacts or Terraform state. Redact by field and pattern, but treat redaction as backup protection rather than authorization.
A pipeline usually needs a secret only for a specific stage. Prefer short-lived federation over storing cloud keys, retrieve at runtime under a stage role, keep the value in memory, disable shell tracing, mask output, and destroy the runner. Better still, use direct service integrations or workload identity so a deployment artifact never contains a credential. Restrict retrieval by secret ARN, environment, network path and KMS context where supported.
Cross-Region replicas remain linked to a primary and need deliberate KMS, policy, failover and promotion design. Measure replication readiness; do not assume immediate consistency during disaster recovery. Deletion has a recovery window unless forced deletion is deliberately used. Alert on deletion scheduling, cancellation, policy/key change, manual value change, disabled rotation and unusual retrieval.
Inspection and workshop
~~~bash aws secretsmanager describe-secret --secret-id SECRET_ARN aws secretsmanager list-secret-version-ids --secret-id SECRET_ARN --include-deprecated aws secretsmanager get-secret-value --secret-id SECRET_ARN --version-stage AWSCURRENT aws secretsmanager get-resource-policy --secret-id SECRET_ARN aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=secretsmanager.amazonaws.com ~~~
Use only authorized test values; command history and captured output are sensitive. Given supplied rotation logs, version metadata, policies and application errors, reconstruct three rotations, identify the first failed step and design recovery without losing the last working credential. Produce single versus alternating-user decisions for a database, API token and third-party credential.
Test 22 failures: broad reader, broad key decrypt, cross-account policy alone, value in logs, shell tracing, artifact embeds value, static runner cache, rotation function outside target network, wrong target host, weak generated password, non-idempotent retry, candidate never stored, test checks login but not permission, finish before test, stranded pending stage, stale connection pool, old credential revoked too early, overlapping rotation, replica unavailable, primary promotion unplanned, deletion unnoticed, and rollback version expired.
Cost and acceptance
Price each secret, API retrievals, rotation Lambda/network/logs, KMS use and replicas. This lesson creates nothing. Submit ownership/data flow, policies, rotation state diagrams, three strategy decisions, cache/retry design, pipeline threat model, monitoring/recovery runbook and all diagnoses. Pass requires no value disclosure, least privilege, idempotent rotation and consumer-level verification.