Lesson 240 · AWS Learning Path

AWS 240: KMS, certificate, secret, and encryption troubleshooting

Anil Jalela · Published Feb 20, 2026 · 12 min read

Why this lesson matters

Encryption failures often look like application failures. An object can exist while its data key cannot be decrypted. A load balancer can be healthy while browsers reject its certificate. Rotation can create a password but never promote it. Parameter Store can allow GetParameter while KMS denies Decrypt.

An architect separates four questions: what protects the data, where the control lives, who performs each operation, and which exact layer failed. This lesson builds that model without displaying a secret, private key, plaintext parameter, or ciphertext.

What you will be able to do

By the end, you can:

• explain encryption at rest, TLS, envelope encryption, and secret rotation;
• distinguish KMS key IDs, ARNs, aliases, policies, grants, state, and context;
• diagnose ACM validation, Region, attachment, renewal, hostname, chain, and TLS failures;
• interpret secret versions, staging labels, schedules, and all four rotation steps;
• choose Parameter Store or Secrets Manager from requirements;
• inspect safe metadata without retrieving protected values;
• produce a root cause, least-privilege correction, rollback, retest, and prevention control;
• identify billable components and prove this exercise created nothing.

Before you start: strict safety boundary

• Use an approved non-root identity and confirm account and Region before every query.
• This is read-only. Use the supplied cases if no owned resources exist.
• Never run secretsmanager get-secret-value, ssm get-parameter --with-decryption, ssm get-parameter-history, acm export-certificate, or kms decrypt.
• Never share values, private keys, passphrases, ciphertext, validation emails, account IDs, or full unique ARNs.
• Encryption context appears in CloudTrail; it must contain non-secret data only.
• Do not disable/delete keys, move staging labels, edit DNS, or request certificates as experiments.

Four different protection jobs

Encryption does not decide who may call an API, validate input, or stop an authorized process leaking plaintext. Identity, network, logging, classification, and application controls still apply.

How these services evolved

The history explains why overlapping choices exist:

• 2014 - AWS KMS: AWS introduced a managed, HSM-backed key control plane integrated with services such as EBS, S3, and Redshift, with key-use auditing through CloudTrail. This reduced the need for every team to operate its own key infrastructure.
• 2016 - ACM: AWS introduced managed public certificate provisioning and renewal for integrated services such as Elastic Load Balancing and CloudFront. The original model deliberately kept ACM-issued public private keys non-exportable.
• 2016 era - Parameter Store: Systems Manager added hierarchical configuration and SecureString. It remains a configuration service first; encrypted storage does not turn it into a full credential-rotation service.
• 2018 - Secrets Manager and ACM Private CA: Secrets Manager added purpose-built secret versioning and rotation. Private CA addressed private certificate hierarchies and internal trust.
• 2021 - KMS multi-Region keys: related keys could share key material across Regions for selected client-side encryption, DR, and signing use cases, while remaining separate Regional resources.
• 2025 - ACM exportable public certificates: newly requested certificates could be marked exportable for EC2, containers, hybrid, and multicloud workloads.
• 2026 - shorter public-certificate lifetime: new and renewed ACM public certificates moved to a maximum 198-day lifetime and managed renewal moved to 45 days before expiry. Existing longer-lived certificates remain valid until renewal or expiry.

The architectural lesson is not “new replaces old.” Each feature solves a different ownership problem. Always inspect the certificate/key creation date and current metadata because legacy resources can behave differently from newly created ones.

KMS from first principles

Most service integrations use envelope encryption:

data --encrypted locally with data key--> encrypted data
data key --encrypted with KMS key-------> encrypted data key
stored together: encrypted data + encrypted data key

KMS normally protects the small data key rather than receiving the large object. A KMS denial can therefore block intact data.

KeySpec describes material and algorithms; KeyUsage permits a class of operations. A signing key cannot decrypt.

Identity, Region, lifecycle, and rotation

• A key ID is the UUID-like ID. A key ARN includes partition, Region, account, and ID. An alias such as alias/orders-prod is a friendly pointer.
• Aliases do not transform old ciphertext when redirected.
• KMS keys are Regional. A multi-Region primary and replicas share key material and key ID, but remain separate resources with separate policy, alias, grant, state, and billing.
• AWS managed keys are service-managed. Customer managed keys add policy/lifecycle control, responsibility, and cost.
• Rotation creates new key material for future operations. It does not rewrite old ciphertext; retained old material permits authorized decryption.

All keys have state. Common states are Enabled, Disabled, and PendingDeletion; imported-material/custom-store/multi-Region designs can show PendingImport, Expired, Unavailable, or PendingReplicaDeletion. Record KeyState, Enabled, Origin, ExpirationModel, ValidTo, MultiRegion, KeySpec, and KeyUsage together.

Do not schedule deletion to fix an incident. After deletion, data protected only by that key can be unrecoverable.

KMS authorization is more than IAM

The key policy is primary. IAM permission works only if the key policy permits the principal directly or enables the owning account to delegate through IAM. Explicit deny wins.

Evaluate:

1. Caller, account, partition, Region, and exact key.
2. Key state/spec/usage.
3. SCP/RCP and permissions boundary.
4. VPC endpoint policy.
5. Key policy.
6. Caller IAM policy.
7. Service-created KMS grant.
8. Conditions such as kms:ViaService, encryption context, tags, or source ARN.

Cross-account use normally needs a key-policy allow in the owner account and an IAM allow for the caller. Services often use narrow grants. New grants can be eventually consistent; a grant token can make a just-created grant effective immediately.

Encryption context is non-secret key-value data cryptographically bound to symmetric encryption operations. Decrypt must supply the required matching context. Policy and grant constraints can require pairs. Wrong key/context, altered ciphertext, wrong Region, or an asymmetric algorithm mismatch can cause InvalidCiphertextException.

Correlate CloudTrail time, caller, key suffix, event/error, source service, and request parameters. Redact identifiers.

ACM and TLS from first principles

A certificate binds domain names to a public key. During TLS, the endpoint presents certificate and chain, proves private-key possession, negotiates protocol/cipher settings, and establishes an encrypted session.

• Public ACM-issued: publicly trusted; commonly attached to integrated AWS services.
• Exportable public: usable on EC2, containers, or elsewhere; private-key protection and renewed-certificate deployment are your responsibility and billable.
• Private: issued from AWS Private CA; clients must trust that private CA.
• Imported: you provide certificate, private key, and chain; you own renewal.

Region and attachment

ACM certificates are Regional. ALB/NLB and Regional API Gateway certificates must be in the resource's Region. A CloudFront viewer certificate must be in us-east-1.

A standard non-exportable certificate is not installed directly on EC2. Use an integrated load balancer, exportable certificate, ACME, imported certificate, or approved alternative.

Validation and status

For PENDING_VALIDATION, inspect every DomainValidationOptions item: SAN, status, exact CNAME name/value, authoritative public answer, duplicated zone suffix, underscore handling, CNAME chain, and CAA authorization.

Keep DNS validation CNAMEs after issuance for managed renewal. If validation does not complete in 72 hours, ACM changes status to VALIDATION_TIMED_OUT. Other important statuses include ISSUED, INACTIVE, EXPIRED, REVOKED, and FAILED. Obtain status from describe-certificate, not a summary assumption.

For renewal, inspect RenewalEligibility, RenewalSummary, NotAfter, InUseBy, AWS Health, and EventBridge. Issuance/renewal does not prove an exportable or imported certificate was deployed.

As of the 2026 review date, newly issued and renewed ACM public certificates have a maximum 198-day validity period and renew 45 days before expiry. Do not hardcode those figures into automation: derive deadlines from NotAfter and current AWS guidance.

Diagnose in layers:

DNS -> TCP -> TLS handshake -> trust chain -> hostname/SAN -> validity
    -> listener/SNI certificate -> protocol/cipher policy -> HTTP

Common faults are wrong Region, no attachment, wrong SNI selection, missing SAN, incomplete imported chain, expiry/revocation, client clock, or incompatible listener policy.

Secrets Manager and rotation

A secret has encrypted versions plus metadata. Staging labels identify roles:

• AWSCURRENT: returned by default;
• AWSPREVIOUS: prior version;
• AWSPENDING: candidate during rotation.

Use describe-secret and list-secret-version-ids; do not retrieve values. Replicas remain linked to a primary, so verify application Region and replica status.

Some services use managed rotation; others use Lambda. Lambda rotation is a four-step, idempotent transaction for one token:

1. createSecret: create/reuse the AWSPENDING candidate.
2. setSecret: apply candidate credentials to the target.
3. testSecret: prove the candidate authenticates and has intended access.
4. finishSecret: promote candidate to AWSCURRENT; old current becomes previous.

Schedules use UTC rate/cron expressions and a rotation window. A stale pending label proves rotation started, not the failure. Inspect Lambda logs by token/step, concurrency, expected JSON schema, KMS, Secrets Manager endpoint, DNS/routes/security groups, and target authentication.

Do not move labels blindly. If setSecret succeeded but finishSecret failed, the target and labels may disagree. A healthy rotation also does not force a long-running app to refresh cached credentials; design cache TTL, retry, and connection renewal.

Parameter Store and SecureString

Parameter types are String, comma-separated StringList, and KMS-encrypted SecureString. Prefer Secrets Manager for credentials requiring rotation and replica lifecycle; use Parameter Store for hierarchical configuration and simpler encrypted settings.

Intelligent-Tiering chooses a tier from features/defaults. Standard can upgrade to advanced, but advanced cannot downgrade in place. Parameter Store retains 100 recent versions. Labels point to versions. Advanced policies support expiration, expiration notification, and no-change notification - not credential rotation.

Standard SecureString encrypts directly with a symmetric KMS key. Advanced uses envelope encryption via AWS Encryption SDK and a unique data key. Creation needs kms:Encrypt for standard or kms:GenerateDataKey for advanced; decryption needs kms:Decrypt plus SSM permission.

Metadata includes name, type, tier, version, labels, data type, and key ID; the value is sensitive. GetParameterHistory includes the current value, so it can bypass an otherwise incomplete read-deny design. Recursive path permission can expose descendants; design path boundaries carefully.

DataType=aws:ec2:image validates AMI IDs asynchronously. AWS public parameters publish useful service data. Neither feature is secret rotation.

Safe Console and CLI investigation

In the Console, record scope first. Inspect KMS configuration/policy/grants; ACM status/validation/attachments/renewal; secret KMS/rotation/version-label/replica metadata; and parameter name/type/tier/version/key/policies. Never choose Retrieve secret value, export, edit, rotate, disable, or delete.

export AWS_DEFAULT_REGION="ap-south-1"
aws sts get-caller-identity --query Arn --output text
aws configure list

aws kms describe-key --key-id alias/replace-owned +  --query 'KeyMetadata.{Arn:Arn,State:KeyState,Enabled:Enabled,Spec:KeySpec,Usage:KeyUsage,Origin:Origin,MultiRegion:MultiRegion,Expiration:ExpirationModel,ValidTo:ValidTo}'
aws kms get-key-policy --key-id alias/replace-owned --policy-name default
aws kms list-grants --key-id alias/replace-owned +  --query 'Grants[].{Name:Name,Grantee:GranteePrincipal,Operations:Operations}'

aws acm list-certificates +  --query 'CertificateSummaryList[].{Domain:DomainName,Arn:CertificateArn}'
aws acm describe-certificate --certificate-arn replace-owned-arn +  --query 'Certificate.{Status:Status,Type:Type,Domains:SubjectAlternativeNames,InUseBy:InUseBy,NotAfter:NotAfter,RenewalEligibility:RenewalEligibility,RenewalSummary:RenewalSummary,Validation:DomainValidationOptions}'

aws secretsmanager describe-secret --secret-id replace-owned-full-arn +  --query '{Arn:ARN,KmsKeyId:KmsKeyId,RotationEnabled:RotationEnabled,RotationRules:RotationRules,LastRotatedDate:LastRotatedDate,NextRotationDate:NextRotationDate,ReplicationStatus:ReplicationStatus,VersionStages:VersionIdsToStages}'
aws secretsmanager list-secret-version-ids --secret-id replace-owned-full-arn +  --query 'Versions[].{VersionId:VersionId,Stages:VersionStages,Created:CreatedDate}'

aws ssm describe-parameters +  --parameter-filters 'Key=Name,Option=BeginsWith,Values=/replace/owned/path/' +  --query 'Parameters[].{Name:Name,Type:Type,Tier:Tier,Version:Version,KeyId:KeyId,DataType:DataType,Modified:LastModifiedDate}'

Use --region us-east-1 for the CloudFront ACM check. These prove metadata, not workload decrypt or endpoint behavior.

One troubleshooting algorithm

1. Capture exact symptom, UTC time, request ID, and redacted error.
2. Identify the workload caller/client, not the Console operator.
3. Confirm account, partition, Region, ARN/alias target, and endpoint.
4. Check resource state/type before permissions.
5. Trace every authorization plane and condition.
6. Check integration, grant, DNS/network, and attachment.
7. Check version, rotation, deployment, and application cache.
8. Change one controlled variable under approval.
9. Retest the failed layer and complete user path.
10. Add an event/alarm, owner, SLO, policy test, or runbook.

Practical work

• Troubleshooting workbook
• Supplied failure cases
• Complete practical pack

For each case, identify layer, safe evidence, correction, rollback, retest, and prevention. Mark unknown facts instead of inventing them.

Cost and cleanup

• Customer managed KMS keys and requests are billable; each multi-Region key resource is billed. AWS managed keys have no monthly key-storage charge.
• Non-exportable public ACM certificates for integrated services have no ACM certificate charge. Exportable public certificates, private certificates, and Private CA are billable. Connected services remain separate.
• Secrets Manager charges per secret and API calls; replicas are billable. Rotation Lambda, logs, networking, and target services can add cost.
• Standard Parameter Store has no additional charge within documented limits. Advanced parameters and higher-throughput interactions are billed; KMS requests can apply.
• This lesson changes nothing. Cleanup proof is matching before/after inventory and no change record.

Check current official pricing for the selected Region/date; do not use course figures in production estimates.

Knowledge check

1. Why can IAM allow decrypt while KMS denies it?
2. Explain envelope encryption.
3. Give four non-IAM causes of invalid ciphertext.
4. Why is a multi-Region replica still a separate resource?
5. Where must a CloudFront viewer certificate live?
6. How does validation differ from listener/TLS failure?
7. Explain current, previous, and pending secret labels.
8. What does each rotation step prove?
9. Why can apps fail after successful rotation?
10. When is Parameter Store the better choice?
11. How do standard and advanced SecureString encryption differ?
12. Why is parameter history sensitive?

Lesson acceptance

A passing submission contains all six cases with root cause and safe evidence; KMS state/policy/context reasoning; certificate Region/validation/attachment/renewal reasoning; rotation timeline; service choice; dated cost evidence; and before/after no-change proof. It contains no protected values, private keys, ciphertext, account IDs, or full unique ARNs, and confirms production was untouched.

Official sources

• KMS concepts
• AWS KMS 2014 launch
• ACM 2016 launch
• Secrets Manager 2018 launch
• KMS multi-Region keys launch
• ACM exportable public certificates launch
• ACM 2026 validity update
• KMS key states
• KMS key policy
• KMS encryption context
• ACM certificate options
• ACM DNS validation
• ACM renewal status
• ACM exportable certificates
• Secret versions
• Rotation troubleshooting
• Parameter Store
• SecureString encryption
• KMS pricing
• ACM pricing
• Secrets Manager pricing
• Systems Manager pricing