AWS 413: CloudTrail protection, centralized audit evidence, and log integrity
Why this lesson matters
CloudTrail records supported account activity, but Event history alone is not a protected organization audit archive. Architects must select required event types, deliver them across accounts/Regions, restrict and retain S3/KMS evidence, monitor delivery, preserve identity context and cryptographically validate log files for investigations.
Event and trail model
Management events describe control-plane operations. Data events describe high-volume resource operations such as S3 object access or Lambda invocation and are not logged by trails by default. Network activity events and Insights events also require deliberate configuration. Select data events with advanced selectors based on threat/audit need and cost; exclude recursive delivery behavior where applicable.
A multi-Region organization trail managed from a security/log-archive design provides a broad baseline, including new member accounts when configured correctly. Verify every Region, organization membership, trail status and selectors. Keep management activity out of the Organizations management account where possible and protect the delegated/log archive accounts with separate identities.
| Evidence plane | Proves | Does not prove alone |
|---|---|---|
| Event history | Recent management-event lookup | Durable complete archive |
| Trail status/selectors | Intended logging and delivery state | Every required event arrived |
| S3 log objects | Delivered event records | Unchanged/deletion-free without controls |
| Digest validation | Delivered files/digest chain unchanged | Event truth or semantic completeness |
| CloudWatch/EventBridge alert | Timely selected detection | Long-term custody |
| S3 inventory/reconciliation | Object/coverage accounting | Cryptographic integrity |
CloudTrail records are not an ordered stack trace and delivery can be delayed. Use eventTime, event ID, request ID, source IP/VPC endpoint context, userIdentity/session issuer/source identity, recipient account, Region, resources, request and response/error fields carefully. Redacted or absent request fields do not prove a parameter was unused.
Protected central custody
Deliver to a dedicated S3 bucket in a log archive account. Bucket policy permits CloudTrail ACL check/write with exact source/account/organization conditions supported by the design and denies insecure transport. Restrict human read, separate administrators from evidence readers, enable versioning and suitable Object Lock/retention if required, and monitor policy, lifecycle, retention and access changes.
Use SSE-KMS when customer key control is required. Key policy must allow CloudTrail encryption and tightly scoped analyst decryption; disabling/deleting the key can make evidence unavailable. Test restoration and cross-Region disaster access. Lifecycle transitions must retain logs and digest files together through legal/audit requirements.
Optional CloudWatch Logs integration supports timely metric filters/alarms but is not the immutable archive. Query/index services may improve analysis, yet the protected trail/S3 evidence and export strategy remain independent custody. Define schemas and parsers that preserve original objects and hashes.
Integrity validation
When log file validation is enabled, CloudTrail hashes delivered logs and produces signed digest files approximately hourly. Digests reference log hashes and chain to the previous digest; signatures use Regional CloudTrail keys. Enabling validation creates digests but does not itself execute validation.
Use the AWS CLI validation command from the original delivery location for the investigation time/Region, preserve output and investigate gaps. Disabling validation or stopping logging breaks the chain. Delivery errors can temporarily make a chain appear incomplete and later redelivery/backfill can arrive out of order. Check trail status and wait/reconcile before declaring tampering, while preserving the anomaly.
Validation proves that referenced delivered files were not altered or deleted after delivery. It cannot prove selectors captured every required event, that an unsupported action was logged, or that credentials map to the human you assume. Coverage and identity governance are separate controls.
~~~bash aws cloudtrail describe-trails --include-shadow-trails aws cloudtrail get-trail-status --name ORG_TRAIL aws cloudtrail get-event-selectors --trail-name ORG_TRAIL aws cloudtrail validate-logs --trail-arn TRAIL_ARN --start-time START --end-time END aws s3api get-bucket-policy-status --bucket AUDIT_BUCKET aws kms get-key-policy --key-id KEY_ARN --policy-name default ~~~
Operations and workshop
Reconcile enabled accounts/Regions, expected trail status, selector policy, latest log and digest delivery, S3 inventory counts and error alarms. Alert on StopLogging/DeleteTrail/PutEventSelectors, validation disable, bucket/key/policy/lifecycle changes, organization trail changes, delivery errors and unusual evidence reads. Keep an emergency collection runbook that does not modify originals.
Given supplied organization/trail/selector data and two days of logs/digests, identify missing coverage, validate chains, distinguish delivery delay from modification, reconstruct three cross-account sessions and produce a custody report. Design selectors for sensitive S3, Lambda, KMS and secrets activity with volume estimates.
Test 24 failures: single-Region trail, new Region missed, member account absent, read events excluded, data events assumed default, selector too broad, recursive logging cost, trail stopped, validation disabled, digest not actually validated, wrong Region key, moved logs validated as original, delayed digest called tampering, bucket policy blocks delivery, bucket admin deletes evidence, lifecycle deletes digest first, KMS key disabled, analyst overprivileged, session issuer ignored, source identity absent, clock/order assumption, Event history treated archive, alert path is only custody, and restored archive never tested.
Cost and acceptance
Price additional management copies, data/network/Insights events, S3 versions/retention/retrieval, KMS, CloudWatch delivery/ingestion, queries and transfer. This lesson creates nothing. Submit event requirements, organization architecture, selector/cost model, S3/KMS policies, monitoring/reconciliation, validation/custody report and all diagnoses. Pass requires verified coverage, protected recoverable originals and successful digest-chain validation with stated limits.