Lesson 396 · AWS Learning Path

AWS 396: Log subscriptions, Firehose delivery, OpenSearch, retention, and protected archives

Anil Jalela · Published Jul 26, 2026 · 4 min read

Why this lesson matters

Operational search, security retention and forensic custody have different requirements. A robust pipeline routes selected CloudWatch Logs through subscriptions and delivery services to searchable OpenSearch and protected S3 archives, while proving throughput, failures, transformations, encryption, replay, privacy, retention and cost.

Data path and service roles

source log groups -> group/account subscription policy
 -> destination (Firehose/Kinesis/Lambda) -> transform/redact/partition
 -> searchable OpenSearch + authoritative S3 archive + failure prefix

Account-level subscriptions simplify onboarding but require explicit selection criteria. Exclude the delivery workflow’s own log groups to prevent recursive ingestion and runaway cost. Inventory group-level and account-level filters, current quotas, filter precedence and expected coverage. CloudWatch Logs may send control messages; consumers must distinguish them.

Throughput, ordering and loss

Estimate peak compressed/uncompressed bytes and records per second, not daily average. Configure destination capacity, Firehose buffers, Lambda concurrency/timeouts, OpenSearch bulk/index capacity and S3 request rate. Buffering trades latency for efficiency. Logs can arrive late, duplicate or out of order; use source event timestamp, log-group/stream identity and event ID, and make transformation/indexing idempotent where practical.

CloudWatch Logs retries throttled destinations for a documented window and can eventually drop undeliverable data. Firehose has its own retry and S3 backup/error behavior. Monitor incoming versus delivered records/bytes, age, throttles, transform failures, destination failures, OpenSearch rejection and archive object freshness. Reconcile sampled counts and synthetic canary events end to end.

Set a loss budget and alert before retry windows expire. Keep source log retention longer than the maximum detection plus repair/replay interval so operators can recover missed delivery. During destination outage, freeze nonessential schema changes, protect buffers and source retention, prioritize regulated/security streams, and estimate catch-up capacity. A recovered destination is not complete until backlog age returns to normal and source/archive counts reconcile.

Search, archive and replay

OpenSearch is a derived search index, not the only evidence copy. Design mappings/templates before ingestion, control dynamic fields/cardinality, isolate tenants, use fine-grained access, encryption, audit logs, snapshots and index-state lifecycle. A malformed field must not poison the stream.

S3 archive design covers immutable source versus transformed copy, prefix by account/Region/service/date, compression/format, checksums, bucket owner, KMS, Object Lock/versioning when required, replication, lifecycle tiers, legal holds, inventory and deletion approval. Keep keys independently recoverable.

Replay reads a bounded manifest, validates checksum/schema, transforms with a versioned job and writes to a new index/prefix or uses idempotent IDs. It must not re-trigger production subscriptions recursively or overwrite forensic originals. Record who replayed what, when, why and resulting counts.

Define schema governance for producers and consumers: required envelope fields, version, timestamp, account/Region/service/release, classification and event ID. Test backward/forward compatibility and route malformed records to an owned quarantine with alert and expiry. Redaction occurs as close to source as practical, while an approved protected original may be retained only when legal/security policy permits and access is tightly separated.

Read-only inspection and workshop

aws logs describe-account-policies --policy-type SUBSCRIPTION_FILTER_POLICY --region ap-south-1
aws logs describe-subscription-filters --log-group-name LOG_GROUP --region ap-south-1
aws firehose describe-delivery-stream --delivery-stream-name STREAM --region ap-south-1
aws cloudwatch get-metric-data --metric-data-queries file://delivery-queries.json --start-time START --end-time END --region ap-south-1
aws s3api list-objects-v2 --bucket ARCHIVE --prefix PREFIX --region ap-south-1

Design a 50-account, three-Region pipeline for 2 TB/day average and 8x peak. Separate operational, security and regulated logs; define filters/exclusions, destination trust, buffering/capacity math, redaction, schemas, OpenSearch lifecycle, protected archives, failure prefixes, alarms, reconciliation, replay and deletion.

Test 20 failures: recursive policy, source role/policy deny, destination throttling, retry expiry, control message parsed as data, decompression mismatch, transform timeout, PII redaction missed, KMS deny, Firehose buffer latency, S3 write deny, failure prefix unmonitored, OpenSearch mapping conflict, shard pressure, index rejected, archive object absent, replication lag, replay duplicates, legal hold blocks deletion, and central pipeline Region unavailable.

Cost and acceptance

Price log ingestion/storage/subscriptions, Firehose/Kinesis/Lambda, decompression/transform, S3 requests/storage/replication/retrieval, OpenSearch nodes/serverless units/storage/snapshots, KMS, transfer and queries. This lesson creates nothing.

Submit architecture, authorization chain, throughput math, recursive exclusion, schema/redaction tests, search/archive custody, reconciliation SLO, replay runbook, 20 failures, cost and retention/deletion. Pass requires measurable loss detection, protected original evidence, bounded sensitive access, and tested replay without recursion.

Official sources

• CloudWatch Logs subscriptions
• Account-level subscription filters
• Firehose CloudWatch Logs delivery
• Firehose data delivery failures
• S3 Object Lock