AWS 399: EventBridge incident routing, enrichment, retry, archive, and replay
Why this lesson matters
EventBridge can route operational events to notifications, tickets and remediation, but delivery is asynchronous and at least once. Reliable incident automation needs a stable event contract, precise matching, safe enrichment, target authorization, retries/DLQs, idempotency, loop prevention, archives/replay and measurable end-to-end latency.
Event contract and routing
| Field/control | Purpose | Failure risk |
|---|---|---|
id, time, source, detail-type | Identity and origin | Duplicates or ambiguous ownership |
account, region, resource IDs | Scope | Cross-account/Region action on wrong target |
| detail schema/version | Domain facts | Consumer breaks on evolution |
| severity/owner/correlation | Routing and incident join | Notification flood or orphan |
| rule pattern | Select exact actionable event | Overmatch, undermatch or future-field change |
| target input | Minimum safe payload | Secret exposure or lost context |
Test patterns against positive, negative, boundary and future-compatible fixtures. Match source/detail type/account/Region plus meaningful detail fields; avoid rules that react to the remediation change they create. Add remediation-origin/action markers and desired-state predicates to prevent infinite loops. Monitor invocation spikes and budget.
Use input transformers for simple deterministic reshaping. Use Lambda/Step Functions enrichment only when needed, with bounded time, least privilege, cache/fallback and explicit failure. Do not retrieve secrets merely to place them in an event. Preserve original event ID and source while adding enrichment version and confidence.
Buses, authorization and delivery
Separate default/custom/partner buses and production/nonproduction boundaries. Cross-account bus resource policies permit PutEvents; target roles permit rule invocation of APIs. Trace producer identity, bus policy, rule, target role, downstream resource policy and KMS. Restrict organization/account, source/detail types and target resources where supported.
EventBridge retries retryable target failures according to configured/default policy and can send exhausted events to an SQS DLQ. A successful target invocation does not prove downstream business completion. Consumers must deduplicate by durable event/action key and make state transitions idempotent. Monitor matched events, invocations, failures, throttles, DLQ age/count and processing outcome.
Define an event-processing ledger for high-impact automation: event ID, schema version, incident/idempotency key, first/last received time, attempt, rule/target version, decision, action ID, result and expiry. Use conditional writes so duplicate or concurrent deliveries cannot trigger two repairs. Reconcile source event count, matched events, target invocations, consumer outcomes and DLQ. Alert on events that matched no required rule and on business outcomes that never arrive.
Archive, replay and incident safety
Archives retain matching events from one bus with selected retention and encryption behavior. Archive arrival and count can lag. Replay sends a selected time range back to the original bus and adds replay metadata; managed archive behavior avoids re-archiving replayed events, but your rules/consumers must still handle replay safely.
Never replay directly into destructive remediation without a replay-aware rule, dry-run/quarantine bus or explicit approval. Freeze source changes, calculate event count/cost, test a tiny interval, make consumers idempotent, monitor DLQ/outcome and stop on error. Replay cannot restore external state or original wall-clock assumptions.
Version event schemas and support additive evolution. Producers must not silently repurpose a field; consumers should reject unknown incompatible versions into quarantine rather than guessing. Contract tests include old/new producer and consumer combinations. Keep event size bounded and place large evidence in encrypted storage with a short-lived authorized reference, while preserving checksum and retention.
Read-only inspection and workshop
aws events list-event-buses --region ap-south-1
aws events list-rules --event-bus-name BUS --region ap-south-1
aws events describe-rule --name RULE --event-bus-name BUS --region ap-south-1
aws events list-targets-by-rule --rule RULE --event-bus-name BUS --region ap-south-1
aws events list-archives --event-source-arn BUS_ARN --region ap-south-1
aws sqs get-queue-attributes --queue-url DLQ_URL --attribute-names All --region ap-south-1Design routing for CloudWatch alarm, Config noncompliance, GuardDuty finding, deployment failure and backup failure. Normalize to one incident envelope; route severity 1 to paging plus Step Functions, severity 2 to ticket/chat, and duplicates to the same incident. Define bus/rule/target policies, correlation, DLQ triage, archive and safe replay.
Test 20 failures: pattern too broad, nested field mismatch, future event adds field, recursive remediation, duplicate delivery, out-of-order state, target role deny, target resource-policy deny, KMS deny, target timeout, retry storm, DLQ absent, DLQ permission deny, enrichment unavailable, secret in event, cross-account spoofing, archive excludes event, replay too early, replay repeats destructive action, and central bus Region outage.
Cost and acceptance
Price custom/cross-account events, Pipes/enrichment/targets, archive storage/replay, Lambda/Step Functions, SQS DLQ, logs/metrics and downstream notifications. This lesson creates nothing.
Submit schemas/fixtures, rule tests, authorization chain, loop/idempotency design, retry/DLQ runbook, archive/replay plan, latency/outcome SLO, 20 failures, cost and cleanup. Pass requires precise matching, no secret, durable deduplication, bounded automation and replay-safe consumers.