Lesson 278 · AWS Learning Path

AWS 278: Architecture: produce a multi-account hybrid network and failure-domain decision record

Anil Jalela · Published Mar 30, 2026 · 13 min read

Why this lesson matters

An architecture diagram records components, but not why they were chosen, which assumptions can invalidate them, what failures they tolerate, or who accepted the consequences. An architecture decision record (ADR) turns requirements and evidence into a reviewable decision. At minimum it captures context, the decision, and consequences; a production network ADR also needs options, route/DNS proofs, security, resilience, operations, cost, migration, tests, and rollback.

This capstone combines AWS267 through AWS277. You will produce a decision for a fictional multi-account enterprise with two data centers, branches, two AWS Regions, Direct Connect, VPN, TGW or Cloud WAN, hybrid DNS, centralized inspection, shared services, and an acquired overlapping network. The lesson creates no AWS resource.

Outcomes

By the end, you can:

• turn business language into measurable network requirements and constraints;
• separate facts, assumptions, risks, decisions, and unresolved questions;
• compare at least three viable architectures using weighted criteria;
• create account, Region, CIDR, ASN, route-domain, DNS, and ownership models;
• prove forward and return paths for approved, denied, degraded, and recovery flows;
• map every dependency to an AZ, Region, site, device, provider, and control-plane failure domain;
• define RTO/RPO, capacity, quotas, security, telemetry, and cost evidence;
• design migration waves, acceptance gates, rollback, decommissioning, and ADR review triggers; and
• produce a professional p15-hybrid-network-adr.md ready for architecture review.

What an ADR is and is not

An ADR is a concise, durable record of a consequential choice. It should let a future operator answer: what problem existed, what evidence was known, what options were considered, what was selected, why, what tradeoffs were accepted, and when the decision must be revisited.

An ADR is not:

• a product inventory with no requirement mapping;
• a diagram whose arrows omit return traffic;
• a promise of “high availability” without failure tests;
• a runbook, although it links to runbooks;
• a cost estimate with no traffic/capacity assumptions;
• a security approval, although it records required approvals; or
• immutable. It can be superseded by a new ADR while preserving history.

Use states such as proposed, accepted, rejected, superseded, and deprecated. Give the ADR an owner, decision authority, date, version, reviewers, links to evidence, and scheduled/retrigger review.

Fictional enterprise brief

Northwind Works has:

• 42 AWS accounts under Organizations: production, nonproduction, security, network, shared services, logging, and sandbox OUs;
• 30 VPCs in ap-south-1 and 12 warm-standby VPCs in ap-southeast-1;
• Mumbai and Bengaluru data centers plus 25 branches;
• two diverse 10-Gbps Direct Connect links at separate locations/providers;
• internet VPN backup and an existing SD-WAN estate;
• production, development, partner, shared, management, and quarantine trust domains;
• private DNS under corp.example and aws.corp.example;
• public web/API ingress, private east-west/hybrid traffic, package/API egress, and AWS service endpoints;
• one acquired network overlapping 10.20.0.0/16;
• a 15-minute critical-network RTO and no claim that network RTO equals application RTO;
• residency controls that permit approved data/log replication only in the two named Regions; and
• forecast aggregate hybrid demand of 12 Gbps normal, 18 Gbps peak, plus recovery growth.

Business constraints: production and development must not communicate directly; shared DNS, patching, identity, and observability are selectively reachable; internet egress must be attributed; a single circuit, router, firewall endpoint, AZ, or DX location must not stop critical connectivity; and no untested global policy may be applied.

Convert statements into testable requirements

Write requirements before options. Mark each as mandatory, target, or preference. Capture measurement method, owner, and date. “Fast,” “secure,” and “resilient” are not requirements until quantified.

Facts, assumptions, constraints, and unknowns

Keep four separate lists:

An assumption is a risk until tested. Add an invalidation condition and consequence: “If measured VPN backup capacity is below critical load, the 15-minute RTO is not met and a second private path/capacity reduction is required.”

Options to compare

Option A: Regional TGWs with DXGW and inter-Region peering

Network account owns one TGW per Region. Transit VIFs through a DX gateway provide hybrid access; VPN terminates on TGWs as backup. TGW route tables separate trust domains. Static TGW peering routes connect approved Regional summaries.

Strengths: explicit routing, familiar operations, controlled Regional blast radius, straightforward fit for 42 VPCs. Costs/risks: static inter-Region routes, automation needed for consistency, separate Regional governance.

Option B: Cloud WAN global core

Segments represent production, development, shared, hybrid, and quarantine. Attachment policies classify VPCs; network function groups insert Regional inspection; routing policies govern global routes. Existing TGW route tables can support staged coexistence.

Strengths: global policy, managed edges/mesh, consistent segmentation, route policy at larger scale. Costs/risks: global policy blast radius, new skills, core/attachment/data charges, migration complexity.

Option C: service-first isolation plus limited routed hubs

Keep overlapping/acquired and lower-trust networks in separate route domains. Publish shared APIs through PrivateLink/Lattice/proxies and exchange asynchronous data. Use TGW only for approved nonoverlapping estates and hybrid management.

Strengths: least network exposure, overlap tolerance, application identity. Costs/risks: service-by-service integration, unsupported legacy protocols, dual operations during migration.

The selected decision may combine A and C now, with Cloud WAN reconsidered at a stated threshold. A real ADR is allowed to reject a sophisticated service when present scale does not justify it.

Weighted decision matrix

Define weights before scores to reduce outcome bias:

Use a documented 1-5 scoring rubric and multiply by weights. Do not assign cost points until completing traffic and standing-resource estimates. Mandatory failures disqualify an option regardless of weighted total.

Selected reference decision

For the fictional brief, select Option A plus targeted Option C:

• TGW per Region in the network account, shared by RAM;
• production, development, shared-services, hybrid, inspection-return, and quarantine TGW tables;
• two transit VIFs through one DXGW using diverse DX links/locations, plus BGP VPN backup;
• static inter-Region TGW peering routes only for approved nonoverlapping summaries;
• Network Firewall inspection VPC per Region with appliance mode and per-AZ endpoints;
• zonal NAT gateways for controlled IPv4 egress, separate IPv6 controls, VPC endpoints for AWS services;
• Regional Resolver inbound/outbound endpoints, Profiles/rules/query logging;
• PrivateLink/proxy for acquired-overlap services, with a 12-month renumber program; and
• Route 53/Global Accelerator choice made by each application ADR, not hidden in the network ADR.

Cloud WAN is reconsidered when active Regions exceed four, branch/global route operations exceed agreed toil thresholds, or static peer/policy inconsistency breaches SLOs. This is a review trigger, not a promise to migrate.

Account and ownership model

Use Organizations, SCPs, IAM boundaries, RAM shares, CloudTrail, Config, and infrastructure pipelines. Separate policy author, reviewer, deployer, and emergency approver where practical.

Address, route-domain, and route proof

The ADR includes:

• IPAM pools by Region/environment, IPv4/IPv6, pod/service/transition space;
• unique VPC and hybrid CIDRs plus acquisition overlap registry;
• TGW ASNs, customer ASNs, BGP communities and advertisements;
• attachment-to-associated-table and propagation matrix;
• exact static, summary, prefix-list, default, and blackhole routes; and
• no unowned or unbounded 0.0.0.0/0/::/0 paths.

For each flow, prove source subnet route, source attachment/table, matching TGW route, inspection subnets/endpoints, second TGW table, destination route/security/app, then every reverse lookup.

Hybrid path and failure domains

Direct Connect redundancy is end-to-end, not merely two logical connections. Record customer routers, ports, cross-connects, provider circuits, DX devices/locations, VIFs, BGP sessions, DXGW, TGW attachments, power, software, and change teams. Maximum-resiliency patterns use separate connections at separate locations; validate current service SLA prerequisites rather than claiming one from a diagram.

VPN over internet is encrypted but performance is best effort. When DX and VPN advertise equal prefixes to TGW, documented route priority normally favors DX. More-specific routes, static routes, customer local preference, AS path, and advertisements can override the intended outcome. Test failure from both AWS and on-premises directions.

Backup capacity must carry the critical recovery traffic, not necessarily all normal traffic. Define traffic shedding and priority. Test MTU/MSS, BGP convergence, session reset, DNS, and application retry.

Create a failure-domain table:

DNS, ingress, egress, and inspection

Corporate DNS conditionally forwards AWS suffixes to inbound endpoint IPs in two AZs. VPC Resolver rules send corporate suffixes through outbound endpoints. Each Region can resolve locally during inter-Region isolation. The ADR lists zone authority, rules, Profiles/associations, security groups, query logs, DNSSEC/DNS Firewall, TTL, failure, and loop prevention.

Public ingress uses application-specific CloudFront/WAF, Global Accelerator, Route 53, ALB/NLB, or API Gateway decisions. Network inspection does not replace WAF/application authorization. Egress distinguishes internet IPv4/NAT, IPv6, AWS service endpoints, private hybrid, DNS, and management.

Stateful inspection uses per-AZ paths and TGW appliance mode. The route matrix proves no direct more-specific route bypasses it. TLS decryption, privacy, certificate trust, performance, and exclusions require a separate approved decision.

Capacity, quotas, observability, and operations

Build normal, peak, degraded, recovery, and growth capacity. Include DX port/provider capacity, VIF/BGP prefixes, VPN tunnels/ECMP, TGW attachments/routes/throughput, firewall/NAT flows and bytes, Resolver QPS, load balancers/endpoints, cross-AZ/Region paths, log ingestion, and application dependencies. Record default/current quotas and increase lead time.

Telemetry layers:

• DX/VIF/BGP and VPN tunnel metrics;
• TGW/VPC Flow Logs with no-route/blackhole evidence;
• Network Firewall flow/alert/TLS logs and endpoint metrics;
• NAT connection/error/byte metrics;
• Resolver query logs and endpoint volume;
• Route 53/Global Accelerator/load-balancer health;
• CloudTrail/Config for changes;
• synthetic positive, negative, degraded, and recovery probes; and
• CUR/Cost Explorer allocation and anomaly alerts.

Define 24x7 owner, severity, alarm threshold, runbook, dashboard, evidence retention, privacy, and escalation for every shared component.

Cost model

Estimate one-time installation/migration and monthly standing/usage costs:

• DX ports, provider circuits, cross-connect/colo, routers and support;
• VPN connections, acceleration/private IP options;
• TGW attachments, peering and processed bytes;
• Network Firewall/GWLB/appliances, NAT, endpoints, load balancers;
• Resolver endpoints/queries, DNS Firewall, Route 53/Global Accelerator;
• inter-AZ, inter-Region, internet and DX transfer;
• logs, metrics, archives, analytics and security operations; and
• warm-standby capacity, game days, licenses and engineering.

For each of the top 20 flows, count bytes at every billable boundary in both directions. Model normal, peak, DX failure, Region recovery, and data-resynchronization months. State currency, Region, pricing date/link, traffic assumptions, tax/support exclusions, confidence range, and cost center.

Migration, rollback, and decommission

Use waves:

1. governance, IPAM, accounts, logs and pipeline;
2. Regional TGWs/route domains with no production propagation;
3. Resolver and shared services canary;
4. inspection and controlled egress canary;
5. VPN bootstrap and low-risk hybrid routes;
6. DX transit VIF/DXGW paths in parallel;
7. production VPC and branch waves;
8. second Region and tested recovery;
9. acquisition service exposure then renumbering; and
10. legacy route/VPN/proxy decommission.

Each wave has baseline, owner, change set, canary, positive/negative tests, metric gates, abort threshold, rollback command/IaC version, session/cache expectations, and post-change observation. Do not delete the old path until zero-use evidence, dependency owner sign-off, backup/rollback expiry, billing verification, and security-log retention pass.

ADR template and required evidence

Create p15-hybrid-network-adr.md with:

1. ID, title, status, owner, date, reviewers and decision authority.
2. Executive summary and decision statement.
3. Business context, scope, exclusions and drivers.
4. Requirements with IDs and acceptance tests.
5. Facts, assumptions, constraints, unknowns and invalidation triggers.
6. Current state and pain evidence.
7. Options, weighted matrix, cost and rejected alternatives.
8. Selected architecture and diagrams.
9. account/RACI, addressing/IPAM, route domains and 20-flow proof.
10. Hybrid/DNS/ingress/egress/inspection/security details.
11. Failure-domain register, RTO/RPO, capacity, quotas and game days.
12. Monitoring, operations, incident and change model.
13. Cost assumptions and ownership.
14. Migration waves, rollback and decommission.
15. Positive, negative, degraded, recovery and failback tests.
16. Consequences: benefits, drawbacks, risks and technical debt.
17. Open decisions, approval signatures and review triggers.
18. Evidence appendix with dated source links and redacted inventories.

Include at least three diagrams: trust/account topology, one packet path with return, and failure-domain/dependency map. Every arrow must have direction, protocol, route owner, inspection, and evidence.

Read-only evidence commands

export AWS_DEFAULT_REGION="ap-south-1"
aws sts get-caller-identity --query Arn --output text
aws ec2 describe-transit-gateways --output json
aws ec2 describe-transit-gateway-vpc-attachments --output json
aws directconnect describe-connections --output json
aws directconnect describe-virtual-interfaces --output json
aws ec2 describe-vpn-connections --output json
aws route53resolver list-resolver-endpoints --output json
aws route53resolver list-resolver-rules --output json
aws network-firewall list-firewalls --output json

Also inspect TGW tables/associations/propagations/routes, VPC/subnet routes, DXGW associations/proposals, BGP/VPN telemetry, IPAM, RAM, Resolver Profiles/logs, firewall/NAT/endpoints, quotas, CloudTrail/Config, and CUR. Commands prove inventory only; behavioral tests and owner evidence are separate. Redact sensitive topology and identities.

ADR review and rejection tests

Reject the ADR if:

• a mandatory requirement has no acceptance evidence;
• only one option was seriously considered;
• route arrows omit return lookup or route-table ownership;
• overlap is hidden behind a broad static route;
• DNS depends on one Region or loops;
• “redundant DX” shares location/provider/router/power without disclosure;
• VPN backup is not capacity/MTU/routing tested;
• inspection has bypass/asymmetry or undefined fail mode;
• IPv6/AWS service/management traffic is omitted;
• cost ignores per-GB and provider/colo charges;
• migration has no canary/rollback/decommission; or
• assumptions, risks, owners, and review triggers are missing.

Approval means stakeholders accept documented consequences, not that every risk disappeared.

Knowledge check

1. What are the minimum ADR elements?

Context, decision, and consequences, expanded here with evidence and lifecycle controls.

1. Why are two DX links not automatically resilient?

They may share routers, provider, location, fiber, power, or change failure domains.

1. Does network RTO prove application RTO?

No. Data and application dependencies have separate readiness.

1. Why include rejected options?

Future reviewers need to understand tradeoffs and changed assumptions.

1. What makes an architecture claim testable?

A measurable requirement, predicted behavior, evidence source, owner, and acceptance threshold.

1. When should this ADR be superseded?

When review triggers invalidate assumptions or a materially different decision is accepted.

Lesson acceptance

The lesson is complete only when the learner submits an ADR that:

• contains all 18 required sections and three labelled diagrams;
• maps every requirement to evidence and ownership;
• compares three viable options with a preweighted rubric and full cost model;
• proves 20 bidirectional allowed, denied, degraded, and recovery flows;
• documents CIDR/ASN/IPAM, routing, BGP, DNS, inspection, ingress/egress and overlap;
• maps end-to-end failure domains and capacity-tested backup paths;
• defines telemetry, quotas, operations, security, privacy and RACI;
• supplies migration, canary, rollback, decommission, game-day and failback evidence;
• records consequences, debt, unknowns, assumptions and review triggers;
• passes the rejection tests and named stakeholder review; and
• attests that no staging or production AWS resource changed.

Official sources

• AWS architectural decision records guidance
• ADR frequently asked questions
• AWS Hybrid Connectivity
• Hybrid connectivity design selection
• Hybrid connectivity reliability
• Hybrid connectivity security
• Hybrid connectivity cost