Lesson 241 · AWS Learning Path

AWS 241: DNS, route, NAT, endpoint, peering, and load-balancer troubleshooting

Anil Jalela · Published Feb 21, 2026 · 13 min read

Labelled process diagram for AWS 241: Source and DNS answer to Forward route and filters to Destination or load balancer to Return path and application evidence, with decision, proof and rejection evidence.

Why this lesson matters

“The network is down” is not a diagnosis. An application first resolves a name, chooses IPv4 or IPv6, selects a route, crosses filters and optional translation or inspection, reaches a listener or host, and needs a valid return path. A green resource status proves only one component.

This lesson teaches a repeatable packet walk. You will locate the first failed decision, preserve evidence, and avoid dangerous shortcuts such as adding 0.0.0.0/0, opening every port, or replacing DNS before identifying the actual layer.

Outcomes

By the end, you can:

distinguish DNS resolution from IP reachability and application health;
calculate the effective subnet route using longest-prefix match;
explain internet gateways, public/private NAT gateways, and IPv6 egress-only gateways;
compare gateway, interface, Gateway Load Balancer, resource, and service-network endpoints;
troubleshoot VPC peering and Transit Gateway route domains in both directions;
explain stateful security groups and stateless, ordered NACLs;
trace ALB DNS, scheme, listener rules, target groups, health checks, and return traffic;
use safe metadata, logs, metrics, and Linux tests to isolate the first failed layer;
document correction, rollback, retest, prevention, and cost ownership.

Safety boundary

Use an approved non-root identity. Confirm account and Region before querying.
This is a read-only, no-create lesson. Use supplied cases when no owned network exists.
Never change routes, DNS, endpoint policy, security controls, listeners, or target registration during diagnosis.
Do not run packet capture against unapproved traffic, port-scan networks, or expose full IP plans/account IDs.
Redact public IPs, full ARNs, resource IDs, domain names, headers, and application payloads.
DNS and access logs may contain customer identifiers. Keep evidence within its approved retention boundary.

A short history: why several connectivity models exist

2009 - Amazon VPC: isolated AWS networks connected to enterprise infrastructure became a first-class service.
2011 - ELB in VPC: load balancers gained VPC subnet and security-group placement while targets could remain private.
2014 - VPC peering: one-to-one private routing connected non-overlapping VPCs without a transit device.
2015 - gateway endpoints: S3 and DynamoDB could be reached privately through route-table prefix-list entries, without NAT.
2017 - AWS PrivateLink: interface endpoints placed service ENIs and private IPs in customer subnets.
2018 - Transit Gateway: hub-and-spoke routing addressed the operational growth of many point-to-point peerings.
2024 onward - resource and service-network endpoints: PrivateLink/VPC Lattice options expanded private access beyond the original NLB endpoint-service model.

Newer does not automatically mean better. Peering can be simple for a few VPCs; Transit Gateway centralizes many networks; PrivateLink publishes a service without creating general network-to-network reachability. Choose from traffic and ownership requirements.

The packet-walk model

For one failing connection, write this tuple:

source IP:port -> destination name -> resolved IP:port -> protocol
source subnet/AZ -> route target -> filters/NAT/inspection -> destination
destination response -> reverse route/filters -> source

The source port is usually ephemeral; the destination port is the service port. The response swaps them. Record IPv4 and IPv6 separately because they can follow different routes and controls.

Troubleshoot in order:

Application selected the expected hostname, protocol, and port.
The source resolver returned the expected records.
The source subnet selected the expected route.
Every forward-path filter and appliance allowed the flow.
NAT or endpoint behavior matched the design.
The destination listener/process accepted the request.
The target/application returned a valid response.
Every reverse-path route and stateless filter allowed the reply.

ping tests ICMP only. A host can reject ICMP while HTTPS works. traceroute is also not a complete VPC topology map because managed components may not answer TTL-expired probes.

DNS: an answer is data, not connectivity

DNS maps a name to records and caches them by TTL. Important records include:

A for IPv4 and AAAA for IPv6;
CNAME for one name aliasing another;
Route 53 alias records for supported AWS resources, including zone-apex use;
NS delegation and SOA zone metadata;
routing policies such as simple, weighted, latency, failover, geolocation, geoproximity, and multivalue.

Test from the same resolver context as the workload:

dig +short A app.example.invalid
dig +short AAAA app.example.invalid
dig +trace app.example.invalid
getent ahosts app.example.invalid

dig +trace follows public delegation; it does not reproduce a VPC private answer. Compare authoritative records, resolver answer, TTL/cache, and application-resolved IP.

VPC Resolver and private hosted zones

Private hosted zones answer only in associated VPCs or connected hybrid resolver paths. Both enableDnsSupport and enableDnsHostnames matter. With split-view DNS, the same name can return private data inside and public data outside.

When private hosted-zone namespaces overlap, VPC Resolver uses the most specific matching zone. It does not fall back to the public zone merely because the matching private zone lacks a requested record. Resolver rules can take precedence for the same namespace. Check:

VPC-to-zone association, including cross-account authorization/association;
outbound/inbound Resolver endpoints and rule association;
conditional-forwarding domain and target DNS server;
Resolver DNS Firewall action and priority;
negative caching after NXDOMAIN;
client stub cache, JVM/process cache, and container DNS configuration.

An ALB/NLB DNS name returns changing addresses. Publish an alias/CNAME as supported; do not copy a current load-balancer IP into an application.

Routing: destination decides the next hop

Every subnet is associated with one route table, explicitly or through the VPC main table. Each route has a destination and target. The router applies longest-prefix match: the most specific destination wins. Therefore 10.20.4.0/24 beats 10.20.0.0/16, which beats 0.0.0.0/0.

The VPC local route provides routing among VPC subnets, subject to filters. Other targets include internet gateway, NAT gateway, egress-only internet gateway, peering connection, Transit Gateway, virtual/private gateway, endpoint, ENI, or appliance.

Route state matters:

active: target is available to routing;
blackhole: target/attachment is unavailable or route deliberately drops traffic.

A public subnet is not defined by a name or tag. For IPv4 internet access it normally has a route to an attached internet gateway, and the instance needs a public IPv4/EIP or translation design. An internet gateway performs one-to-one public IPv4 translation for addressed instances; it does not make every subnet resource public.

Always build forward and return tables. Route tables are not stateful. An appliance that receives the outbound packet but not the reply can drop the session even when both one-way routes look valid.

Security groups and network ACLs

Control	Attachment	Rules	Evaluation	Return traffic
Security group	ENI/resource	Allow only	All applicable rules	Automatically allowed for tracked flow
Network ACL	Subnet	Allow and deny	Lowest numbered matching rule first	Must be explicitly allowed

Security groups are stateful, but do not confuse that with application listening. The target SG should usually permit the application/health-check port from the load balancer SG rather than the whole VPC.

NACLs are stateless. For a client-to-server TCP flow, allow the destination service port in the forward direction and the relevant ephemeral return ports in the reverse direction on every involved subnet NACL. Exact ephemeral ranges depend on client operating system and architecture; document the range used rather than blindly copying one range.

Also check:

source/destination check when an EC2 appliance routes traffic;
host firewall (nftables/iptables), SELinux policy, and process bind address;
overlapping CIDRs and unintended more-specific routes;
IPv6 SG/NACL entries, which are separate from IPv4;
managed prefix-list versions and entries.

NAT: translation for initiated flows, not inbound publishing

A public NAT gateway resides in a public subnet, uses an Elastic IP, and needs that subnet's route to an internet gateway. Private workload subnets route IPv4 internet destinations to the NAT gateway. It permits return traffic for connections initiated through it; unsolicited internet connections do not use it to publish a server.

A private NAT gateway translates private addresses for private connectivity through Transit Gateway or a virtual private gateway; sending its traffic to an internet gateway does not provide internet access. IPv6 generally uses native addressing and, when outbound-only internet behavior is required, an egress-only internet gateway - not NAT44 reasoning.

NAT investigation order:

Gateway state and state message.
Workload route to NAT.
NAT subnet route to internet gateway for public NAT.
SG on source and NACLs on both subnets.
Destination reachability/protocol.
NAT metrics and connection behavior.

Useful metrics include ConnectionAttemptCount, ConnectionEstablishedCount, ErrorPortAllocation, IdleTimeoutCount, PacketsDropCount, and packet/byte direction counters. Connections idle for 350 seconds can be closed; clients reusing stale sockets then fail. Each NAT IPv4 address supports a finite number of simultaneous connections per unique destination; ErrorPortAllocation > 0 points to source-port exhaustion.

A NAT gateway cannot be used as a transit next hop through VPC peering for clients in another VPC. Do not test NAT by pinging the gateway itself; it does not respond as a general host.

VPC endpoints: private path plus authorization

“Endpoint” is a family:

Type	Data-plane shape	Typical purpose
Gateway	Prefix-list route-table target	S3 and DynamoDB
Interface	PrivateLink ENIs with private IPs and SGs	Supported AWS/endpoint services
Gateway Load Balancer	Endpoint steers traffic to virtual appliances	Transparent inspection
Resource	PrivateLink access to a specific shared resource	Database/IP/domain-style resource
Service network	PrivateLink/VPC Lattice access to a service network	Multiple governed services/resources

For a gateway endpoint, inspect endpoint state, associated route tables, service prefix-list route, endpoint policy, and bucket/table policy. Traffic from an unassociated subnet may still use NAT/public service endpoints.

For an interface endpoint, inspect:

endpoint service name and Region;
subnet ENIs and available IPs;
endpoint SG inbound from clients on service port;
private DNS enabled and VPC DNS attributes;
what the workload actually resolves;
endpoint policy and service resource/IAM policies;
endpoint connection acceptance for custom services.

Private DNS lets the normal public service hostname resolve to endpoint private IPs inside the VPC. It does not make that name privately resolvable from every peered or on-premises network automatically. Hybrid DNS needs Resolver architecture.

An endpoint removes a NAT/internet dependency for that service; it does not bypass identity/resource policies. Endpoint policy is an additional authorization boundary, not a replacement.

Peering and Transit Gateway

VPC peering is one-to-one and requires non-overlapping CIDRs. It is not transitive: if A peers with B and A peers with C, B cannot route through A to C. Both subnet route tables need peer-CIDR routes, and SG/NACL controls must permit the real source. Peering DNS-resolution options affect whether public hostnames of peer resources resolve to private addresses; you cannot query the peer VPC's Amazon DNS server directly.

Check peering lifecycle state: pending acceptance, active, rejected, failed, expired, or deleted. Cross-account owners must coordinate both routes and security.

Transit Gateway adds a second routing decision:

source subnet route -> TGW attachment
incoming attachment association -> TGW route table lookup
TGW route -> destination attachment
destination VPC attachment subnet -> destination subnet route/filters

Association controls which TGW route table an attachment uses for traffic entering the TGW. Propagation adds learned routes to selected TGW tables. A route can be static, propagated, or blackhole. Inspect all four VPC-side/TGW-side decisions for forward and return traffic.

Stateful inspection requires symmetry. Transit Gateway appliance mode keeps a flow on the same appliance-zone path; without it, return traffic can reach a different appliance instance that has no connection state.

Load balancer troubleshooting

First identify type and layer:

ALB: HTTP/HTTPS/gRPC, host/path/header/query rules, redirects/fixed responses.
NLB: TCP/TLS/UDP and static IP-style requirements.
GWLB: transparent appliance fleets using GENEVE.
Classic Load Balancer: legacy L4/L7 feature set.

For an ALB trace:

DNS resolves to the intended load balancer.
Scheme is internet-facing or internal as intended.
Enabled subnets/AZs and load-balancer state are correct.
LB SG/NACL permits client listener traffic and target/return traffic.
Listener exists; HTTPS certificate/security policy is correct.
Rules are evaluated by priority; conditions select expected action/target group.
Target type, protocol, port, IP family, and registered AZ are valid.
Target SG permits LB SG on traffic and health-check ports.
Health-check protocol, port, path, host handling, matcher, interval, timeout, and thresholds match the app.
Application listens on the target IP/port and returns the expected result.

Target states include initial, healthy, unhealthy, unused, draining, and unavailable. Read reason codes. Target.Timeout suggests path/listener/load; Target.ResponseCodeMismatch means the app responded outside the matcher; Target.NotInUse points to target group/listener/AZ use.

ALB health checks send a Host header based on the target private IP and health-check port. Virtual-host-only applications may reject it unless a default host handles the check. Never “fix” a health check by accepting every error code; create a lightweight endpoint that proves dependencies appropriate to the service's readiness definition.

Distinguish:

HTTPCode_ELB_4XX/5XX: generated by the load balancer;
HTTPCode_Target_4XX/5XX: returned by targets;
ALB access log elb_status_code versus target_status_code;
502: malformed/reset/backend TLS/connection behavior;
503: no ready targets or unavailable action;
504: connect/response timeout.

Cross-zone behavior, deregistration delay, slow start, stickiness, idle timeout, and HTTP keep-alive affect symptoms but should be changed only after evidence.

Safe evidence collection

export AWS_DEFAULT_REGION="ap-south-1"
aws sts get-caller-identity --query Arn --output text

aws ec2 describe-vpc-attribute --vpc-id vpc-REDACTED --attribute enableDnsSupport
aws ec2 describe-vpc-attribute --vpc-id vpc-REDACTED --attribute enableDnsHostnames
aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=subnet-REDACTED
aws ec2 describe-security-groups --group-ids sg-REDACTED
aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=subnet-REDACTED
aws ec2 describe-nat-gateways --nat-gateway-ids nat-REDACTED
aws ec2 describe-vpc-endpoints --vpc-endpoint-ids vpce-REDACTED
aws ec2 describe-vpc-peering-connections --vpc-peering-connection-ids pcx-REDACTED

aws elbv2 describe-load-balancers --names replace-owned-name
aws elbv2 describe-listeners --load-balancer-arn replace-owned-arn
aws elbv2 describe-rules --listener-arn replace-owned-listener-arn
aws elbv2 describe-target-groups --target-group-arns replace-owned-tg-arn
aws elbv2 describe-target-health --target-group-arn replace-owned-tg-arn

For Transit Gateway, inspect attachments, route-table associations/propagations, and search exact destination routes. For Route 53, list only the owned hosted zone and redact names. A large unfiltered inventory is weaker evidence and leaks topology.

Linux tests should name the layer:

getent ahosts app.example.invalid
curl -v --connect-timeout 5 https://app.example.invalid/health
openssl s_client -connect app.example.invalid:443 -servername app.example.invalid </dev/null
ip route get 203.0.113.10
ss -lntp

Never put credentials in command arguments or URLs. Capture only approved headers/body.

Practical investigation

Download:

For each case, draw forward and return paths, record every decision, mark the first failed layer, propose the smallest correction, define rollback and retest, and add prevention. Unknown evidence must remain unknown.

Cost, resilience, and cleanup

Public IPv4 addresses, NAT gateway hours/data processing, and cross-AZ data paths can be material. One NAT per active AZ improves AZ independence and can avoid cross-AZ hairpinning.
Gateway endpoints for S3/DynamoDB have no endpoint hourly/data-processing charge; interface, GWLB, resource, and service-network endpoint pricing differs by type and volume.
Transit Gateway charges attachments/data processing according to current pricing and path. Peering/data transfer pricing depends on Region and direction.
ALB/NLB/GWLB charge running time/capacity units; logs add S3/CloudWatch delivery, storage, and query cost.
Route 53 hosted zones, queries, Resolver endpoints, DNS Firewall, and health checks can be billable.
This lesson creates nothing. Prove the resource inventory and change-event count are unchanged.

Do not memorize prices. Record Region, date, path direction, bytes, hours, endpoint/AZ count, and current official calculator/pricing URL.

Knowledge check

Why can DNS succeed while HTTPS fails?
Which route wins among /24, /16, and /0 matches?
Why does an IPv4 public NAT need two different route tables?
Why can NACL return traffic fail when the inbound service port is allowed?
Compare gateway and interface endpoints.
Why does endpoint policy allow not prove service authorization?
Why can VPC B not reach VPC C through peer VPC A?
Explain TGW association versus propagation.
Why does stateful inspection need symmetric routing/appliance mode?
What does Target.ResponseCodeMismatch prove?
Why should an ALB DNS name not be replaced by a copied IP?
Which evidence separates ELB-generated 5xx from target-generated 5xx?

Lesson acceptance

A passing submission contains six complete packet walks; exact DNS context; effective longest-prefix route; SG and both NACL directions; NAT/endpoint/peering/TGW decisions where applicable; listener-rule-target-health evidence; first failed layer; least-risk correction, rollback, retest, and prevention; dated cost evidence; and before/after no-change proof. It exposes no protected topology/customer data and confirms production was untouched.

Official sources

View complete AWS learning path