AWS 090: Troubleshoot peering, DNS, endpoint, IPv6, and asymmetric-route failures
The real problem
A team recognizes the name Troubleshoot peering, DNS, endpoint, IPv6, and asymmetric-route failures but has not connected the feature to a real requirement, identity boundary, network or data path, failure mode, price dimension, and cleanup owner. A plausible configuration could still fail the workload.
Final outcome
The learner will produce a requirement-led artifact for Troubleshoot peering, DNS, endpoint, IPv6, and asymmetric-route failures, inspect the matching AWS control plane in the Management Console, run a matching CloudShell or AWS CLI query, interpret the output, diagnose one failure, defend one architecture choice, and prove cleanup or approved retained state.
The practical outcome is not a command transcript. It must show what was expected, what happened, what the result proves, what it does not prove, and which evidence would change the decision.
Learning objectives
By the end of this lesson, the learner can:
- explain peering is not transitive;
- explain dns has separate controls;
- explain endpoint failures cross layers;
- explain ipv6 is a separate path;
- explain asymmetry breaks state;
- connect control-plane state to the real data, network, identity, or application behavior;
- identify cost and cleanup ownership before any optional mutation;
- troubleshoot from evidence without opening broad access or adding broad permissions.
Relationship model
Requirement
|
v
Identity and policy -> AWS configuration -> network or data path -> workload behavior
| | | |
+--------------------+----------------------+--------------------+
|
v
monitoring, cost, recovery, cleanup
Use this model to separate an AWS object that exists from a result that actually works. Every arrow is a verification boundary.
Prerequisites, permissions, Region, and safety
- Learning baseline: This sequence assumes practical Linux knowledge but no prior cloud-computing or AWS knowledge. Cloud, networking, security, data, automation, and architecture concepts must come from completed earlier lessons. If a prerequisite checkpoint is incomplete, return to its linked lesson before continuing.
- Confirm a non-root caller with
aws sts get-caller-identityand keep the account number private. - Use
ap-south-1unless this lesson explicitly names a second Region. - Confirm the intended profile and Region with
aws configure listbefore interpreting an empty result. - The live track requires only the named create, describe, tag, test, and delete actions for the lab resources. If the personal-account identity lacks them, use the instructor evidence track. Do not attach AdministratorAccess as a shortcut.
- The live track can incur small charges or consume credits. Record current prices and use immediate cleanup. Never promise that a personal account is free.
- Never publish account IDs, public addresses, ARNs containing private account data, session IDs, presigned URLs, object data, credentials, or KMS material.
- Do not use root, world-open SSH or RDP, disabled TLS verification, unowned resources, or irreversible retention controls in a training exercise.
Core model
| Concept | What the learner must understand |
|---|---|
| Peering is not transitive | A route through one VPC peering connection cannot carry traffic onward through a second peering connection. Overlapping CIDRs also block or complicate valid designs. |
| DNS has separate controls | VPC DNS attributes, private hosted-zone associations, resolver rules, peering DNS options, endpoint private DNS, and client resolver configuration can fail independently of IP routing. |
| Endpoint failures cross layers | A private DNS answer can be correct while endpoint security groups, NACLs, endpoint policy, identity policy, resource policy, or KMS policy denies the operation. |
| IPv6 is a separate path | IPv6 needs its own subnet association, routes, security-group rules, NACL rules, DNS records, and egress decision. An IPv4-only correction does not repair IPv6. |
| Asymmetry breaks state | Forward and return paths through different stateful appliances or missing NACL return rules can make a valid-looking route fail. |
| Evidence order | Start with caller, Region, source and destination addresses, DNS answer, effective routes, filtering, listener, application, and return path. Change one control only after locating the failing boundary. |
How it works
Troubleshooting should move hop by hop. A successful name lookup proves only DNS, an accepted packet proves only a network boundary, and a successful API response also depends on TLS, identity, resource policy, encryption policy, and application behavior.
Read the result in layers:
- Scope: account, Region, VPC, bucket, AZ, endpoint, principal, object version, or resource ARN.
- Control plane: the requested configuration exists and reached an expected state.
- Behavior: the request, connection, health check, replication, restore, or application result meets the requirement.
- Operations: monitoring, failure owner, cost, retention, rollback, and cleanup are known.
Control-plane success is necessary but not sufficient. A resource can be available while policy, routing, DNS, health, data, or application behavior remains wrong.
Architecture decision table
| Requirement | Preferred direction | Why |
|---|---|---|
| A to B works but A to C through B fails | Reject transitive-peering assumption | Use direct peering or a transit architecture with explicit routes. |
| Service hostname returns public addresses inside the VPC | Inspect private DNS and VPC DNS prerequisites | Do not change endpoint policy until name resolution is proven. |
| IPv4 succeeds and IPv6 fails | Trace the IPv6 path independently | Check /64, ::/0, egress-only IGW or IGW, SG, NACL, and AAAA behavior. |
| Request reaches endpoint but returns AccessDenied | Inspect authorization layers | Adding a route or broad SG rule cannot repair an IAM or endpoint-policy denial. |
Professional questions normally contain several valid services. State the requirement that selects one option, why the nearest alternative fails it, and what changed requirement would reverse the choice.
AWS Management Console guided practice
Before opening a service page, write the expected account, Region, starting state, and evidence. Do not choose Create, Save, Purchase, Lock, or Delete unless the lesson explicitly authorizes the live track.
- Open VPC Peering connections, Endpoints, Route tables, Network ACLs, and Reachability Analyzer; identify the supplied fault scope before changing anything.
- Open Route 53 Resolver and VPC DNS settings, then compare the actual private-hosted-zone or endpoint-private-DNS association with the intended source VPC.
- Run one analysis for the failing address family and direction, correct one instructor-approved fault, repeat the original test, and restore the baseline.
For each step, capture the field name and value in text. A screenshot may support the record but does not replace the explanation. Console labels can evolve, so use the service search and current documentation if a navigation label differs.
CloudShell and AWS CLI practice
CloudShell is the default browser-based command environment taught in AWS 028. AWS 029 and AWS 030 cover local CLI installation and authentication. This lesson therefore does not assume that an unconfigured local shell is ready.
Start every session with:
export AWS_DEFAULT_REGION="ap-south-1"
aws sts get-caller-identity --query Arn --output text
aws configure list
Redact the account portion of the ARN before sharing. Then perform the topic query:
Capture the topology without mutation and use it to locate the first boundary that contradicts the intended flow.
aws ec2 describe-vpc-peering-connections --query 'VpcPeeringConnections[].{Id:VpcPeeringConnectionId,Status:Status.Code,Requester:RequesterVpcInfo.CidrBlock,Accepter:AccepterVpcInfo.CidrBlock}' --output table
aws ec2 describe-vpc-endpoints --query 'VpcEndpoints[].{Id:VpcEndpointId,Type:VpcEndpointType,State:State,PrivateDNS:PrivateDnsEnabled}' --output table
aws ec2 describe-route-tables --output json
Expected interpretation:
The inventory should support or disprove one hypothesis. It does not prove DNS answers, NACL evaluation, host listeners, application health, or return traffic by itself.
Replace every replace-with-... sample value before running its command, and use only an explicitly owned resource. Explain each option first. These queries are read-only; a successful response does not authorize a later create or delete operation.
Practical work
Use p05-network-break-fix.md to diagnose five supplied cases: missing return route, transitive peering assumption, wrong private-DNS scope, endpoint security-group or policy denial, and IPv6 rule omission. If AWS 089 optional resources were retained for this same session, apply no more than two faults there and perform final cleanup afterward. Otherwise use the supplied JSON and packet-path evidence. Record symptom, evidence, hypothesis, one change, retest, rollback, and cleanup.
The evidence package must contain:
- the problem and final requirement in the learner's own words;
- caller type and Region with private identifiers redacted;
- exact planned values, ownership, and cost class;
- one Console observation and matching CLI or API evidence;
- one behavior result or supplied data-plane record;
- one denied, failed, or counterexample result and evidence-led diagnosis;
- one architecture choice plus the rejected alternative;
- cleanup proof or explicit retained-state owner, expiry, and next lesson.
Verification standard
Use expected state before observed state. Record timestamps in UTC and preserve the original failure before changing anything. A passing submission answers all four questions:
- What exact requirement was tested?
- Which evidence proves the AWS configuration?
- Which evidence proves the workload behavior?
- What remains unproven or requires later monitoring?
If AWS returns no rows, verify account, Region, permission, filters, pagination, resource type, and deletion state before concluding that nothing exists.
Common failures and troubleshooting
| Symptom | Evidence first | Likely boundary | Smallest safe response |
|---|---|---|---|
| object appears missing | caller, Region, filters, pagination, tags | scope or read permission | align scope before creating a duplicate |
| state remains pending or unavailable | service state, events, dependencies, quotas | dependency or capacity | correct the named dependency and wait with a bound |
| AccessDenied | principal, action, resource, explicit-deny context | identity, resource, endpoint, organization, or KMS policy | change only the proven policy layer |
| configuration exists but behavior fails | route, DNS, security, listener, health, logs, object version | data path or application | test the next boundary and change one control |
| bill is higher than expected | hours, bytes, requests, AZs, addresses, retention | cost model or retained resource | stop optional work and reconcile the ledger |
| cleanup is blocked | dependency inventory and owning service | deletion order or immutable state | remove owned dependants in reviewed reverse order |
Do not troubleshoot by attaching administrator access, opening administration ports to the internet, disabling encryption, retrying uncontrolled creation, deleting unknown resources, or weakening retention.
Cost, cleanup, and retained state
No AWS resource is created. Close CloudShell and remove or redact downloaded evidence.
Cleanup evidence requires terminal state and an after-inventory. Search related ENIs, public IPv4 addresses, EBS volumes and snapshots, load balancers, target groups, Auto Scaling instances, endpoints, logs, S3 versions and delete markers, backup recovery points, and global IAM roles when they apply. Billing data can lag, so schedule a later review.
Architecture and certification decisions
- Certification coverage: SAA-C03; SOA-C03; SAP-C02; DOP-C02.
- Exam mapping: SAA D1-D3.
- Explain service scope, failure boundary, consistency, recovery, security, operations, and price rather than matching a keyword.
- Treat availability and durability, encryption and authorization, routing and filtering, health and lifecycle, backup and replication, and discount and capacity as separate concepts.
- Do not reproduce protected certification questions.
Knowledge check
- Can VPC peering route through a third VPC?
Expected direction: No. Peering is not transitive.
- Does a private endpoint DNS answer prove API permission?
Expected direction: No. It proves a name-to-address result only.
- Why can IPv6 fail while IPv4 works?
Expected direction: It has separate addressing, routing, DNS, and filtering controls.
- What is the safest first correction?
Expected direction: The smallest reversible change at the first proven failing boundary.
Completion gate and assessment
| Area | Points | Passing evidence |
|---|---|---|
| Requirement and model | 15 | Correct scope, terminology, and final outcome |
| Console evidence | 15 | Current path and interpreted fields |
| CLI or API evidence | 15 | Scoped command, expected result, and limitations |
| Behavior or decision exercise | 20 | Reproducible result or defensible architecture reasoning |
| Troubleshooting | 15 | Original symptom, hypothesis, one change, retest, rollback |
| Security and cost | 10 | Least privilege, data protection, current price dimensions |
| Cleanup and handoff | 10 | Terminal-state proof or approved retained-state record |
Pass at 80 out of 100 with no critical safety failure. A missing practical artifact, unexplained output, unsafe access, destructive action outside the owned scope, unplanned billed resource, or false cleanup claim requires remediation and a changed retest.