AWS 078: VPC connectivity troubleshooting
The problem
A team uses VPC connectivity troubleshooting as a label or Console setting without proving the identity, scope, behavior, failure boundary, cost, or operational result. The configuration appears complete but the real requirement remains untested.
Final outcome
The learner will inspect, explain, test, and document VPC connectivity troubleshooting. The submission must connect the Console and CLI view to the same AWS control, state what the evidence proves, diagnose one likely failure, and make a requirement-based architecture decision.
Learning outcomes
You will be able to:
- explain troubleshoot one flow at a time;
- explain dns is evidence, not routing;
- explain routes must work both ways;
- explain every security layer must agree;
- explain reachability analyzer models configuration;
- distinguish successful configuration from successful workload behavior;
- preserve redacted evidence and complete the stated cleanup or retention decision.
Mental model
requirement
-> identity and permission
-> account, Region, and resource scope
-> configuration or request
-> observable state
-> workload result
-> failure evidence
-> cleanup or controlled retention
Never begin with a create button or a copied command. Start with the result that must be proven and the boundary that must remain protected.
Core facts
| Concept | What it means in practice |
|---|---|
| Troubleshoot one flow at a time | Write source address and ENI, destination name and address, protocol, port, time, expected return path, and the last known working point. |
| DNS is evidence, not routing | Confirm the answer from the actual source. A private, public, stale, or IPv6 answer can send the packet down a different path than expected. |
| Routes must work both ways | Use the source subnet table, longest-prefix selection, target state, middlebox routes, and destination return route. Stateful services do not repair missing network routes. |
| Every security layer must agree | Check security groups, NACLs, Network Firewall or appliances, host firewall, endpoint policy, identity policy, and application listener according to the path. |
| Reachability Analyzer models configuration | It performs static analysis of supported network configuration and explains a blocking component or reachable path. It does not send packets or prove application health. |
| Logs answer different questions | Flow Logs show metadata and accept or reject; CloudTrail shows control-plane changes; load balancer and application logs show protocol and business outcomes. |
How to reason about it
Use a fixed order to avoid random changes: identity and DNS, source addressing, forward route, middleboxes, destination rules and listener, return route, and logs. Change one variable, record the result, and restore temporary test rules. Opening 0.0.0.0/0 everywhere can hide the cause and create an incident.
Use the following decision table as a starting point, then change the answer when the scenario changes.
| Requirement | Preferred direction | Why |
|---|---|---|
| Need static explanation of a VPC path | Reachability Analyzer | It identifies supported configuration components without running a packet. |
| Need to know whether SG or NACL rejected traffic | VPC Flow Logs | REJECT metadata narrows the interface, direction, address, and port. |
| Connectivity broke after a change | CloudTrail and AWS Config timeline | Control-plane evidence identifies who changed routes, groups, ACLs, or gateways. |
| Network accepts flow but HTTP fails | Listener, TLS, target-health, and application logs | The problem is now above the network-authorization layer. |
Prerequisites, permissions, Region, and cost
Run this lesson as the approved non-root course identity. Begin with aws sts get-caller-identity, confirm the private account record, and set the fixed project Region before any regional query. IAM resources are global within an account, while STS endpoints and the services reached by an identity can be regional.
Use only the read or change actions required for this lesson. An AccessDenied result is evidence to analyse, not permission to switch to root or attach AdministratorAccess. Record the action, resource, principal type, request context, and smallest justified correction.
The listed cost tier is T0 - no resource creation. Free Tier eligibility and credits are account-specific. Before a mutating lab, identify every resource that can charge, estimate its duration, start a timer, and write the reverse cleanup order. A budget reports cost after billing data arrives and is not a real-time stop control.
AWS Management Console method
- Open VPC Reachability Analyzer and create a path between supported source and destination resources.
- Set the protocol and destination port, start analysis, and inspect the hop-by-hop explanation or blocking component.
- Compare the static result with Flow Logs, CloudTrail changes, target health, and application logs before changing rules.
For every step record the service page, selected account and Region, exact object, visible state, and why that state matters. A Console label or green status is not enough unless it is tied to the final workload outcome.
AWS CLI or API evidence
Create a Reachability Analyzer path between supported source and destination resources, then start analysis.
aws ec2 create-network-insights-path --source eni-SOURCE --destination eni-DESTINATION --protocol tcp --destination-port 443
aws ec2 start-network-insights-analysis --network-insights-path-id nip-0123456789abcdef0
The analysis reports reachable or identifies a blocking component in modeled configuration. Test the actual application separately because no packet was sent.
Before running the command, replace every placeholder, explain each option, and decide whether the operation is read-only or mutating. Capture the exit code immediately. Redact account IDs, ARNs, public addresses, request identifiers, and personal data before sharing.
The CLI and Console are clients of AWS APIs. Matching state across them increases confidence, but neither substitutes for data-plane or application verification.
Practical work
Create p04-connectivity-runbook.md. Trace source identity and address, DNS, ENI, source SG/NACL, route selection, gateway, destination SG/NACL, listener, host firewall, application, and return path. Apply the runbook to stored evidence for Session Manager unavailable, outbound package failure, and local Apache success with external failure. If a live retest is needed, start nw-p04-web-1 only for a timed evidence window and stop it afterward. Use Reachability Analyzer and Flow Logs as evidence sources, not automatic proof of application success.
The evidence package must include:
- non-root principal type, account verified privately, and Region;
- exact intended and observed state;
- one Console observation and the matching CLI or API result;
- one successful result and one denied, failed, or counterexample result;
- what each result does not prove;
- cost state and elapsed lab time;
- cleanup evidence or a named retained-state owner and next lesson.
Verification standard
Use three levels of proof:
- Control plane: the object or policy exists with the intended configuration.
- Data plane or behavior: the request, packet, session, storage path, or application does what the requirement states.
- Operations: monitoring, failure diagnosis, cost, ownership, and cleanup are known.
A control-plane response can precede final readiness. Use waiters or state polling where supported, then test the actual behavior. If a request times out, do not assume it failed. Inspect state and use documented idempotency before retrying a mutation.
Troubleshooting method
| Symptom | Evidence first | Smallest safe response |
|---|---|---|
| command cannot authenticate | credential source, expiry, caller preflight | restore approved temporary login |
| access is denied | action, resource, principal, all policy layers | correct only the missing or conflicting control |
| object appears missing | account, Region, filters, pagination, permission | align scope before creating anything |
| configured state exists but behavior fails | route, identity, dependency, logs, service state | test the next boundary in the path |
| cleanup is blocked | dependency inventory and owning service | remove dependants in reviewed reverse order |
Keep the original symptom and timestamp. State one hypothesis, make one reversible change, repeat the original test, and record rollback. Never open a management port to the world, expose credentials, disable TLS verification, format an unknown disk, or add broad permissions as a generic fix.
Architecture and certification decisions
Professional-level questions provide competing valid features. Identify the requirement that decides between them: human or workload identity, same-account or cross-account access, regional or zonal scope, stateful or stateless filtering, durable or ephemeral data, latency, RTO/RPO, cost, or operational ownership.
Explain why the selected option fits and why each plausible alternative fails one stated requirement. Do not rely on feature memorization or reproduce protected certification questions.
Knowledge check
- Does a successful Reachability Analyzer result prove HTTPS works?
Expected direction: No. It proves a supported network path in configuration, not listener, TLS, or application health.
- Why record the exact DNS answer?
Expected direction: The address family and destination determine which route and controls apply.
- What does Flow Logs ACCEPT prove?
Expected direction: The captured network controls accepted the flow, not that the application responded successfully.
- What should be checked after the forward path?
Expected direction: The complete return route and reverse stateless controls.
Cost, cleanup, and retained state
Retain only the state explicitly required by the next P04 lesson and record it in the private resource ledger.
Cleanup evidence includes the final state query, not only a successful delete response. Search related resources, other Regions used by the lab, retained storage, public IPv4 addresses, logging destinations, and service-managed dependencies. Schedule a later billing review because cost data can lag.
Completion gate
Pass when the practical artifact explains the real problem, matches Console and CLI evidence, answers every knowledge check, diagnoses one failure without broadening access, records cost, and proves cleanup or approved retention. The learner must defend one decision orally and name the requirement that would change it.