AWS 076: Status checks and automatic recovery
The problem
A team uses Status checks and automatic recovery as a label or Console setting without proving the identity, scope, behavior, failure boundary, cost, or operational result. The configuration appears complete but the real requirement remains untested.
Final outcome
The learner will inspect, explain, test, and document Status checks and automatic recovery. The submission must connect the Console and CLI view to the same AWS control, state what the evidence proves, diagnose one likely failure, and make a requirement-based architecture decision.
Learning outcomes
You will be able to:
- explain ec2 runs three status-check categories;
- explain checks run every minute;
- explain system failure points below the guest;
- explain instance failure points inside the guest;
- explain automatic recovery preserves identity;
- distinguish successful configuration from successful workload behavior;
- preserve redacted evidence and complete the stated cleanup or retention decision.
Mental model
requirement
-> identity and permission
-> account, Region, and resource scope
-> configuration or request
-> observable state
-> workload result
-> failure evidence
-> cleanup or controlled retention
Never begin with a create button or a copied command. Start with the result that must be proven and the boundary that must remain protected.
Core facts
| Concept | What it means in practice |
|---|---|
| EC2 runs three status-check categories | System status checks cover AWS host infrastructure, instance status checks cover guest reachability and configuration, and attached EBS status checks cover EBS reachability and I/O on supported Nitro instances. |
| Checks run every minute | Each returns pass or fail and publishes CloudWatch status metrics. Status checks cannot be disabled or deleted. |
| System failure points below the guest | Power, host hardware, or physical-network problems can fail the system check. Stop/start, EC2 recovery, or AWS remediation can move or restore the instance. |
| Instance failure points inside the guest | Bad network or startup configuration, exhausted memory, corrupted file system, or kernel trouble can fail the instance check and normally needs customer action. |
| Automatic recovery preserves identity | Supported recovery after system-check failure retains the instance ID, AZ, IP addresses, metadata, placement group, and EBS volumes, but RAM and operating-system uptime are lost. |
| Status checks are not application checks | An instance can pass EC2 checks while its process, dependency, TLS certificate, or business function is broken. Use load-balancer health checks, synthetic tests, logs, and application metrics. |
How to reason about it
Recovery keeps one instance identity after eligible host failure; replacement creates a fresh instance from known configuration. For stateless services, Multi-AZ Auto Scaling replacement usually provides stronger application resilience. Use recovery when preserving a particular instance identity is a documented requirement, and still protect data independently.
Use the following decision table as a starting point, then change the answer when the scenario changes.
| Requirement | Preferred direction | Why |
|---|---|---|
| System status check fails on a supported singleton | Automatic recovery or stop/start | The likely problem is the underlying host. |
| Instance status check fails | Guest troubleshooting, reboot, or replacement | The operating system or its configuration normally needs customer action. |
| Attached EBS status check remains impaired | Investigate volume and host path; fail over or replace | The check identifies EBS reachability or I/O impact on supported Nitro instances. |
| Web process is unhealthy but EC2 checks pass | Load-balancer health replacement and application diagnostics | Infrastructure reachability does not prove the service works. |
Prerequisites, permissions, Region, and cost
Run this lesson as the approved non-root course identity. Begin with aws sts get-caller-identity, confirm the private account record, and set the fixed project Region before any regional query. IAM resources are global within an account, while STS endpoints and the services reached by an identity can be regional.
Use only the read or change actions required for this lesson. An AccessDenied result is evidence to analyse, not permission to switch to root or attach AdministratorAccess. Record the action, resource, principal type, request context, and smallest justified correction.
The listed cost tier is T0 - no resource creation. Free Tier eligibility and credits are account-specific. Before a mutating lab, identify every resource that can charge, estimate its duration, start a timer, and write the reverse cleanup order. A budget reports cost after billing data arrives and is not a real-time stop control.
AWS Management Console method
- Open EC2 Instances, select an instance, then Status checks and Monitoring.
- Compare System reachability, Instance reachability, and attached-EBS status with application and load-balancer health.
- Open CloudWatch Alarms and inspect the StatusCheckFailed metrics and recovery or notification action without changing production alarms.
For every step record the service page, selected account and Region, exact object, visible state, and why that state matters. A Console label or green status is not enough unless it is tied to the final workload outcome.
AWS CLI or API evidence
Read all status categories and instance events before choosing recovery.
aws ec2 describe-instance-status --instance-ids i-0123456789abcdef0 --include-all-instances --query 'InstanceStatuses[0].{State:InstanceState.Name,System:SystemStatus.Status,Instance:InstanceStatus.Status,EBS:AttachedEbsStatus.Status,Events:Events}'
Impaired System suggests underlying infrastructure. Impaired Instance suggests guest trouble. A missing EBS field can reflect instance support or no reported impairment; confirm metrics and documentation.
Before running the command, replace every placeholder, explain each option, and decide whether the operation is read-only or mutating. Capture the exit code immediately. Redact account IDs, ARNs, public addresses, request identifiers, and personal data before sharing.
The CLI and Console are clients of AWS APIs. Matching state across them increases confidence, but neither substitutes for data-plane or application verification.
Practical work
Start nw-p04-web-1 for a maximum 15-minute evidence window. Inspect system, instance, and attached-EBS status checks plus application, SSM Agent, and Apache health. Create p04-health-map.md that maps each failed signal to owner, likely layer, evidence, stop/start or recovery implications, CloudWatch alarm metric, and safe response. Inspect automatic recovery options without enabling a paid or unreviewed action, then stop the instance unless AWS 077 starts immediately.
The evidence package must include:
- non-root principal type, account verified privately, and Region;
- exact intended and observed state;
- one Console observation and the matching CLI or API result;
- one successful result and one denied, failed, or counterexample result;
- what each result does not prove;
- cost state and elapsed lab time;
- cleanup evidence or a named retained-state owner and next lesson.
Verification standard
Use three levels of proof:
- Control plane: the object or policy exists with the intended configuration.
- Data plane or behavior: the request, packet, session, storage path, or application does what the requirement states.
- Operations: monitoring, failure diagnosis, cost, ownership, and cleanup are known.
A control-plane response can precede final readiness. Use waiters or state polling where supported, then test the actual behavior. If a request times out, do not assume it failed. Inspect state and use documented idempotency before retrying a mutation.
Troubleshooting method
| Symptom | Evidence first | Smallest safe response |
|---|---|---|
| command cannot authenticate | credential source, expiry, caller preflight | restore approved temporary login |
| access is denied | action, resource, principal, all policy layers | correct only the missing or conflicting control |
| object appears missing | account, Region, filters, pagination, permission | align scope before creating anything |
| configured state exists but behavior fails | route, identity, dependency, logs, service state | test the next boundary in the path |
| cleanup is blocked | dependency inventory and owning service | remove dependants in reviewed reverse order |
Keep the original symptom and timestamp. State one hypothesis, make one reversible change, repeat the original test, and record rollback. Never open a management port to the world, expose credentials, disable TLS verification, format an unknown disk, or add broad permissions as a generic fix.
Architecture and certification decisions
Professional-level questions provide competing valid features. Identify the requirement that decides between them: human or workload identity, same-account or cross-account access, regional or zonal scope, stateful or stateless filtering, durable or ephemeral data, latency, RTO/RPO, cost, or operational ownership.
Explain why the selected option fits and why each plausible alternative fails one stated requirement. Do not rely on feature memorization or reproduce protected certification questions.
Knowledge check
- Which check normally indicates host infrastructure trouble?
Expected direction: System status check.
- Which check detects guest network or kernel problems?
Expected direction: Instance status check.
- Does automatic recovery handle an unhealthy web process?
Expected direction: No. It responds to eligible system-status failure, not ordinary application health.
- What does recovery lose even when identity is preserved?
Expected direction: Volatile RAM and operating-system uptime; applicable instance-store considerations also matter.
Cost, cleanup, and retained state
Retain only the state explicitly required by the next P04 lesson and record it in the private resource ledger.
Cleanup evidence includes the final state query, not only a successful delete response. Search related resources, other Regions used by the lab, retained storage, public IPv4 addresses, logging destinations, and service-managed dependencies. Schedule a later billing review because cost data can lag.
Completion gate
Pass when the practical artifact explains the real problem, matches Console and CLI evidence, answers every knowledge check, diagnoses one failure without broadening access, records cost, and proves cleanup or approved retention. The learner must defend one decision orally and name the requirement that would change it.