# P10 CloudOps observability lab

This optional paid lab supplies the concrete environment used by AWS214–217. It creates one Amazon Linux 2023 EC2 instance with a public IPv4 address for outbound AWS API/package access, but its security group has no inbound rules. Systems Manager—not SSH—is the administration path. The stack also creates two seven-day log groups and one reviewed CloudWatch agent configuration in Parameter Store.

## Cost and authority gate

The EC2 instance, 8-GiB gp3 volume, public IPv4 address, custom metrics, log ingestion/storage, API usage and later alarms/dashboards can charge. Confirm current `ap-south-1` prices and service quotas, obtain the account owner's approval, set a same-session deletion deadline, and use only the `NitWings-P10` resources. The AWS managed policies in this isolated lab simplify first-time setup; a production design must reduce them from observed actions and organizational controls.

## Validate and deploy

```bash
set -euo pipefail
export AWS_DEFAULT_REGION="ap-south-1"
stack_name="nw-p10-observability"
template="template.yaml"

aws sts get-caller-identity --query Arn --output text
aws cloudformation validate-template --template-body "file://${template}"
aws cloudformation estimate-template-cost --template-body "file://${template}"
aws cloudformation deploy \
  --stack-name "$stack_name" \
  --template-file "$template" \
  --capabilities CAPABILITY_IAM \
  --tags Project=NitWings-P10 Cleanup=stack-owned

instance_id="$(aws cloudformation describe-stacks --stack-name "$stack_name" \
  --query 'Stacks[0].Outputs[?OutputKey==`ManagedNodeId`].OutputValue' --output text)"
test -n "$instance_id" && test "$instance_id" != "None"
```

Wait until the node is actually online in Systems Manager; EC2 `running` is insufficient.

```bash
for attempt in $(seq 1 30); do
  online="$(aws ssm describe-instance-information \
    --filters Key=InstanceIds,Values="$instance_id" \
    --query 'length(InstanceInformationList[?PingStatus==`Online`])' --output text)"
  test "$online" = "1" && break
  sleep 10
done
test "$online" = "1"
aws ssm describe-instance-information --filters Key=InstanceIds,Values="$instance_id" \
  --query 'InstanceInformationList[0].{Id:InstanceId,Ping:PingStatus,Agent:AgentVersion,Platform:PlatformName,LastPing:LastPingDateTime}' --output table
```

## Install the CloudWatch agent through Distributor

`AWS-ConfigureAWSPackage` is an AWS-owned Systems Manager document. Preserve its command ID and per-node output.

```bash
install_command="$(aws ssm send-command \
  --instance-ids "$instance_id" \
  --document-name AWS-ConfigureAWSPackage \
  --parameters 'action=Install,name=AmazonCloudWatchAgent,version=latest' \
  --comment 'NitWings P10 install CloudWatch agent' \
  --query 'Command.CommandId' --output text)"
aws ssm wait command-executed --command-id "$install_command" --instance-id "$instance_id"
aws ssm get-command-invocation --command-id "$install_command" --instance-id "$instance_id" \
  --query '{Status:Status,Code:ResponseCode,Output:StandardOutputContent,Error:StandardErrorContent}' --output json
```

## Fetch the reviewed parameter and start the agent

The instance role can read only `/nw/p10/agent-config` in addition to its AWS managed lab policies. The configuration publishes `mem_used_percent` and root `disk_used_percent` in `NitWings/P10`, plus the application and agent files into their dedicated log groups.

```bash
configure_command="$(aws ssm send-command \
  --instance-ids "$instance_id" \
  --document-name AmazonCloudWatch-ManageAgent \
  --parameters 'action=configure,mode=ec2,optionalConfigurationSource=ssm,optionalConfigurationLocation=/nw/p10/agent-config,optionalRestart=yes' \
  --comment 'NitWings P10 fetch reviewed config and start' \
  --query 'Command.CommandId' --output text)"
aws ssm wait command-executed --command-id "$configure_command" --instance-id "$instance_id"
aws ssm get-command-invocation --command-id "$configure_command" --instance-id "$instance_id" \
  --query '{Status:Status,Code:ResponseCode,Output:StandardOutputContent,Error:StandardErrorContent}' --output json
```

If that managed document's current parameter schema differs, inspect it read-only with `aws ssm get-document --name AmazonCloudWatch-ManageAgent --document-version '$DEFAULT'`, use the current Console fields, and record the difference. Do not substitute a downloaded unverified script.

## Generate harmless structured events and inspect agent state

Run Command executes as a privileged node operation. The following bounded command writes only the course-owned file and reads agent status/log tails.

```bash
evidence_command="$(aws ssm send-command \
  --instance-ids "$instance_id" \
  --document-name AWS-RunShellScript \
  --parameters 'commands=["set -euo pipefail","printf '\''{\"timestamp\":\"%s\",\"level\":\"INFO\",\"service\":\"p10-demo\",\"event\":\"agent_validation\",\"request_id\":\"p10-positive\"}\\n'\'' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" >> /var/log/nw-p10-app.log","/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status","tail -n 20 /opt/aws/amazon-cloudwatch-agent/logs/configuration-validation.log || true","tail -n 20 /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"]' \
  --comment 'NitWings P10 positive telemetry marker and agent evidence' \
  --query 'Command.CommandId' --output text)"
aws ssm wait command-executed --command-id "$evidence_command" --instance-id "$instance_id"
aws ssm get-command-invocation --command-id "$evidence_command" --instance-id "$instance_id" --output json
```

The stack also installs `/usr/local/bin/nw-p10-publish-order-events`, a reviewed helper used in AWS215–217. It appends five fake `order_validation` JSON events with current UTC timestamps, bounded fields, and no customer data. Invoke it only through the lesson's Systems Manager command. Repeated runs intentionally reuse fake request IDs, so distinguish runs by event and ingestion time.

## Prove metrics and logs

Allow at least two collection intervals. Discovery and datapoint retrieval are different checks.

```bash
sleep 130
aws cloudwatch list-metrics --namespace NitWings/P10 \
  --dimensions Name=InstanceId,Value="$instance_id" --output table

start="$(date -u -d '15 minutes ago' +%Y-%m-%dT%H:%M:%SZ)"
end="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
aws cloudwatch get-metric-statistics --namespace NitWings/P10 \
  --metric-name mem_used_percent --dimensions Name=InstanceId,Value="$instance_id" \
  --start-time "$start" --end-time "$end" --period 60 --statistics Average Maximum --output json
aws logs filter-log-events --log-group-name /nw/p10/app \
  --filter-pattern '"p10-positive"' --limit 20 --output json
```

Pass only if the command reports a running agent, recent metric datapoints have the exact instance dimension, and the structured marker appears with plausible event/ingestion timestamps. A metric name in `list-metrics` without recent datapoints is not a pass.

## Restart and changed retest

Use the same reviewed configuration so restart behavior is deterministic.

```bash
restart_command="$(aws ssm send-command \
  --instance-ids "$instance_id" --document-name AmazonCloudWatch-ManageAgent \
  --parameters 'action=configure,mode=ec2,optionalConfigurationSource=ssm,optionalConfigurationLocation=/nw/p10/agent-config,optionalRestart=yes' \
  --comment 'NitWings P10 controlled agent restart' \
  --query 'Command.CommandId' --output text)"
aws ssm wait command-executed --command-id "$restart_command" --instance-id "$instance_id"
aws ssm get-command-invocation --command-id "$restart_command" --instance-id "$instance_id" --output json
```

Record restart start/completion, the last point before it, first point after it and any gap. Repeat with a new request ID rather than reusing the previous marker.

## Cleanup and zero-residual proof

Export only redacted evidence, then delete the stack. The explicit log groups and parameter are stack-owned and will be deleted with it.

```bash
aws cloudformation delete-stack --stack-name "$stack_name"
aws cloudformation wait stack-delete-complete --stack-name "$stack_name"
if aws cloudformation describe-stacks --stack-name "$stack_name" >/dev/null 2>&1; then
  echo "FAIL: P10 stack still exists" >&2
  exit 1
fi

aws resourcegroupstaggingapi get-resources \
  --tag-filters Key=Project,Values=NitWings-P10 --output json
aws logs describe-log-groups --log-group-name-prefix /nw/p10/ --output json
if aws ssm get-parameter --name /nw/p10/agent-config >/dev/null 2>&1; then
  echo "FAIL: P10 parameter remains" >&2
  exit 1
fi
```

Also reconcile the exact instance, ENI, public IPv4, EBS volume, SG, route table, subnet, VPC, IAM role/profile, command output destinations and delayed Billing/Cost Explorer record. The tagging API does not cover every resource and is not sufficient alone. Never delete by name similarity.