AWS 397: OpenTelemetry traces, X-Ray service maps, sampling, and trace correlation
Why this lesson matters
Distributed traces explain one request’s path across services and dependencies when context is propagated correctly. OpenTelemetry provides vendor-neutral APIs, SDKs, semantic conventions, collectors and OTLP. AWS Distro for OpenTelemetry integrates that model with AWS. X-Ray-compatible analysis and service maps are destinations/views, not substitutes for sound instrumentation.
Trace data model
| Element | Meaning | Design rule |
|---|---|---|
| Trace | End-to-end transaction identified by trace ID | Preserve context across trust boundaries safely |
| Span | Timed operation with parent/link relationships | Name by low-cardinality operation, not URL/user |
| Resource | Service/environment/runtime identity | Include service name/version and deployment env |
| Attribute | Searchable span metadata | Follow semantic conventions; prohibit secrets/PII |
| Event/status | Important occurrence and outcome | Record bounded exception class, not raw payload |
| Link | Causal relation outside strict parent tree | Useful for batch, queue, replay and fan-out |
| Baggage | Propagated key/value context | Strict allowlist because it crosses services |
Instrument inbound/outbound HTTP/RPC, database/cache calls, queue publish/consume, important internal work and user/business operation. Avoid a span per loop item or payload. Propagate W3C traceparent/tracestate or the selected interoperable format; validate incoming context and start a safe new trace when malformed/untrusted.
Async messaging needs producer, broker and consumer semantics. A consumer may use a parent or link depending on processing model. Preserve message/event ID, retry attempt and idempotency identity, but never make a replay appear to have happened at original time. Batch processing may link multiple producer contexts rather than inventing one parent.
Collector and sampling architecture
Applications export OTLP to an in-process/sidecar/daemon/gateway collector path. Collectors receive, batch, memory-limit, enrich/filter, sample and export. Design queues, retry, backpressure, endpoint TLS/auth, health metrics, configuration version and failure behavior. Telemetry must not crash or block the application indefinitely.
Head sampling decides near trace start and bounds overhead but can miss rare failures. Tail sampling observes completed traces and can retain errors/latency/selected traffic, but needs buffering, memory, consistent routing and a decision point. Parent-based sampling maintains trace consistency. Always-on in production can be too costly; too little sampling makes service maps and error analysis misleading.
Record effective sampling probability and account for biased analysis. Metrics for total/error/latency remain authoritative for unsampled population; traces explain examples. Protect high-value security/payment traces through policy only when privacy allows.
X-Ray and migration
AWS recommends migration toward OpenTelemetry as X-Ray SDK/daemon enters maintenance mode under the published support timeline. Inventory language SDKs, annotations/metadata, sampling rules, daemon endpoints, trace-header propagation, Lambda/ECS/EKS/EC2 integration, service maps and queries. Migrate one boundary at a time while testing mixed propagation and duplicate spans.
Replace X-Ray SDK instrumentation with OTel auto/manual instrumentation; migrate daemon to CloudWatch Agent or OTel/ADOT Collector as appropriate; export in a compatible format; compare trace counts, topology, latency, errors and costs. Do not remove old instrumentation until duplicates and coverage are understood. Avoid two agents instrumenting the same library.
Workshop and read-only evidence
Instrument a small HTTP service calling a database and queue consumer. Define resources, spans, semantic attributes, context propagation, log correlation and collector pipeline. Test success, dependency timeout, retry, async consumer, duplicate message, invalid incoming context and collector outage.
aws xray get-trace-summaries --start-time START --end-time END --filter-expression 'service("SERVICE")' --region ap-south-1
aws xray batch-get-traces --trace-ids TRACE_ID --region ap-south-1
aws xray get-service-graph --start-time START --end-time END --region ap-south-1
aws logs start-query --log-group-name LOG_GROUP --start-time START_EPOCH --end-time END_EPOCH --query-string 'fields @timestamp, trace_id, span_id, release, message | filter trace_id="TRACE_ID"' --region ap-south-1Failure game day
Analyze 20 failures: missing service name, high-cardinality span name, secret attribute, malformed context trusted, proxy strips header, queue loses context, retry parent wrong, batch parent misleading, clock skew negative duration, double instrumentation, collector memory pressure, exporter TLS deny, queue overflows, head sampling misses errors, tail sampling capacity, inconsistent sampling across collectors, service-map edge absent, logs use different trace ID, old/new SDK duplicates, and collector outage harms application.
Cost and acceptance
Price collector compute/network, span ingestion/storage/search, Application Signals/transaction search, X-Ray traces, logs, cross-account copies and long retention. This lesson creates nothing.
Submit instrumentation contract, context threat model, collector config/topology, sampling math, async design, log/metric correlation, migration plan, seven tests, 20 failures, cost and deletion/privacy policy. Pass requires interoperable propagation, no sensitive attributes, bounded overhead, observable telemetry loss, and metrics-backed sampling interpretation.