Lesson 114 · AWS Learning Path

AWS 114: S3 performance, multipart upload and Transfer Acceleration

Anil Jalela · Published Oct 17, 2025 · 15 min read

The real problem

A team recognizes the name S3 performance, multipart upload and Transfer Acceleration but has not connected the feature to a real requirement, identity boundary, network or data path, failure mode, price dimension, and cleanup owner. A plausible configuration could still fail the workload.

Final outcome

The learner will produce a requirement-led artifact for S3 performance, multipart upload and Transfer Acceleration, inspect the matching AWS control plane in the Management Console, run a matching CloudShell or AWS CLI query, interpret the output, diagnose one failure, defend one architecture choice, and prove cleanup or approved retained state.

The practical outcome is not a command transcript. It must show what was expected, what happened, what the result proves, what it does not prove, and which evidence would change the decision.

Learning objectives

By the end of this lesson, the learner can:

• explain horizontal request scaling;
• explain scaling is gradual;
• explain multipart upload;
• explain byte-range requests;
• explain transfer acceleration;
• connect control-plane state to the real data, network, identity, or application behavior;
• identify cost and cleanup ownership before any optional mutation;
• troubleshoot from evidence without opening broad access or adding broad permissions.

Relationship model

Requirement
   |
   v
Identity and policy -> AWS configuration -> network or data path -> workload behavior
        |                    |                      |                    |
        +--------------------+----------------------+--------------------+
                                      |
                                      v
                         monitoring, cost, recovery, cleanup

Use this model to separate an AWS object that exists from a result that actually works. Every arrow is a verification boundary.

Prerequisites, permissions, Region, and safety

• Learning baseline: This sequence assumes practical Linux knowledge but no prior cloud-computing or AWS knowledge. Cloud, networking, security, data, automation, and architecture concepts must come from completed earlier lessons. If a prerequisite checkpoint is incomplete, return to its linked lesson before continuing.
• Confirm a non-root caller with aws sts get-caller-identity and keep the account number private.
• Use ap-south-1 unless this lesson explicitly names a second Region.
• Confirm the intended profile and Region with aws configure list before interpreting an empty result.
• Use read-only List, Get, and Describe permissions for the named services. Design exercises run locally and require no resource-creation permission.
• This is a no-create lesson. Console and CLI work is read-only, and every design artifact is created locally.
• Never publish account IDs, public addresses, ARNs containing private account data, session IDs, presigned URLs, object data, credentials, or KMS material.
• Do not use root, world-open SSH or RDP, disabled TLS verification, unowned resources, or irreversible retention controls in a training exercise.

Core model

How it works

Performance is an end-to-end result across client CPU and network, internet distance, TLS, request concurrency, object and part size, key distribution, S3 service behavior, KMS, application retry logic, and destination processing.

Read the result in layers:

1. Scope: account, Region, VPC, bucket, AZ, endpoint, principal, object version, or resource ARN.
2. Control plane: the requested configuration exists and reached an expected state.
3. Behavior: the request, connection, health check, replication, restore, or application result meets the requirement.
4. Operations: monitoring, failure owner, cost, retention, rollback, and cleanup are known.

Control-plane success is necessary but not sufficient. A resource can be available while policy, routing, DNS, health, data, or application behavior remains wrong.

Architecture decision table

Professional questions normally contain several valid services. State the requirement that selects one option, why the nearest alternative fails it, and what changed requirement would reverse the choice.

Performance starts with a measurable path

application serialization / disk read
        -> DNS -> TCP/TLS -> endpoint/network path
        -> request signing -> S3 request processing
        -> optional KMS operation
        -> response bytes -> client network/disk/CPU
        -> application validation and downstream processing

Measure object size distribution, concurrency, operations per second, throughput, first-byte latency, end-to-end latency percentiles, retry/error rate, CPU, memory, file I/O, network bandwidth, NAT/endpoint path, KMS utilization, and cost. A faster S3 response cannot overcome a single-threaded client, saturated instance network, slow local disk, distant Region, undersized NAT path, KMS throttling, or downstream consumer bottleneck.

Define a workload before tuning: for example, “sustain 2 GiB/s aggregate reads of 256-MiB immutable objects with p99 completion under 3 seconds from EC2 in the same Region, error rate below 0.1%, and no public path.” Without units and percentile, “S3 is slow” is not testable.

Request-rate scaling and prefix design

For general purpose buckets, AWS documents baseline scaling of at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes. S3 scales beyond those values, but rapid increases can temporarily produce 503 Slow Down while partitioning adapts.

Modern S3 does not require randomizing the first characters of every key for ordinary performance. Prefix parallelism still matters for very high request rates, but key design must also serve policy, lifecycle, listing, analytics, event, and operational needs. Measure before introducing hash prefixes that make operations harder.

Clients must use bounded retries with exponential backoff and jitter, distinguish retryable status from permanent authorization/input failures, reuse connections, cap concurrency, and preserve idempotency. A retrying PUT to a versioned key can create additional versions; unique operation IDs or conditional writes can prevent ambiguous duplicate business effects.

Multipart upload from first principle

Multipart upload is a three-stage protocol:

1. CreateMultipartUpload returns an upload ID and fixes metadata/encryption options for the planned object.
2. Upload numbered parts independently and retain each returned part ETag/checksum. Parts may be retried and uploaded out of order.
3. CompleteMultipartUpload sends the ordered part-number/ETag list. Only then does S3 assemble and expose the final object. AbortMultipartUpload removes uploaded parts when abandoning the operation.

Current limits include up to 10,000 parts, part numbers 1–10,000, normal part size 5 MiB–5 GiB, no minimum for the final part, and a maximum object near 48.8 TiB. Tool/SDK and feature support can lag changed service limits, so validate the entire transfer stack.

Part-size formula:

minimum practical part size = max(5 MiB, ceiling(object bytes / 10,000))

Then increase it to account for memory, retry cost, network bandwidth-delay product, desired concurrency, API request cost, and SDK behavior. A 40-TiB object needs parts larger than 4 MiB merely by count, but the 5-MiB service minimum is still far too small because it would exceed 10,000 parts; around 4.1 GiB or larger is needed by count. Leave headroom rather than designing exactly at a limit.

Multipart upload improves retry granularity and parallelism; it does not make one object partly readable before completion. Incomplete parts incur storage charges and normal object listing will not reveal them. Inventory multipart uploads and apply lifecycle abort automation as a safety net, while applications abort known failures directly.

Multipart checksums and trustworthy completion

Choose a supported checksum algorithm and configure the SDK/CLI intentionally. Store local expected checksum evidence, verify the API's returned whole-object/checksum behavior, and download/validate a test object. ETags are required in multipart completion but are not universal MD5 content proofs.

Common integrity mistakes include mixing part order, reusing an upload ID for the wrong local file, recording only ETags, trusting a successful HTTP status without validating bytes, and calculating a local checksum before a file has finished changing. Freeze input content for the upload and make the operation idempotent.

Parallel downloads and range requests

Range GET requests can retrieve byte ranges in parallel. For an object originally uploaded in known parts, ranges aligned with part boundaries can simplify performance and integrity reasoning. Select range size and worker count from client memory, CPU, network, object size, downstream write behavior, and request cost. More threads can reduce performance through contention or throttling.

S3 does not provide one atomic multi-range GET response. The client must ensure all ranges belong to the intended version, place bytes correctly, handle retry/short response, and validate final integrity. Pin a version ID or conditional ETag/checksum when an object could be overwritten during download.

Transfer paths and tools

Transfer Acceleration benefits clients far from the bucket over variable internet paths. It can be slower or economically unjustified for nearby clients or already optimized private AWS paths. Use the AWS speed comparison where applicable and a representative benchmark containing cold/warm runs, meaningful object sizes, multiple locations, error rates, and total cost.

S3 Express One Zone performance decision

S3 Express One Zone uses directory buckets in one AZ and is designed for high request rates and single-digit-millisecond access. Place compute in the same AZ, use zonal endpoints and current authorization/session behavior, and validate SDK support. Its performance benefit does not erase single-AZ resilience, feature differences, naming/API constraints, or request/storage cost. It is not simply a faster storage class toggle on a general purpose bucket.

Worked benchmark designs

Example 1: 200-GiB upload from Mumbai to another continent

Compare normal Regional endpoint, Transfer Acceleration, and - if repeated/managed migration - DataSync. Use the same immutable input/checksum, at least three runs per path, bounded concurrency, elapsed and p95 part time, retries, CPU, throughput, and projected charges. A one-time fastest run is not enough.

Example 2: 80,000 8-KiB GETs per second

Object count and request rate dominate bytes. Distribute traffic across designed prefixes, reuse connections, scale clients, watch 503s and latency, and model GET/KMS/logging cost. Consider caching or packaging only if application semantics permit. Multipart is irrelevant to tiny GET objects.

Example 3: 10-TiB scientific object

Use multipart with part sizing safely below 10,000 parts, parallel upload and durable upload-state tracking. Pin exact version/checksum for parallel range download, reconstruct to preallocated/streamed local output, validate whole-object checksum, and abort abandoned uploads. Confirm SDK maximum-object support before committing to the format.

Example 4: intermittent 503 during launch event

Correlate per-prefix request rate, sudden traffic ramp, retry attempts, latency, client connection pools, KMS throttle metrics, and CloudFront cache misses. Add jittered retries and gradual ramp where appropriate; spread designed keys/prefixes or caching paths based on evidence. Do not change bucket permissions or Region because a retryable scaling response occurred.

Required positive, negative, and dependency tests

1. Upload and download a large generated non-sensitive file through controlled multipart settings; verify byte count and checksum.
2. Start a multipart upload, upload one part, inventory it, abort it, and prove the multipart inventory is clear.
3. Attempt completion with an incorrect/missing part record and preserve the expected failure; do not abandon billable parts.
4. Run the same bounded benchmark at concurrency 1, 4, and a justified higher value; graph throughput, p95, retries, CPU, and cost rather than assuming linear scaling.
5. Test an endpoint/network/KMS dependency failure separately from S3 performance and identify the exact failing boundary.

AWS Management Console guided practice

Before opening a service page, write the expected account, Region, starting state, and evidence. Do not choose Create, Save, Purchase, Lock, or Delete unless the lesson explicitly authorizes the live track.

1. Open an approved bucket, Properties, Transfer acceleration and inspect Disabled or Enabled state without changing it.
2. Use supplied upload traces to compare single PUT, multipart part retries, concurrency, checksum evidence, elapsed time, and abandoned-part cost.
3. Open Management lifecycle configuration and verify the planned 7-day incomplete multipart abort rule for AWS 120.

For each step, capture the field name and value in text. A screenshot may support the record but does not replace the explanation. Console labels can evolve, so use the service search and current documentation if a navigation label differs.

CloudShell and AWS CLI practice

CloudShell is the default browser-based command environment taught in AWS 028. AWS 029 and AWS 030 cover local CLI installation and authentication. This lesson therefore does not assume that an unconfigured local shell is ready.

Start every session with:

export AWS_DEFAULT_REGION="ap-south-1"
aws sts get-caller-identity --query Arn --output text
aws configure list

Redact the account portion of the ARN before sharing. Then perform the topic query:

List incomplete multipart uploads and inspect acceleration status without enabling it.

NW_BUCKET="replace-with-owned-bucket-name"
aws s3api list-multipart-uploads --bucket "$NW_BUCKET" --output table
aws s3api get-bucket-accelerate-configuration --bucket "$NW_BUCKET"

Expected interpretation:

An empty multipart list supports no currently visible incomplete uploads in that bucket. It does not prove client retry quality, transfer throughput, or other buckets are clean.

Replace every replace-with-... sample value before running its command, and use only an explicitly owned resource. Explain each option first. These queries are read-only; a successful response does not authorize a later create or delete operation.

Practical work

Create p06-transfer-plan.md. Choose a multipart threshold and part size for a 20 GiB example, calculate part count, define concurrency, retry/backoff, checksum, resume ledger, abort workflow, and current request cost. Compare standard and accelerated endpoints from three supplied client locations. AWS 120 uploads only tiny text objects and validates the abort rule configuration without creating a large file.

The evidence package must contain:

• the problem and final requirement in the learner's own words;
• caller type and Region with private identifiers redacted;
• exact planned values, ownership, and cost class;
• one Console observation and matching CLI or API evidence;
• one behavior result or supplied data-plane record;
• one denied, failed, or counterexample result and evidence-led diagnosis;
• one architecture choice plus the rejected alternative;
• cleanup proof or explicit retained-state owner, expiry, and next lesson.

Verification standard

Use expected state before observed state. Record timestamps in UTC and preserve the original failure before changing anything. A passing submission answers all four questions:

1. What exact requirement was tested?
2. Which evidence proves the AWS configuration?
3. Which evidence proves the workload behavior?
4. What remains unproven or requires later monitoring?

If AWS returns no rows, verify account, Region, permission, filters, pagination, resource type, and deletion state before concluding that nothing exists.

Common failures and troubleshooting

Do not troubleshoot by attaching administrator access, opening administration ports to the internet, disabling encryption, retrying uncontrolled creation, deleting unknown resources, or weakening retention.

Cost, cleanup, and retained state

No AWS resource is created. Close CloudShell and remove or redact downloaded evidence.

Cleanup evidence requires terminal state and an after-inventory. Search related ENIs, public IPv4 addresses, EBS volumes and snapshots, load balancers, target groups, Auto Scaling instances, endpoints, logs, S3 versions and delete markers, backup recovery points, and global IAM roles when they apply. Billing data can lag, so schedule a later review.

Architecture and certification decisions

• Certification coverage: SAA-C03; SOA-C03; SAP-C02; DOP-C02.
• Exam mapping: SAA D1-D4.
• Explain service scope, failure boundary, consistency, recovery, security, operations, and price rather than matching a keyword.
• Treat availability and durability, encryption and authorization, routing and filtering, health and lifecycle, backup and replication, and discount and capacity as separate concepts.
• Do not reproduce protected certification questions.

Knowledge check

1. Why use multipart upload?

Expected direction: Parallelism, retrying individual parts, and support for objects beyond the single PUT limit.

1. What happens to abandoned parts?

Expected direction: They remain stored and billable until aborted.

1. Does Transfer Acceleration always improve speed?

Expected direction: No. Benchmark representative paths and cost.

1. Is ETag always the object MD5?

Expected direction: No. Multipart and encryption cases differ.

Completion gate and assessment

Pass at 80 out of 100 with no critical safety failure. A missing practical artifact, unexplained output, unsafe access, destructive action outside the owned scope, unplanned billed resource, or false cleanup claim requires remediation and a changed retest.

Official sources

• S3 performance guidelines
• S3 performance design patterns
• Multipart upload overview
• Current multipart upload limits
• Transfer Acceleration
• S3 checksums
• S3 Express One Zone
• AWS DataSync with S3