Lesson 108 · AWS Learning Path

AWS 108: Amazon S3 concepts

Anil Jalela · Published Oct 11, 2025 · 26 min read

Why object storage exists: the problem before the service

Before object storage, applications commonly saved data on a server's local disks or on a shared file server. That works at small scale, but it couples valuable data to machines, filesystems, mount points, capacity planning, RAID, backup jobs, and replacement procedures. Adding another application server creates a harder question: which server owns the file, and how do all servers see the same current copy?

Object storage changes the interface. An application sends an authenticated HTTPS request containing a bucket, an object key, data, and optional metadata. The storage service handles placement and durability behind that API. The caller does not select a disk, partition, inode, RAID group, or storage host. This separation makes object storage suitable for application assets, backups, logs, data lakes, software artifacts, media, static websites, and many other datasets that are retrieved as whole objects.

Amazon Simple Storage Service (Amazon S3) launched in March 2006 as one of AWS's earliest infrastructure services. Its architectural importance is larger than “a place to upload files”: S3 helped make durable storage an API-accessible service that independent compute fleets could share. That history explains several design choices today. S3 uses buckets and keys rather than disks and paths, HTTP operations rather than normal POSIX system calls, and policy-controlled service endpoints rather than a filesystem mount as its native interface.

The problem for an architect is therefore not merely creating a bucket. The architect must choose the right bucket type and storage class, design object identity and concurrency, authorize every request path, protect and recover versions, control lifecycle and replication, observe data access, and predict charges. A bucket can exist while the workload remains insecure, unrecoverable, inaccessible, unexpectedly expensive, or semantically incorrect.

Final outcome

The learner will build a complete mental model of S3, trace an object request from caller to stored version and downstream event, distinguish all current S3 bucket types, select the correct protection and access controls, and produce a requirement-led storage design. The learner will also inspect the matching AWS control plane, run read-only CLI queries, diagnose a denied or missing-object case, defend one architecture choice, and prove cleanup or approved retained state.

The practical outcome is not a command transcript. It must show what was expected, what happened, what the result proves, what it does not prove, and which evidence would change the decision.

Learning objectives

By the end of this lesson, the learner can:

• explain why object storage developed and how its API differs from block and file storage;
• define bucket, object, key, prefix, metadata, tag, version ID, delete marker, ETag, checksum, storage class, endpoint, access point, and replication;
• distinguish general purpose, directory, table, and vector buckets and avoid applying one bucket type's behavior to another;
• trace the identity, TLS endpoint, authorization, data-protection, storage, event, monitoring, and billing boundaries of an S3 request;
• explain regional placement, namespaces, strong consistency, object-level atomicity, concurrency, durability, and availability;
• separate encryption, authorization, public-access prevention, immutability, version recovery, backup, and replication;
• identify every major S3 price dimension and cleanup obligation before creating data;
• connect control-plane state to the real data, network, identity, or application behavior;
• identify cost and cleanup ownership before any optional mutation;
• troubleshoot from evidence without opening broad access or adding broad permissions.

Complete S3 request and data relationship model

Human, application, or AWS service
            |
            | credentials + signed HTTPS request (or presigned request)
            v
 DNS -> S3 endpoint / access point / Multi-Region Access Point
            |
            v
 Authentication -> authorization policy evaluation -> Block Public Access guardrail
   IAM identity       bucket/access-point policy       ACL if still enabled
   session policy     VPC endpoint policy              Organizations controls
   permissions boundary                        KMS key policy/grants when required
            |
            v
 bucket type + Region/AZ scope -> key or table/index identity -> optional version ID
            |
            v
 encryption + integrity check -> atomic object operation -> response/version/checksum
            |
            +--> lifecycle / storage-class transition / expiration
            +--> replication / Batch Replication / backup
            +--> event notification -> SQS, SNS, Lambda, or EventBridge
            +--> CloudTrail data event / server access log / metrics
            +--> Inventory / Storage Lens / analytics and cost records

Every arrow is a verification boundary. A successful CreateBucket call proves only control-plane creation. It does not prove that the intended principal can upload, that an application can resolve and reach the endpoint, that a KMS key permits decrypt, that a replica meets its recovery objective, or that lifecycle and retention preserve data for the required period.

Prerequisites, permissions, Region, and safety

• Learning baseline: This sequence assumes practical Linux knowledge but no prior cloud-computing or AWS knowledge. Cloud, networking, security, data, automation, and architecture concepts must come from completed earlier lessons. If a prerequisite checkpoint is incomplete, return to its linked lesson before continuing.
• Confirm a non-root caller with aws sts get-caller-identity and keep the account number private.
• Use ap-south-1 unless this lesson explicitly names a second Region.
• Confirm the intended profile and Region with aws configure list before interpreting an empty result.
• Use read-only List, Get, and Describe permissions for the named services. Design exercises run locally and require no resource-creation permission.
• This is a no-create lesson. Console and CLI work is read-only, and every design artifact is created locally.
• Never publish account IDs, public addresses, ARNs containing private account data, session IDs, presigned URLs, object data, credentials, or KMS material.
• Do not use root, world-open SSH or RDP, disabled TLS verification, unowned resources, or irreversible retention controls in a training exercise.

Vocabulary: build the model before memorizing features

The four current S3 bucket types

Do not treat “S3” as one bucket implementation with identical APIs. Select the type before applying feature assumptions.

New services evolve. General purpose buckets remain the default teaching path in AWS 108–115 and the live lab in AWS 120. Directory, table, and vector buckets are introduced here so an architect recognizes that their scope and APIs differ; current feature support must be checked before production design.

Object storage is not a Linux filesystem

A Linux filesystem offers directories, byte-range updates, file descriptors, append behavior, rename operations, permissions attached to filesystem objects, and often POSIX locking semantics. Native S3 offers HTTP API operations against object keys. The console may draw folders and tools may present S3 through a filesystem-like interface, but presentation does not create full POSIX semantics.

Three consequences matter immediately:

1. Renaming a general purpose bucket object is normally implemented as copy plus delete. It is not a metadata-only filesystem rename.
2. Updating a normal object means putting or copying a replacement object. Readers see a complete old or complete new object, but a workflow that changes many keys is not one atomic transaction.
3. An application that requires shared locking, frequent small random writes, or POSIX permissions should evaluate EFS, FSx, or block storage rather than assuming S3 is a drop-in mount.

Region, namespace, and data location

A general purpose bucket has one home Region. A bucket name's uniqueness scope is a naming rule; it does not mean the data is automatically global. AWS now supports both the traditional shared global namespace within an AWS partition and an account Regional namespace for eligible new general purpose buckets. The learner must inspect which namespace was selected rather than relying on an old universal statement that every bucket name is globally unique.

S3 does not copy data to another Region merely because clients access it globally. Cross-Region Replication, S3 Batch Replication, an application copy, backup copy, or another explicit mechanism creates another Regional copy. Directory buckets are zonal, so a single-AZ failure consideration is part of the type choice. Data-residency analysis must include replicas, backups, analytics exports, logs, KMS keys, and downstream event consumers - not only the source bucket.

Consistency, atomicity, and concurrent writers

General purpose bucket object PUT, COPY, and DELETE operations and related reads/listing have strong consistency: after a successful write, a subsequent read does not need an eventual-consistency delay to discover that write. Strong consistency does not create a multi-object database transaction.

Suppose two processes read version A and both upload a replacement to the same unversioned key. Each PUT can succeed, while the later completed write becomes current. The application can still lose an update. Controls include versioning, conditional requests such as If-Match or If-None-Match, checksums, application-level idempotency, object naming that avoids overwrites, and a transactional coordination service when several objects must change as one business operation.

One completed object operation is atomic: a reader does not receive half of an old object and half of its replacement. Multipart upload is a transfer protocol; the completed multipart object becomes visible after successful completion. Uncompleted parts can remain billable and should be aborted explicitly or by lifecycle automation.

Durability, availability, resilience, and recovery are different

• Durability asks whether stored data survives failures over time. S3 Standard is designed for eleven nines of durability, but that statement does not recover an authorized accidental deletion or malicious overwrite by itself.
• Availability asks whether a permitted request succeeds when needed. Storage class, Region/AZ scope, endpoint path, DNS, policies, KMS, quotas, and application retries affect it.
• Resilience asks how the workload continues or degrades through a component, AZ, or Regional event.
• Recovery asks which point in time can be restored and how long restoration takes. Versioning, Object Lock, AWS Backup, replication, archive restore, and tested runbooks address different threats.

Replication can also reproduce an unwanted change depending on configuration. Versioning preserves recoverable versions but still needs permission controls, lifecycle, monitoring, and tested recovery. Object Lock protects selected versions from deletion for a retention requirement but creates operational and legal consequences. No single checkbox means “backup, disaster recovery, ransomware protection, and compliance completed.”

Authorization: why an S3 request is allowed or denied

Evaluate the effective request, not only one policy document. Relevant layers can include:

• the caller's IAM identity policy, session policy, permissions boundary, and Organizations service control policy;
• the bucket policy or access-point policy;
• S3 Block Public Access at account, bucket, or access-point scope;
• Object Ownership and ACLs where ACLs remain enabled;
• a VPC endpoint policy and policy conditions such as source VPC endpoint, network origin, TLS, principal, prefix, object tags, or encryption headers;
• the AWS KMS key policy and grants for SSE-KMS or DSSE-KMS operations;
• Object Lock retention/legal hold for delete or overwrite attempts.

An explicit deny wins over an allow. s3:ListBucket targets a bucket ARN and can be constrained by prefix; object actions such as s3:GetObject target object ARNs. A principal may list a bucket but not read objects, read one prefix but not another, or read object metadata but fail decrypt because KMS authorization is missing. Block Public Access is a guardrail against public-policy/ACL exposure; it does not replace least-privilege private policies.

For modern general purpose buckets, prefer Bucket owner enforced Object Ownership so ACLs are disabled unless a documented integration requires them. Keep all four Block Public Access settings enabled unless a reviewed public distribution requirement proves otherwise. For public content, evaluate CloudFront with Origin Access Control so the bucket can remain private.

Encryption and integrity: separate choices

S3 encrypts new uploads at rest by default with SSE-S3, but an architect still chooses the required control and verifies existing data and policy behavior.

Encryption at rest does not authorize a caller. TLS protects transport but does not prove object integrity after application processing. Use checksum validation where required, require secure transport in policy, and test both S3 permission and KMS permission paths.

The S3 feature map for this module

This lesson establishes the system. The following lessons deepen and test each part without changing the 423-page sequence:

Related capabilities that must appear in designs and later architecture lessons include event notifications and EventBridge, CloudTrail data events, server access logging, request metrics, Storage Lens, Inventory, Batch Operations, S3 Select alternatives/current support, static website hosting versus CloudFront, CORS, access logs, AWS Backup, Storage Gateway, DataSync, Snow Family transfer options, and cost allocation.

Service-completeness contract for an AWS architect

The S3 family is not complete until the learner can reason through each of these categories. “Know the service name” is never a pass.

1. Origin and fit: why object storage exists; object versus block, file, database, queue, and cache behavior.
2. Resource model: all current bucket types; buckets, objects, keys, prefixes, versions, metadata, tags, checksums, endpoints, tables/namespaces, and vector indexes.
3. Interfaces and transfer: Console, CLI, SDK/REST, SigV4, presigned URLs, multipart upload/copy, range requests, Transfer Acceleration, DataSync, Storage Gateway, Snow, and private endpoint paths.
4. Consistency and concurrency: strong consistency, atomic object replacement, conditional requests, idempotency, concurrent writers, multipart completion/abort, and absence of multi-key transactions.
5. Storage economics: Standard, Intelligent-Tiering tiers and monitoring, Standard-IA, One Zone-IA, Glacier Instant Retrieval, Glacier Flexible Retrieval, Glacier Deep Archive, Express One Zone, archive restore, minimum storage duration, minimum billable object size, request/retrieval/transition/data-transfer charges, and lifecycle break-even reasoning.
6. Identity and exposure: identity/resource policies, explicit deny, cross-account access, access points, MRAP, Access Grants where applicable, Object Ownership, ACL legacy cases, Block Public Access, policy validation, and Access Analyzer.
7. Network and delivery: public service endpoints, gateway/interface endpoint differences, endpoint policies, DNS, TLS-only enforcement, CloudFront Origin Access Control, website endpoint limitations, CORS, acceleration, dual-stack/FIPS requirements, and on-premises paths.
8. Data protection: default encryption, SSE-S3, SSE-KMS, DSSE-KMS, SSE-C, client-side encryption, KMS key policy/grants, Bucket Keys, checksum validation, versioning, MFA Delete limitations/operations, Object Lock, legal hold, backup, and restore.
9. Lifecycle and governance: rule filters and precedence, transitions, current/noncurrent expiration, expired delete markers, incomplete multipart uploads, retention conflicts, legal discovery, data classification, ownership, and deletion approval.
10. Replication and multi-Region design: SRR, CRR, live versus existing objects, Batch Replication, RTC, delete-marker behavior, replica modification/ownership, KMS permissions, metrics, failure reasons, MRAP routing and failover controls, RPO/RTO, and replication cost.
11. Events and automation: notification destinations, EventBridge, duplicate/out-of-order event handling, recursion prevention, S3 Batch Operations, Inventory-driven remediation, IaC configuration, policy-as-code, and safe deployment/rollback.
12. Observability and audit: CloudTrail management versus data events, server access logs, CloudWatch request/storage metrics, Storage Lens, Inventory, replication metrics, KMS logs, AWS Config/Security Hub controls, ownership of alarms, and evidence retention.
13. Performance: request-rate scaling, parallelism, multipart thresholds, byte-range fetches, retry/backoff, connection reuse, checksums, small-object overhead, latency-sensitive directory buckets, and measurement before optimization.
14. Failure and troubleshooting: AccessDenied, wrong Region/endpoint, missing current version, delete marker, KMS deny, policy/BPA conflict, CORS versus authorization, signature/clock errors, replication failure, archive-not-restored response, throttling/retry behavior, incomplete multipart cost, and cleanup blocked by versions or retention.
15. Architecture and certification: select S3 from requirements; reject close alternatives with evidence; calculate failure, security, recovery, operational, and cost consequences; build and test with least privilege; then explain what remains unproven.

Completeness does not mean memorizing every REST header or obscure API parameter. It means no concept needed to understand, build, secure, automate, operate, troubleshoot, recover, price, compare, or select the service at the target architect level is silently omitted.

How a general purpose bucket request works

An S3 architecture starts with data owner, bucket type, object identity, Region, access pattern, retention, recovery point, recovery time, mutability, encryption, authorization, network path, replication, event consumers, audit evidence, and deletion obligations.

For a representative GetObject request:

1. The application determines the exact endpoint, bucket/access point, key, and optional version ID. DNS resolves the endpoint and the client establishes TLS.
2. The client signs the request with temporary credentials, uses a presigned signature, or invokes through an AWS service integration. Authentication identifies the principal and session context.
3. AWS evaluates applicable organization, session, boundary, identity, endpoint, bucket/access-point, ACL, and Block Public Access controls. An explicit deny ends the request.
4. S3 resolves the requested version. If the current version is a delete marker and no older version ID was requested, the result behaves as deleted.
5. For SSE-KMS or DSSE-KMS, the KMS authorization/key state must also permit the cryptographic operation. S3 retrieves/decrypts the stored value and validates the service-side integrity path.
6. S3 returns status, headers, metadata, checksum information where requested/supported, and bytes. The client must handle retries, ranges, checksums, and application validation correctly.
7. Configured logs, CloudTrail data events, request metrics, and downstream application telemetry create different evidence. No one signal automatically provides a complete audit record.
8. Storage, request, retrieval, transfer, KMS, acceleration, replication, analytics, and logging dimensions can generate charges depending on the chosen path.

Read the result in layers:

1. Scope: account, Region, VPC, bucket, AZ, endpoint, principal, object version, or resource ARN.
2. Control plane: the requested configuration exists and reached an expected state.
3. Behavior: the request, connection, health check, replication, restore, or application result meets the requirement.
4. Operations: monitoring, failure owner, cost, retention, rollback, and cleanup are known.

Control-plane success is necessary but not sufficient. A resource can be available while policy, routing, DNS, health, data, or application behavior remains wrong.

Architecture decision table

Professional questions normally contain several valid services. State the requirement that selects one option, why the nearest alternative fails it, and what changed requirement would reverse the choice.

Worked examples: reason from requirements

Example 1: application images shared by an Auto Scaling fleet

Requirement: any EC2 instance may upload or retrieve a profile image; instances are disposable; images must survive instance replacement.

Choose a private general purpose bucket. Give the EC2 workload an IAM role scoped to the required object prefix, use HTTPS and default encryption, enable versioning if overwrite recovery is required, and define lifecycle/retention. Do not store the only copy on an instance-store disk or one instance's EBS volume. CloudFront may become the controlled read path when global caching and public delivery are required, while uploads remain authenticated to an API/S3 path.

What is not yet proven: image authorization between tenants, safe content validation, lifecycle economics, recovery behavior, and application handling of retries and concurrent writes. Those require tests.

Example 2: Linux application requiring shared append and file locking

Requirement: several EC2 instances open the same files, append small records, use filesystem locks, and expect POSIX rename behavior.

Do not select S3 merely because the data is called “files.” Evaluate EFS for a managed shared Linux filesystem or an appropriate FSx service when its protocol/features fit. Re-architecting the application to immutable objects may make S3 viable, but that is an application-design change, not a mount option.

The reversal condition: if the application writes complete immutable log segments and readers retrieve by object key, S3 can become the better durable destination.

Example 3: regulated records with seven-year non-deletion

Requirement: each final record must be retained for seven years, unauthorized deletion must be prevented, auditors need evidence, and a separate Region is required.

Evaluate a versioned general purpose bucket with Object Lock. Compliance mode may fit a true non-bypass requirement, whereas governance mode permits specifically authorized bypass. Add least-privilege write/read roles, encryption based on the key-control requirement, replication designed for locked objects, audit events, inventory/evidence, and a tested retrieval process. Confirm retention dates and legal-hold procedures with the data owner and legal/compliance authority before creating irreversible settings.

Versioning alone fails the non-deletion requirement; replication alone can copy bad writes/deletes depending on configuration; encryption alone does not prevent deletion.

Example 4: daily telemetry queried as changing SQL tables

Requirement: analysts use SQL; schemas and partitions evolve; table-file compaction and snapshot maintenance are an operational burden.

Evaluate an S3 table bucket and its Apache Iceberg integration with the intended query engines. A normal general purpose bucket can store Iceberg files, but the team would own more table maintenance. Validate Region availability, engine compatibility, IAM/resource policies, quotas, replication/recovery, and pricing before choosing the newer bucket type.

Cost model: calculate requests as well as bytes

Never copy a numeric price into a permanent lesson and assume it remains current. Use the current S3 pricing page for the selected Region, then calculate from measured or estimated units.

monthly estimate = storage GB-month by class
                 + PUT/COPY/POST/LIST and transition requests
                 + GET/SELECT and other request classes
                 + retrieval GB and archive restore requests
                 + internet, inter-Region, acceleration, or other transfer
                 + replication destination storage and replication requests
                 + Intelligent-Tiering monitoring where applicable
                 + Inventory, Storage Lens advanced, Batch Operations, or analytics
                 + KMS API usage and logging/monitoring destinations
                 + early-deletion or minimum-object-size effects

Worked reasoning: one million 10-KiB objects total only about 9.54 GiB, but lifecycle transitions can create one million transition requests and some classes apply a minimum billable object size. “Very little data” therefore does not mean “negligible S3 bill.” Conversely, one 100-GiB object has low object count but can create meaningful retrieval and transfer charges. Always model object count, average size, request frequency, retention, retrieval probability, and transfer path together.

AWS Management Console guided practice

Before opening a service page, write the expected account, Region, starting state, and evidence. Do not choose Create, Save, Purchase, Lock, or Delete unless the lesson explicitly authorizes the live track.

1. Open Amazon S3, General purpose buckets in ap-south-1, and distinguish account bucket inventory from the Region stored on each bucket.
2. Select only an owned or instructor-supplied bucket and inspect Properties, Permissions, Management, Metrics, and object Version ID fields without changing them.
3. Classify ten workload statements as object, block, or file storage and defend the S3 choices by API and consistency behavior.

For each step, capture the field name and value in text. A screenshot may support the record but does not replace the explanation. Console labels can evolve, so use the service search and current documentation if a navigation label differs.

CloudShell and AWS CLI practice

CloudShell is the default browser-based command environment taught in AWS 028. AWS 029 and AWS 030 cover local CLI installation and authentication. This lesson therefore does not assume that an unconfigured local shell is ready.

Start every session with:

export AWS_DEFAULT_REGION="ap-south-1"
aws sts get-caller-identity --query Arn --output text
aws configure list

Redact the account portion of the ARN before sharing. Then perform the topic query:

List owned general purpose buckets, then query the Region and security posture only for an instructor-approved bucket.

NW_BUCKET="replace-with-owned-bucket-name"
aws s3api list-buckets --query 'Buckets[].{Name:Name,Created:CreationDate}' --output table
aws s3api get-bucket-location --bucket "$NW_BUCKET"
aws s3api get-public-access-block --bucket "$NW_BUCKET"

Expected interpretation:

A bucket list is account-scoped but not proof of object access, encryption mode, versioning, public exposure, data residency, lifecycle, or recovery. Never query or modify an unowned bucket name.

Replace every replace-with-... sample value before running its command, and use only an explicitly owned resource. Explain each option first. These queries are read-only; a successful response does not authorize a later create or delete operation.

Practical work

Create p06-data-requirements.md for a protected document dataset. Define object key scheme, source Region ap-south-1, replica Region ap-southeast-1, data owner, expected object sizes, read/write frequency, retention, RPO, RTO, encryption, public-access prohibition, identity boundary, lifecycle, replication, restore test, evidence, and final deletion. No bucket is created.

The evidence package must contain:

• the problem and final requirement in the learner's own words;
• caller type and Region with private identifiers redacted;
• exact planned values, ownership, and cost class;
• one Console observation and matching CLI or API evidence;
• one behavior result or supplied data-plane record;
• one denied, failed, or counterexample result and evidence-led diagnosis;
• one architecture choice plus the rejected alternative;
• cleanup proof or explicit retained-state owner, expiry, and next lesson.

Verification standard

Use expected state before observed state. Record timestamps in UTC and preserve the original failure before changing anything. A passing submission answers all four questions:

1. What exact requirement was tested?
2. Which evidence proves the AWS configuration?
3. Which evidence proves the workload behavior?
4. What remains unproven or requires later monitoring?

If AWS returns no rows, verify account, Region, permission, filters, pagination, resource type, and deletion state before concluding that nothing exists.

Common failures and troubleshooting

Do not troubleshoot by attaching administrator access, opening administration ports to the internet, disabling encryption, retrying uncontrolled creation, deleting unknown resources, or weakening retention.

Cost, cleanup, and retained state

No AWS resource is created. Close CloudShell and remove or redact downloaded evidence.

Cleanup evidence requires terminal state and an after-inventory. Search related ENIs, public IPv4 addresses, EBS volumes and snapshots, load balancers, target groups, Auto Scaling instances, endpoints, logs, S3 versions and delete markers, backup recovery points, and global IAM roles when they apply. Billing data can lag, so schedule a later review.

Architecture and certification decisions

• Certification coverage: SAA-C03; SOA-C03; SAP-C02; DOP-C02.
• Exam mapping: SAA D1-D4.
• Explain service scope, failure boundary, consistency, recovery, security, operations, and price rather than matching a keyword.
• Treat availability and durability, encryption and authorization, routing and filtering, health and lifecycle, backup and replication, and discount and capacity as separate concepts.
• Do not reproduce protected certification questions.

Knowledge check

1. Is S3 a POSIX filesystem?

Expected direction: No. It is an object service.

1. Does global bucket-name uniqueness make bucket data global?

Expected direction: No. A bucket is created in a Region.

1. Does eleven nines durability equal constant availability?

Expected direction: No. Durability and availability are different requirements.

1. What identifies a versioned object precisely?

Expected direction: Bucket name, object key, and version ID.

Completion gate and assessment

Pass at 80 out of 100 with no critical safety failure. A missing practical artifact, unexplained output, unsafe access, destructive action outside the owned scope, unplanned billed resource, or false cleanup claim requires remediation and a changed retest.

Official sources

• AWS origins and the problem infrastructure services were built to solve
• Amazon S3 overview, current bucket types, resources, and consistency model
• General purpose bucket namespaces
• Directory buckets and S3 Express One Zone
• S3 Tables and table buckets
• S3 Vectors and vector buckets
• S3 storage classes
• S3 security best practices
• S3 Block Public Access
• S3 Bucket Keys for SSE-KMS
• Amazon S3 pricing