Lesson 301 · AWS Learning Path

AWS 301: AWS Lake Formation

Anil Jalela · Published Apr 22, 2026 · 14 min read

Why this lesson matters

An analyst can have an IAM policy that allows Athena and S3 yet still be denied by Lake Formation. Another analyst can have a Lake Formation SELECT grant yet fail because IAM does not permit the Glue or Athena API. A third can query every column because the default IAMAllowedPrincipals grant was never removed. All three outcomes follow from different authorization paths.

AWS Lake Formation governs analytical metadata and underlying data used through integrated engines. It builds on the AWS Glue Data Catalog and adds database-style grants, registered S3 locations, temporary data credential vending, LF-tag-based access control, row/column/cell filters, and cross-account sharing. It does not replace IAM, classify data automatically, make a bad S3 layout efficient, or govern every possible direct S3 read.

This lesson uses the AWS300 sales table as a concrete foundation. You will model an IAM-only starting state, migrate one analyst and one table with hybrid access mode, design named-resource and LF-tag grants, restrict a support persona through a data cells filter, test positive and negative access, revoke access, and prove what changed. The default path is a design and evidence exercise. Run mutations only in a dedicated lab account with explicit approval.

What you will be able to do

By the end, you can:

• separate IAM API authorization, Lake Formation grants, Glue metadata, registered S3 locations, and engine behavior;
• explain data-lake administrators, database creators, LF-tag creators, data engineers, analysts, workflow roles, and auditors;
• distinguish DESCRIBE, SELECT, INSERT, ALTER, DROP, CREATE_TABLE, DATA_LOCATION_ACCESS, ASSOCIATE, and grant options;
• identify when IAMAllowedPrincipals makes access effectively IAM-only;
• choose Lake Formation mode, IAM-only mode, or hybrid access mode without accidental lockout;
• trace credential vending from Athena to a registered S3 location;
• use named resources for exceptions and LF-tags for scalable policy intent;
• design row, column, and cell filtering without presenting it as data masking;
• explain cross-account sharing, AWS RAM, recipient grants, and resource links;
• test allow, deny, revoke, direct-S3, and administrator paths from evidence;
• diagnose failures by locating the exact authorization gate; and
• produce a reversible migration and governance package suitable for architecture review.

Before you start

• Complete AWS300 and retain only synthetic data, table definitions, and redacted evidence.
• Use a non-production account. Do not change Lake Formation defaults, administrators, registered locations, or grants in a shared account.
• The examples use ap-south-1. Lake Formation permissions and Data Catalog resources are Regional.
• Do not grant yourself data-lake administrator solely to make a query pass. Administration and data use should be separate.
• Do not paste account numbers, complete role ARNs, sensitive row values, or query-result URLs into shared submissions.
• The optional mutation path needs a lab administrator, a separately assumable analyst role, an AWS300-style Catalog table, and an S3 prefix containing synthetic data.
• Record the pre-change grants and registration state. A rollback that does not preserve prior access is not safe.

Set read-only variables. Replace placeholders locally:

export AWS_DEFAULT_REGION="ap-south-1"
export LF_DB="nw_aws300_lab"
export LF_TABLE="sales_parquet"
export DATA_BUCKET="your-learner-owned-data-bucket"
export DATA_PREFIX="sales_parquet"
export ANALYST_ROLE_NAME="nw-lf-analyst"

export CALLER_ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
export ANALYST_ARN="arn:aws:iam::${CALLER_ACCOUNT}:role/${ANALYST_ROLE_NAME}"

aws sts get-caller-identity --query Arn --output text
aws configure list

Redact the account portion before sharing evidence. If the named database, table, bucket, prefix, or role is not explicitly learner-owned, stay on the T0 design path.

1. Build the five-layer authorization model

principal session
      |
      v
1. Organizations, boundary, session policy, and IAM API permission
      |
      v
2. Lake Formation permission on Catalog resource or LF-tag expression
      |
      v
3. Glue Data Catalog metadata: database, table, columns, partitions, location
      |
      v
4. Registered S3 location and Lake Formation temporary credential vending
      |
      v
5. Integrated engine reads data and applies an authorized data filter
      |
      v
query result, CloudTrail history, and revocation evidence

IAM and Lake Formation are complementary. A principal might have Lake Formation SELECT but still need IAM permission for lakeformation:GetDataAccess, Glue read APIs, Athena query APIs, the selected workgroup, result storage, and KMS keys. Conversely, broad Glue IAM permission does not override an enforced Lake Formation table denial.

Registration changes the underlying-data path. For a registered location, Lake Formation can assume the registration role and vend scoped temporary credentials to an integrated engine. The analyst does not need broad permanent S3 source access for that path. Registration does not stop the same principal from using a separately granted direct S3 permission, so direct access must be removed or explicitly governed if bypass prevention is a requirement.

2. Understand defaults before changing them

Each AWS account has a Data Catalog in each Region. Lake Formation is integrated with it, but existing environments often begin with IAM-compatible defaults. The virtual IAMAllowedPrincipals group can hold Super on databases and tables. When present, principals allowed by IAM and Glue policies can continue to access those resources without restrictive Lake Formation grants.

Inspect current state without changing it:

aws lakeformation get-data-lake-settings --output json \
  | tee lf-data-lake-settings.json

aws lakeformation list-resources --output json \
  | tee lf-registered-locations.json

aws lakeformation list-permissions \
  --resource "{\"Table\":{\"DatabaseName\":\"${LF_DB}\",\"Name\":\"${LF_TABLE}\"}}" \
  --output json | tee lf-table-permissions-before.json

aws glue get-table --database-name "$LF_DB" --name "$LF_TABLE" \
  --query 'Table.{Name:Name,Location:StorageDescriptor.Location,Registered:IsRegisteredWithLakeFormation,Columns:StorageDescriptor.Columns,PartitionKeys:PartitionKeys}' \
  --output json | tee lf-table-before.json

Do not infer too much from one caller's output. Metadata visibility itself is permission-filtered. A non-administrator can receive an incomplete inventory. Compare administrator inventory, principal-specific tests, and CloudTrail events.

Default-setting choices for future databases and tables are account-and-Region governance changes. Turning off IAM compatibility without a complete grant inventory can break Athena, Glue, EMR, Redshift Spectrum, jobs, and automation. Prefer an explicit migration plan or hybrid onboarding.

3. Learn the permission vocabulary

Lake Formation grants resemble database permissions, but resource type matters:

DATA_LOCATION_ACCESS does not grant permission to query data. It enables metadata creation against a registered path and works with other IAM and Lake Formation grants. Grantable permissions let a grantee pass a permission onward; they should be rarer than use permissions.

Administrators have implicit Lake Formation authority, which makes them poor test identities. Test with the intended analyst session.

4. Choose an adoption mode

Hybrid is not a global magic switch. The S3 location is registered in hybrid mode, and specific principal/resource pairs are opted in. For an opted-in principal, Lake Formation permissions become effective. Non-opted principals can continue through IAM compatibility where configured.

Create a migration matrix before mutation:

5. Register one location safely

Registration covers the exact S3 ARN and child paths. The service-linked role is convenient for same-account locations; use a custom role when trust, KMS, cross-account, logging, or policy requirements differ.

Dry-design the command first:

export DATA_ARN="arn:aws:s3:::${DATA_BUCKET}/${DATA_PREFIX}"

aws lakeformation describe-resource --resource-arn "$DATA_ARN" \
  --output json | tee lf-registration-check.txt

Only in the approved lab, register it in hybrid mode:

aws lakeformation register-resource \
  --resource-arn "$DATA_ARN" \
  --use-service-linked-role \
  --hybrid-access-enabled

aws lakeformation describe-resource --resource-arn "$DATA_ARN" \
  --output json | tee lf-registration-after.json

Prove the Catalog table location is inside the prefix and inspect IsRegisteredWithLakeFormation as the intended caller. That property can vary with hybrid opt-in context, so it is evidence, not the whole decision.

6. Grant a named resource and opt in the analyst

The analyst still needs a scoped IAM policy for Athena, Glue read calls, lakeformation:GetDataAccess, the workgroup, result access, and relevant KMS use. Do not attach administrator policies.

aws lakeformation grant-permissions \
  --principal "DataLakePrincipalIdentifier=${ANALYST_ARN}" \
  --permissions SELECT DESCRIBE \
  --resource "{\"Table\":{\"DatabaseName\":\"${LF_DB}\",\"Name\":\"${LF_TABLE}\"}}"

aws lakeformation create-lake-formation-opt-in \
  --principal "DataLakePrincipalIdentifier=${ANALYST_ARN}" \
  --resource "{\"Table\":{\"DatabaseName\":\"${LF_DB}\",\"Name\":\"${LF_TABLE}\"}}"

aws lakeformation list-lake-formation-opt-ins \
  --principal "DataLakePrincipalIdentifier=${ANALYST_ARN}" \
  --output json | tee lf-opt-in.json

Depending on inherited visibility and client behavior, the analyst may also need database DESCRIBE. Grant the minimum based on a failed test, not a blanket wildcard.

Assume the analyst role, record the caller, and run:

1. a permitted aggregate against the exact table;
2. a query against an ungranted table, expected to fail;
3. a direct S3 GetObject attempt, expected to fail if bypass prevention is required;
4. a query-result retrieval test; and
5. the same tests after revocation.

The direct-S3 negative test is essential. Lake Formation governs supported integrated access, not an independent S3 allow.

7. Scale policy with LF-tags

Named grants are explicit and good for exceptions, but become expensive across thousands of tables. LF-tag-based access control expresses intent through labels.

Only authorized tag creators should define values. Only principals with ASSOCIATE for the key-value should label resources. Separate tag creation, assignment, data ownership, and approval to prevent self-service privilege escalation.

An LF-tag policy grant is dynamic: matching resources can enter or leave access as tags change. Test tag changes like authorization changes, log them, and prevent unreviewed inheritance surprises. Database tags can be inherited by tables, with resource assignments affecting the effective tags.

Write an expression allowing a sales analyst to SELECT resources matching:

domain=sales
AND environment=prod
AND classification IN (internal, confidential)

Test five tables, including one differing by each attribute and one named exception. Document whether named and LF-tag grants combine to broaden access.

8. Design row, column, and cell controls

A data cells filter names one table, chooses included or excluded columns, and can apply a row expression. Lake Formation can grant SELECT on that filtered resource.

Design support_west:

{
  "TableCatalogId": "REDACTED_ACCOUNT",
  "DatabaseName": "nw_aws300_lab",
  "TableName": "sales_parquet",
  "Name": "support_west",
  "RowFilter": {"FilterExpression": "customer_region = 'west'"},
  "ColumnNames": ["order_id", "customer_region", "item", "quantity"]
}

Test expected rows, excluded regions, excluded unit_price, aggregates, joins, views, exports, and revocation. Do not call this masking. Authorized engines apply a filter; source values are unchanged. Direct S3, unsupported engines, exports, administrator access, and downstream copies need separate controls.

Athena engine version 3 supports current cell filtering, but engine, expression, type, statement, Region, and grant limitations must be checked for the exact path. Filters are authorization objects and need version control, ownership, review, and regression tests.

9. Understand cross-account sharing

producer Catalog resource and registered data
       |
       +--> LF named-resource or LF-tag grant
       +--> AWS RAM share and current cross-account setting
                          |
                          v
consumer accepts share when required
       |
       +--> creates local resource link
       +--> grants local principal on target and link
       +--> configures engine, results, IAM, and KMS
                          |
                          v
                       query test

A resource link is a local pointer, not a copy. Permissions on link and target differ. The producer retains source ownership and can revoke sharing. The consumer controls local delegation within the granted boundary.

Current cross-account version 5 uses wildcard resource patterns to scale beyond earlier association limits; upgrade is one-way. Version 4 or higher is needed for hybrid sharing. Existing Glue resource policies, glue:ShareResource, RAM organization sharing, KMS policy, and data ownership can affect success. Never upgrade production merely to follow a lesson.

10. Audit, revoke, and prove removal

CloudTrail can show Lake Formation and Glue control calls, grants, opt-ins, registrations, and engine requests. Athena history shows query outcome and identity. S3 and KMS evidence covers storage and keys. No single log proves the whole decision.

Build a grant inventory with principal, resource, permission, grant option, LF-tag expression, opt-in, owner, approval, created time, review date, and expiry. Flag:

• data-lake administrators used for routine querying;
• Super outside controlled administration;
• delegation without a business need;
• IAMAllowedPrincipals on supposedly enforced resources;
• stale external accounts or roles;
• direct S3 bypass access;
• orphaned filters and links; and
• tags with no owner or contradictory classification.

Revocation must be tested. Remove the analyst's grant, wait for propagation where relevant, start a new session, run the formerly allowed query, and verify denial. Existing result files, caches, downloads, or direct S3 permission do not disappear when SELECT is revoked.

For hybrid rollback, remove opt-in before removing a grant only if intentionally restoring the documented prior IAM path. Deregister a location only after proving no governed consumer depends on it. Deregistration does not delete S3 data.

11. Diagnose failures from the exact gate

Do not grant administrator, Super, broad S3, and wildcards simultaneously. That destroys evidence of the failed layer.

12. Cost, quotas, and operating ownership

The governed system can incur S3 storage and requests, Athena scans or capacity, Glue jobs/crawlers/Catalog usage, KMS calls, CloudTrail data events, CloudWatch retention, AWS Config, data transfer, and downstream copies. Fine-grained filters can affect plans and scans; measure rather than assuming savings.

Track quotas for grants, LF-tags, values, expressions, filters, locations, API rates, and cross-account resources. Values can change by Region. Record current quota evidence and increase lead time in the ADR.

13. Practical submission

Submit a p18-lake-formation/ package containing:

1. current IAM, Catalog, S3, KMS, and Lake Formation path;
2. redacted pre-change administrators, defaults, registrations, and grants;
3. data-owner and persona matrix;
4. IAM-only, enforced, and hybrid ADR;
5. one-location registration plan and rollback;
6. named grant and no-grant-option justification;
7. principal/resource opt-in map;
8. LF-tag taxonomy, assignment authority, and five-table tests;
9. one data cells filter with positive, negative, and bypass tests;
10. producer/consumer cross-account sequence;
11. ten-case authorization transcript;
12. revocation and new-session denial evidence;
13. direct-S3 and retained-result risk analysis;
14. CloudTrail/Athena/S3/KMS audit map;
15. cost, quota, and lifecycle register; and
16. residual risks, exceptions, and review date.

No screenshot alone passes. Each conclusion needs a redacted API result, behavior, grant record, or explicit assumption.

Knowledge check

1. Why can IAM allow but Athena deny? The request must also pass applicable Lake Formation authorization.
2. What does IAMAllowedPrincipals do? It preserves IAM-compatible access and can defeat an assumed restrictive model.
3. Does DATA_LOCATION_ACCESS let an analyst query? No; it permits metadata to point at a registered location.
4. Why is an administrator a poor test identity? Administrators have implicit Lake Formation authority.
5. What changes when S3 is registered? Integrated services can receive scoped temporary credentials.
6. What is hybrid opt-in for? It applies LF permissions to selected principal/resource pairs.
7. When are LF-tags preferable? When policy should scale across consistently classified resources.
8. Is a data cells filter masking? No; it filters supported engine results without changing source data.
9. What is a resource link? A local pointer to a shared target, not a copy.
10. Does revoking SELECT remove downloads? No; results and copies need separate controls.

Lesson acceptance

Pass only when all are true:

• five authorization layers are traced without claiming Lake Formation replaces IAM;
• defaults, IAMAllowedPrincipals, administrators, registrations, and grants are inventoried;
• IAM-only, enforced, and hybrid modes have a reversible adoption decision;
• permissions and grant options map to correct resource types;
• registration role, prefix, KMS, and credential-vending boundaries are explicit;
• one named policy has positive, negative, bypass, and revoke tests;
• LF-tag design separates creation, association, grant, and ownership;
• a data filter includes engine limits and does not claim source masking;
• cross-account producer, RAM, consumer, link, KMS, and local grants are complete;
• administrator, direct-S3, old-result, and downstream-copy bypasses are covered;
• diagnosis identifies the failed gate before permissions change;
• cost and quotas include analytics, storage, keys, network, and audit services;
• mutable changes have exact rollback and behavioral verification; and
• a reviewer can reproduce each access decision from evidence.

Official sources

• What is AWS Lake Formation?
• Lake Formation terminology
• Permissions overview
• Permissions reference
• Set up Lake Formation
• Hybrid access mode
• Manage LF-tags
• LF-tag-based grants
• Data filtering limitations
• Use Lake Formation with Athena
• Cross-account version settings
• Hybrid cross-account sharing