Why does my AWS Glue crawler or ETL job fail with the error "Insufficient Lake Formation permission(s)"?

Last updated: 2022-02-15

My AWS Glue Crawler or ETL job fails with the error "Insufficient Lake Formation permission(s)". However, the AWS Identity and Access Management (IAM) role associated with the AWS Glue crawler or AWS Glue ETL job has the necessary IAM permissions.

Short description

Access to AWS Glue Data Catalog and Amazon Simple Storage Service (Amazon S3) resources are managed not only with IAM policies, but also with AWS Lake Formation permissions. You get the Insufficient Lake Formation permission(s) error when the IAM role associated with the AWS Glue crawler or ETL job doesn't have the necessary Lake Formation permissions to read and/or write from/to the following:

  • Database/table in the Data Catalog
  • Underlying data in Amazon S3

Resolution

Access issues with Data Catalog database

If the error is caused due to access issues related to the Data Catalog database, then the error message looks similar to the following:

ERROR : Insufficient Lake Formation permission(s) on example-database (Database name: example-database)

To resolve this error, sign in to the AWS Lake Formation console with data lake administrator role, and then grant the Create database permission to the IAM role that's associated with the crawler/ETL job:

  1. Open the AWS Lake Formation console.
  2. In the navigation pane, under Permissions, choose Administrative roles and tasks.
  3. Under Database creators, choose Grant.
  4. For IAM users and roles, from the dropdown list, select the IAM role for which you want to grant the access.
  5. Under Catalog permissions, select Create database.
  6. If you need the IAM role to grant permissions to other roles in your account, select Create database under Grantable permissions.
  7. Choose Grant.

Access issues with Data Catalog table

If the error is caused due to access issues related to the Data Catalog table, then the error message looks similar to the following:

ERROR : Insufficient Lake Formation permission(s) on example-table (Database name: example-database, Table Name: example-table)

To resolve this error, grant the Create table permission for example-database to the IAM role that's associated with the crawler/ETL job:

  1. Open the AWS Lake Formation console.
  2. In the navigation pane, under Permissions, choose Data lake permissions.
  3. Choose Grant.
  4. Under Principals, select IAM users and roles.
  5. For IAM users and roles, select the IAM role.
  6. Under LF-Tags or catalog resources, select Named data catalog resources.
  7. For Databases, select the database.
  8. Under Database permissions, select Create table or Super based on your use case.
  9. If you need the IAM role to grant permissions to other roles in your account, select the required permissions under Grantable permissions.
  10. Choose Grant.

Access issues with Amazon S3 path

If the error is caused due to access issues related to the Amazon S3 path, the error message includes the S3 path and looks similar to the following:

ERROR : Insufficient Lake Formation permission(s) on s3://S3-example-bucket/example-prefix/ (Database name: example-database, Table Name: example-table)

This means that the IAM role that's associated with the crawler/ETL job doesn't have the required permission to access the S3 path.

To resolve this error, do the following:

  1. Open the AWS Lake Formation console.
  2. In the navigation pane, under Register and ingest, choose Data lake locations.
    You can view the data lake locations.
  3. Verify that the Amazon S3 path or prefix of the S3 path mentioned in the error message is a registered location in the Data lake locations list.
  4. If the S3 path or prefix in the error message is different from that listed in the Data lake locations list, then choose Register location.
  5. For Amazon S3 path, choose Browse, and select the correct S3 path.
  6. For IAM role, leave the default selection AWSServiceRoleForLakeFormationDataAccess. If you need to use a custom IAM role, then be sure that the relevant requirements are met.
    Important: When you register an S3 location, Lake Formation assumes the preceding IAM role to grant temporary credentials to integrated AWS services that access data in that location. Therefore, be sure that the IAM role that's associated with the crawler/ETL job has the required permissions to read/write to the S3 bucket to prevent the AccessDenied error.
  7. In the navigation pane, under Permissions, choose Data locations.
  8. Choose Grant.
  9. For Grant permissions, select My account.
  10. Under IAM users and roles, select the IAM role for which you want to grant access.
  11. Choose Grant.

Did this article help?


Do you need billing or technical support?