AWS Glue crawlers support cross-account crawling to support data mesh architecture

Data lakes have come a long way, and there’s been tremendous innovation in this space. Today’s modern data lakes are cloud native, work with multiple data types, and make this data easily available to diverse stakeholders across the business. As time has gone by, data lakes have grown significantly and have evolved to data meshes as a way to scale. Thoughtworks defines a data mesh as “a shift in a modern distributed architecture that applies platform thinking to create self-serve data infrastructure, treating data as the product.”

Data mesh advocates for decentralized ownership and delivery of enterprise data management systems that benefit several personas. Data producers can use the data mesh platform to create datasets and share them across business teams to ensure data availability, reliability, and interoperability across functions and data subject areas. Data consumers now have better data sharing with data mesh and federation across business units without compromising data security. The data governance team can support distributed data, where all data is accessible to those with the proper authority to access it. With data mesh, data doesn’t have to be consolidated into a single data lake or account and can remain within different databases and data lakes. An essential capability needed in such a data lake architecture is the ability to continuously understand changes in the data lakes in various other domains and make those available to data consumers. Without such a capability, manual work is needed to understand producers’ updates and make them available to consumers and governance.

AWS customers use a modern data architecture to facilitate governance and data sharing across logical or physical governance boundaries to create data domains aligned to lines of business. Each line of business creates and manages their dataset on Amazon Simple Storage Service (Amazon S3) and uses AWS Glue crawlers to discover new datasets and register them to the AWS Glue Data Catalog, add new tables and partitions, and detect schema changes. These datasets are shared with data consumers that access the data using services like Amazon Athena, Amazon Redshift, Amazon EMR, and more.

In the post Introducing AWS Glue crawlers using AWS Lake Formation permission management, we introduced a new set of capabilities in AWS Glue crawlers and AWS Lake Formation that simplifies crawler setup and supports centralized permissions for in-account and cross-account crawling of S3 data lakes. In this post, we demonstrate the same capability for a data mesh architecture in which we establish a central governance layer to catalog the data owned by the data producer and share it with the data consumer for ease of discovery. The AWS Glue crawler cross-account capability allows you to crawl data sources in different producer accounts while still having those changes cataloged in a centralized governance account. Customers prefer the central governance experience over writing bucket policies separately in each bucket owning the account of a data mesh producer. To build a data mesh architecture, now you can author permissions in a single Lake Formation governance to manage access to data locations and crawlers spanning multiple accounts in the data mesh.

In this post, we walk through the creation of a simplified data mesh architecture that shows how to use an AWS Glue crawler with Lake Formation to automate bringing changes from data producer domains to data consumers while maintaining centralized governance.

Solution overview

In a data mesh architecture, you have several producer accounts that own S3 buckets, several consumer accounts who wants to access shared datasets, and a central governance account to manage data shares between producers and consumers. This central governance account doesn’t own any S3 bucket or actual tables.

The following figure shows a simplified data mesh architecture with a single producer account, a centralized governance account, and a single consumer account. The data mesh producer account hosts the encrypted S3 bucket, which is shared with the central governance account. The central governance account registers the S3 bucket with Lake Formation using an AWS Identity and Access Management (IAM) role, which has permissions to the S3 bucket and AWS Key Management Service (AWS KMS). The central account creates the database for storing the dataset schema and shares it with the producer account. The producer account, as the S3 bucket owner, runs a crawler to crawl the buckets registered with the central account using Lake Formation permissions and populates the database. Now the shared database with new datasets are available to share with consumers in the data mesh. The central governance account can now share the database with a consumer admin, who can delegate access to other personas (such as data analysts) in the consumer account for data access.

In the following sections, we provide AWS CloudFormation templates to set up the resources in each account. Then we provide the steps to configure the crawler, manage permissions and sharing, and validate the solution by running queries with Athena.

Prerequisites

Complete the following steps in each account (producer, central governance, and consumer) to update the Data Catalog settings to use Lake Formation permissions to control catalog resources instead of IAM-based access control:

1. Sign in to the Lake Formation console as admin.
2. If this is the first time accessing the Lake Formation console, add yourself as the data lake administrator.
   
3. In the navigation pane, under Data catalog, choose Settings.
4. Uncheck Use only IAM access control for new databases.
5. Uncheck Use only IAM access control for new tables in new databases.
6. Keep Version 3 as the current cross-account version.
7. Choose Save.

Set up resources in the central governance account

The CloudFormation template for the central account creates a CentralDataMeshOwner user assigned as Lake Formation admin. The CentralDataMeshOwner user in the central governance account performs the necessary steps to share the central catalogs with the producer and consumer accounts. The CentralDataMeshOwner user also sets up a custom Lake Formation service role to register the S3 data lake location. Complete the following steps:

1. Log in to the central governance account console as IAM administrator.
2. Choose Launch Stack to deploy the CloudFormation template:
   
3. For DataMeshOwnerUserName, keep the default (CentralDataMeshOwner).
4. For ProducerAWSAccount, enter the producer account ID.
5. Create the stack.
6. After the stack launches, on the AWS CloudFormation console, navigate to the Resources tab of the stack.
7. Note down the value of RegisterLocationServiceRole.
8. Choose the LFUsersPassword value to navigate to the AWS Secrets Manager console.
9. In the Secret value section, choose Retrieve secret value.
10. Note down the secret value for the password for IAM user CentralDataMeshOwner.

Set up resources in the producer account

The CloudFormation template for the producer account creates the following resources:

• IAM user LOBProducerSteward
• S3 bucket retail-datalake-<producer account id >-<producer region>
• KMS key used for bucket encryption
• Required S3 bucket policies to provide access to the central governance account
• AWS Glue crawler and crawler IAM role with necessary permissions

Complete the following steps:

1. Log in to the producer account console as IAM administrator.
2. Choose Launch Stack to deploy the CloudFormation template:
   
3. For CentralAccountID, enter the central account ID.
4. For CentralAccountLFServiceRole, enter the value of RegisterLocationServiceRole from CloudFormation noted earlier.
5. Create the stack.
6. When the stack is complete, on the AWS CloudFormation console, navigate to the Resources tab of the stack.
7. Note down the AWSGlueServiceRole value.
8. Choose the ProducerStewardUserCredentials value to navigate to the Secrets Manager console.
9. In the Secret value section, choose Retrieve secret value.
10. Note down the secret value for the password for IAM user LOBProducerSteward.
11. On the Amazon S3 console, check the bucket policies for retail-datalake-<producer account id >-<producer region> and make sure it is shared with the central governance account IAM role.

This is required for registering the bucket with Lake Formation in the central account so that the account can manage the data sharing.

12. On the AWS KMS console, check that the bucket is encrypted with the customer managed key and the key is shared with the central governance account.

Set up resources in the consumer account

The CloudFormation template for the consumer account creates the following resources:

• IAM user ConsumerAdminUser assigned to the data lake admin
• IAM user LFBusinessAnalyst1
• S3 bucket for Athena output
• Athena workgroup

Complete the following steps:

1. Log in to the consumer account console as IAM administrator.
2. Choose Launch Stack to deploy the CloudFormation template:
   
3. Create the stack.
4. When the stack is complete, on the AWS CloudFormation console, navigate to the Resources tab of the stack.
5. Choose the AllConsumerUsersCredentials value to navigate to the Secrets Manager console.
6. In the Secret value section, choose Retrieve secret value.
7. Note down the secret value for the password for the IAM user ConsumerAdminUser.

Now that all the accounts have been set up, we set up cross-account sharing on AWS with a central governance account to manage sharing of permissions across producers and consumers.

Configure the central governance account to manage sharing with the producer account

Sign in to the central governance account as CentralDataMeshOwner using the password noted earlier through the central governance account CloudFormation stack. Then complete the following steps:

1. On Lake Formation console, choose Data lake locations under Register and ingest in the navigation pane.
2. For Amazon S3 path, provide the path retail-datalake-<producer account id >-<region>.
3. For IAM role, choose the IAM role created using the CloudFormation stack.

This role has permissions for the accessing the encrypted S3 bucket and its key. Do not choose the role AWSServiceRoleForLakeFormationDataAccess.

4. Choose Register location.
   
5. In the navigation pane, choose Databases.
6. Choose Create database.
7. For Database name¸ enter datameshtestdatabase.
8. Choose Create database.
9. In the navigation pane, choose Data locations and choose Grant.
   
10. Select External account and provide the producer account for AWS account ID, AWS organization ID, or IAM principal ARN.
11. For Storage location, provide the data lake bucket path.
12. Select Grantable, then choose Grant.
13. Choose Data lake permissions, then choose Grant.
14. Select External accounts and provide the producer account number.
15. For Databases, choose datameshtestdatabase.
16. For Database permissions and Grantable permissions, select Create table, Alter, and Describe.
17. Choose Grant.
    
    

Configure the crawler in the producer account to populate the schema

Sign in to producer account as LOBProducerSteward with the password noted earlier through the producer account CloudFormation stack, then complete the following steps:

1. On the AWS RAM console, accept the pending resource share from the central account.
2. On the Lake Formation console, choose Databases under Data catalog in the navigation pane.
3. Choose datameshtestdatabase, and on the Action menu, choose Create resource link.
4. For Resource link name, enter datameshtestdatabaselink.
5. Choose Create.
   
6. On the AWS Glue console, choose Crawlers in the navigation pane.
7. Choose the crawler CrossAccountCrawler-<accountid>.
8. Choose Edit, then choose Configure security settings.
9. Select Use Lake Formation credentials for crawling S3 data source.
10. Select In a different account and provide the account ID of the central governance account.
11. Choose Next.
    
12. Choose datameshtestdatabaselink as the database and choose Update.
    
13. In the navigation pane, choose Data locations and choose Grant.
14. Select My account, and choose the crawler IAM role for IAM users and roles.
15. For Storage locations, choose the bucket retail-datalake-<accountid>-<region>.
16. For Registered account location, enter the central account ID.
17. Choose Grant.
    Alternatively, you can also use the AWS CLI to grant data location permission on bucket registered in central account to the crawler role using below command:

    aws lakeformation grant-permissions 
    --principal DataLakePrincipalIdentifier="<Crawler Role ARN>" 
    --permissions "DATA_LOCATION_ACCESS” 
    --resource ‘{ "DataLocation": {"ResourceArn":"<S3 bucket arn>", "CatalogId": "<Central Account id>"}}'

    For using CLI, refer to Installing or updating the latest version of the AWS CLI.

18. In the navigation pane, choose Data lake permissions.
19. Choose the crawler IAM role for the principal account.
20. Choose datameshtestdatabase for the database.
21. For Database permissions, select Create, Describe, and Alter.
22. Choose Grant.
    
23. Choose the crawler IAM role for the principal account.
24. Choose datameshtestdatabaselink for the database.
25. For Resource link permissions, select Describe.
26. Choose Grant.
27. Run the crawler.

The following screenshot shows the details after a successful run.

When the crawler is complete, you can validate the table created under the database datameshtestdatabaselink.

This table is owned by the producer account and available in the central governance account under the shared database datameshtestdatabase. Now the data lake admin in the central governance account can share the database and populated table with the consumer account.

Configure the central governance account to manage sharing of read-only access with the consumer account

Sign in to the central governance account as CentralDataMeshOwner with the password noted earlier through the central governance account CloudFormation stack, then complete the following steps:

1. Grant database permissions to the consumer account.
2. For Principals, choose external account and provide <consumer accountID>
3. For Databases, select datameshtestdatabase.
4. For Database permissions, select Describe.
5. For Grantable permissions¸ select Describe.
6. Choose Grant.
   
   
7. Grant table permissions to the consumer account.
8. For Principals, choose external account and provide <consumer accountID>.
9. For Databases, select datameshtestdatabase.
10. For Tables, select retail_datalake_<accountID>_<region>.
11. For Table permissions, select Select and Describe.
12. For Grantable permissions¸ select Select and Describe.
13. Choose Grant.
    
    

Configure the consumer account as the consumer account data lake admin

Sign to the consumer account as ConsumerAdminUser with the password noted earlier through the consumer account CloudFormation stack. (Note that in the consumer account Lake Formation configuration, both ConsumerAdminUser and LFBusinessAnalyst1 have the same password.)

1. On the AWS RAM console, accept the resource share from the central account.
2. On the Lake Formation console, validate that the shared database datameshtestdatabase is available and create the resource link datameshtestdatabaselink using the shared database.

The following screenshot shows the details after the resource link is created.

3. On the Lake Formation console, choose Grant.
4. Choose LFBusinessAnalyst1 for IAM users and roles.
5. Choose datameshtestdatabase for the database under Named data catalog resources.
6. Select Describe for Database permissions.
7. On the Lake Formation console, choose Grant.
8. Choose LFBusinessAnalyst1 for IAM users and roles.
9. Choose datameshtestdatabaselink for the database under Named data catalog resources.
10. Select Describe for Resource link permissions.
11. On the Lake Formation console, choose Grant.
12. Choose LFBusinessAnalyst1 for IAM users and roles.
13. Choose retail_datalake_<accountid>_<region> for the table under Named data catalog resources.
14. Select Select and Describe for Table permissions.

Run queries in the consumer account

Sign to the consumer account console as LFBusinessAnalyst1 with the password noted earlier through the consumer account CloudFormation stack, then complete the following steps:

1. On the Athena console, and choose lfconsumer-workgroup as the Athena workgroup.
2. Run the following query to validate access:

select * from datameshtestdatabaselink.retail_datalake_<accountid>_<region>

We have successfully registered the dataset and created a Data Catalog in the central governance account. We crawled the data lake that was registered with the central governance account using Lake Formation permissions from the producer account and populated the schema. We granted Lake Formation permission on the database and table from the central account to the consumer user and validated consumer user access to the data using Athena.

Clean up

To avoid unwanted charges to your AWS account, delete the AWS resources:

1. Sign in to the CloudFormation console as the IAM admin used for creating the CloudFormation stack in all three accounts.
2. Delete the stacks you created.

Conclusion

In this post, we showed how to set up cross-account crawling using a central governance account with the new AWS Glue crawler capability of Lake Formation integration. This capability allows data producers to set up crawling capabilities in their own domain so that changes are seamlessly available to data governance and data consumers. Implementing a data mesh with AWS Glue crawlers, Lake Formation, Athena, and other analytical services provide a well-understood, performant, scalable, and cost-effective solution to integrate, prepare, and serve data.

If you have questions or suggestions, submit them in the comments section.

For more resources, refer to the following:

• Design a data mesh architecture using AWS Lake Formation and AWS Glue
• Introducing AWS Glue crawlers using AWS Lake Formation permission management

About the authors

Sandeep Adwankar is a Senior Technical Product Manager at AWS. Based in the California Bay Area, he works with customers around the globe to translate business and technical requirements into products that enable customers to improve how they manage, secure, and access data.

Srividya Parthasarathy is a Senior Big Data Architect on the AWS Lake Formation team. She enjoys building data mesh solutions and sharing them with the community.

Piyali Kamra is a seasoned enterprise architect and a hands-on technologist who believes that building large scale enterprise systems is not an exact science but more like an art, in which tools and technologies must be carefully selected based on the team’s culture , strengths , weaknesses and risks , in tandem with having a futuristic vision as to how you want to shape your product a few years down the road.