Manage fine-grained access control using AWS Lake Formation

AWS Lake Formation is a fully managed service that helps you build, secure, and manage data lakes, and provide access control for data in the data lake. Customers across lines of business (LOBs) need a way to manage granular access permissions for different users at the table and column level. Lake Formation helps you manage fine-grained access for internal and external customers from a centralized location and in a scalable way.

In this post, we describe an approach to manage granular permissions on datasets shared between AWS accounts using Lake Formation.

Solution overview

Our use case assumes you’re using AWS Organizations to manage your AWS accounts. The user of Account A in one organizational unit (OU1) grants access to users of Account B in OU2. You can use this same approach when not using Organizations, such as when you only have a few accounts.

The following diagram illustrates the fine-grained access control of datasets in the data lake. The data lake is available in the Account A. The data lake administrator of Account A provides fine-grained access for Account B. The diagram also shows that a user of Account B provides column-level access of the Account A data lake table to another user in Account B.

Prerequisites

You need the following resources for this walkthrough:

• Two organizational units:
  • OU1 – Contains Account A
  • OU2 – Contains Account B
• An Amazon Simple Storage Service (Amazon S3) data lake location (bucket) in Account A.
• A data lake administrator user in Account A. You can create a data lake administrator using the Lake Formation console or the PutDataLakeSettings operation of the Lake Formation API.
• Lake Formation configured in Account A, and the S3 data lake location registered with Lake Formation in Account A.
• Two users in Account B with the following AWS Identity and Access Management (IAM) managed policies:
  • testuser1 – Has the AWS managed policies AWSLakeFormationDataAdmin attached.
  • testuser2 – Has the AWS managed policy AmazonAthenaFullAccess attached.
• A database testdb in the Lake Formation database for Account B.

Provide fine-grained access to another account

In this section, we demonstrate how a data lake administrator of Account A provides fine-grained access for Account B.

1. 1. Sign in to the AWS Management Console in Account A as the user who is a data lake administrator.
   2. Open the Lake Formation console.
   3. Choose Get started.
      
   4. In the navigation pane, choose Databases.
   5. Choose Create database.
   6. In the Database details section, select Database.
   7. For Name, enter a name (for this post, we use sampledb01).
   8. Make sure that Use only IAM access control for new tables in this database is not selected. Leaving this unselected allows us to control access from Lake Formation.
   9. Choose Create database.
      
   10. On the Databases page, choose your database sampledb01.
   11. On the Actions menu, choose Grant.
       
   12. In the Grant permissions section, select External account.
   13. For AWS account ID or AWS organization ID, enter the account ID for Account B in OU2.
       
   14. For Table, choose the table you want Account B to have access to (for this post, we use table acc_a_area). Optionally, you can grant access to columns within the table, which we do in this post.
   15. For Columns, choose Include columns.
   16. For Include columns¸ choose the columns you want Account B to have access to (for this post, we grant permissions to type, name, and identifiers).
   17. For Table permissions, select Select.
   18. For Grantable permissions, select Select. Grantable permissions are required so admin users in Account B can grant permissions to other users in Account B.
   19. Choose Grant.
       
   20. In the navigation pane, choose Tables.
   21. You could see one active connection in the AWS accounts and AWS organizations with access section.
       

   Create a resource link

2. Integrated services like Amazon Athena can’t directly access databases or tables across accounts, hence we will create resource link so that Athena can access resource links in your account to databases and tables in other accounts. We now create a resource link to our table so Account B users can query its data with Amazon Athena.
   1. Sign in to the console in Account B as testuser1.
   2. On the Lake Formation console, in the navigation pane, choose Tables. You should see the tables that Account A has provided access to.
   3. Choose the table acc_a_area.
   4. On the Actions menu, choose Create resource link.
      
   5. For Resource link name, enter a name (for this post, acc_a_area_rl).
   6. For Database, choose your database (testdb).
   7. Choose Create.
      
   8. In the navigation pane, choose Tables.
   9. Choose the table acc_b_area_rl.
   10. On the Actions menu, choose View data.
3. You’re redirected to the Athena console, where you should see the database and table.You can now run a query on the table to see the column value for which access was provided to testuser1 from Account B.
   

   Provide fine-grained access to a user in the same account

   In this section, we demonstrate how a user in Account B (testuser1), acting as a data steward, provides fine-grained access to another user in the same account (testuser2) to the column name in the shared table aac_b_area_rl.

   1. Sign in to the console in Account B as testuser1.
   2. On the Lake Formation console, in the navigation pane, choose Tables.
   3. You can grant permissions on a table through its resource link. To do so, on the Tables page, select the resource link acc_b_area_rl , and on the Actions menu, choose Grant on target. 
   4. In the Grant permissions section, select My account.
   5. For IAM users and roles¸ choose the user testuser2.
   6. For Column, choose the column name.
   7. For Table permissions, select Select.
   8. Choose Grant.
   9. When you create a resource link, only you can view and access it. To permit other users in your account to access the resource link, we need to grant permissions on the resource link itself. We need to grant DESCRIBE or DROP permissions. On the Tables page, select your table again and on the Actions menu, choose Grant.
   10. In the Grant permissions section, select My account.
   11. For IAM users and roles, select the user testuser2.
   12. For Resource link permissions¸ select Describe.
   13. Choose Grant.
   14. Sign in to the console in Account B as testuser2.

   On the Athena console, you should see the database and table acc_b_area_rl.

   You can now run a query on the table to see the column value that testuser2 has access to.

   

   Conclusion

   In this post, we showed how, when managing multiple accounts with Organizations, you can quickly and easily share datasets using Lake Formation. We defined granular permissions to control access to sensitive data. We also showed how a data lake administrator of Account A can provide fine-grained access for Account B, and how a user in Account B, acting as a data steward, can grant fine-grained access to the shared table for other users in their account. Data stewards within each account can independently delegate access to their own users, giving each team or LOB autonomy.

   ---

   About the Authors

   Niyati Upadhyay is a Solutions Architect at AWS. She joined AWS in 2019 and specializes in building and supporting Big Data solutions that help customers analyze and get value out of their data.

    

    

   

   Dipayan Sarkar is a Specialist Solutions Architect for Analytics at AWS, where he helps customers to modernise their data platform using AWS Analytics services. He works with customer to design and build analytics solutions enabling business to make data-driven decisions.

    

    

   Suman Banerjee is a Global Enterprise Solution Architect and and a Builder at heart. He has spent 20+ years helping enterprises to architect and build solutions to achieve their business goals. Architecting solutions for customers is what keeps him motivated. When he is not helping customer, he enjoys playing with his 2 kids Swapnil and Ayushmaan.