AWS Lake Formation provides database, table, column and tag-based access controls, and cross-account sharing at no charge. Governed Tables provide ACID transactions as a fully managed service that enable you to reliably update multiple tables, while maintaining a consistent view for all users. Managing concurrent transactions and being able to travel back to previous version of the table requires storing metadata about each transaction. Lake Formation charges a fee for transaction requests and for metadata storage. In addition, to providing a consistent view of data and enforce row and cell-level security the Lake Formation Storage API scans data in Amazon S3 and applies row and cell filters before returning results to applications. There is a fee for this filtering.
-
Data filtering
-
Transaction metadata storage and requests
-
Storage optimizer
-
Data filtering
-
When querying Governed Tables or tables secured with row and cell-level permissions, AWS Lake Formation Storage API scans and filters the data on behalf of the calling service or application. You are charged for the number of bytes scanned by the Storage API, rounded to the next megabyte, with a 10MB minimum.
Optimizing filtering costs: You can optimize your filtering by storing your data in columnar file formats like Parquet and ORC. The Lake Formation Storage API can leverage predicate pushdown and partition pruning optimizations to only read the data required to satisfy the query and scan less data, resulting in a lower data scanned cost.
-
Transaction metadata storage and requests
-
AWS Lake Formation's Storage API manages metadata associated with Governed Tables that tracks S3 objects, partitions, and transactions. You are charged for the metadata storage and the API requests.
-
Storage optimizer
-
The storage optimizer runs in the background and optimizes the layout of Governed Tables by compacting small files into larger ones to improve query performance in analytics engines. You are charged for the number of bytes processed by the storage optimizer, rounded to the next megabyte.
Pricing examples
Data catalog free tier example: Let’s consider that you store one million tables in your data catalog in a given month and make one million requests to access these tables. You pay $0 because your usage is covered under the data catalog free tier. You can store the first million objects and make one million requests per month for free.
Data catalog example: Now consider that your storage usage remains the same at one million tables per month, but your requests double to two million per month. Let’s say you also use crawlers to find new tables, and they run for 30 minutes and consume two DPUs
Your storage cost is still $0, as the storage for your first million tables is free. Your first million requests are also free. You will be billed for one million requests above the free tier, which is $1. Crawlers are billed at $0.44 per DPU-hour, so you will pay for 2 DPUs * 0.5 hours at $0.44 per DPU-hour, or $0.44. You will have a total monthly bill of $1.44.
ML Transforms example: As with AWS Glue jobs, the cost of running ML Transforms (including FindMatches) on your data will vary based on the size of your data, the content of your data, and the number and types of nodes that you use. In the following example, we used FindMatches to integrate points of interest information from multiple data sources. With a data set size of ~11,000,000 rows (1.6 GB), a size of Label data (examples of true matches or true no-matches) of ~8,000 rows (641 KB), running on 16 instances of type G.2x, you would have a labelset generation runtime of 34 minutes at a cost of $8.23, a metrics estimation runtime of 11 minutes at a cost of $2.66, and a FindingMatches job runtime of 32 minutes at a cost of $7.75
View the Global Regions table to learn more about AWS Lake Formation availability.
Additional pricing resources
Easily calculate your monthly costs with AWS
Contact AWS specialists to get a personalized quote
Find resources on AWS Lake Formation such as documentation, videos, and articles.
Start building with AWS Lake Formation in the AWS Management Console.