AWS Cloud Operations Blog
Exploring AWS Config data using Amazon Athena and Amazon Managed Grafana
This post is co-written with Jacob Rickerd, Principal Security Engineer at Attentive.
The post walks through an example dashboard that Attentive, an AI-powered mobile marketing platform, uses for resource inventory, serving as a starting point for you to build comprehensive dashboards tailored to your environment and tag policies.
Attentive is the AI-powered SMS and email marketing platform that delivers unparalleled messaging performance on the channels that matter most. Infusing intelligence at every stage of the consumer’s purchasing journey, Attentive empowers businesses to achieve hyper-personalized communication with their customers on a large scale. Leveraging AI-powered tools, a mobile-first approach, two-way conversations, and enterprise-grade technology, Attentive drives billions in online revenue for brands around the globe.
Attentive’s growing cloud infrastructure demanded a comprehensive visibility solution to track AWS resource configurations and reduce security risks. By integrating AWS Config for continuous monitoring, Amazon Athena for querying configuration snapshots, and Amazon Managed Grafana for visualization, we built a powerful system that allowed us to track misconfigurations, drive security initiatives, and ensure compliance. Customizations like partitioning data using an AWS Lambda function ensured the solution scaled effectively, providing actionable insights and helping us efficiently take action on resources across our environment. – Jacob Rickerd, Principal Security Engineer, Attentive
In this blog post, we describe how to create comprehensive Amazon Managed Grafana dashboards for resource inventory across your AWS environment based on AWS Config data. This helps you understand inventory of AWS resources and changes to those resources across your single or multi-account AWS environment. By leveraging AWS Config data and tag policies, you gain visibility into your AWS resources in a central platform, enabling better organization, cost allocation, and help comply with industry regulations.
AWS Config is a service that continuously monitors and records the configurations of AWS resources. Amazon Managed Grafana is a fully managed service for creating dashboards and visualizations using various data sources, including AWS Config data through its built-in Amazon Athena integration.
Combining AWS Config and Amazon Managed Grafana allows you to create dashboards that provide comprehensive insights into AWS resource configurations, historical changes, and compliance status. This combination empowers you to monitor resource inventory, identify misconfigurations, and streamline auditing processes within your AWS environment. By using tag policies to standardize resource tagging, you can maintain a structured and searchable inventory, enabling accurate resource identification, cost allocation, and effective access control.
Solution overview
AWS Config delivers configuration snapshots to an Amazon Simple Storage Service (Amazon S3) bucket on a periodic basis specified when configuring the delivery channel. Delivery to an S3 bucket can be set to every 1, 3, 6, 12 or 24 hours. Those snapshots contain configuration history and inventory. This allows accessing complete configuration history for auditing, troubleshooting, and compliance purposes.
Amazon Athena is used to query data stored in S3 buckets. In order to query data stored in an S3 bucket, it is necessary to define a specific schema that supports AWS Config data. This schema can be made available and queryable through Amazon Athena by using an AWS Glue crawler. Once the schema is defined, an external table is created in Amazon Athena that points to the data location in S3. This external table allows Amazon Athena to query data in S3 using standard SQL. Amazon Athena uses the schema metadata from AWS Glue to understand the data structure. Once we have the schema available, it is possible to run SQL queries on this external table to analyze data in S3 without the need to copy it elsewhere.
Partitioning in Amazon Athena serves the purpose of enabling efficient querying by reducing the amount of data scanned. It allows to target specific data subsets based on partition keys, thus optimizing query performance. An AWS Lambda function is invoked when a new configuration snapshot is added to the S3 bucket. This function partitions the data by configuring keys with values that correspond to the region and date of each snapshot file.
In Amazon Athena, views are instrumental in building dashboards within Amazon Managed Grafana, offering users insights and visual representations derived from AWS Config data. An Amazon Athena view is a logical table, not a physical one. Its defining query executes whenever the view is accessed in a query. Views are generated from SELECT queries and can be referenced in subsequent queries.
The defined architecture, as outlined in Figure 1, aims to efficiently organize and query AWS Config data, enabling seamless data exploration and visualization using Amazon Managed Grafana dashboards.
Figure 1: AWS Config data exploration and visualization with Amazon Managed Grafana
Setting up the environment for AWS Config data visualization in Amazon Managed Grafana
This post focuses on creating Amazon Managed Grafana dashboards with AWS Config data. It does not provide detailed instructions for setting up the environment needed for Amazon Managed Grafana dashboard creation.
Prerequisites
- Enable AWS Config and set up an S3 bucket for configuration snapshot delivery of your account(s)
- For a single account, follow the steps in Getting Started with AWS Config
- For a multi-account setup, you have two main options for organization-wide configuration management. You can set up an AWS Config Aggregator using a delegated administrator account, allowing centralized control across your AWS Organization with flexibility in account designation. Alternatively, if using AWS Control Tower, follow its setup guide, which automatically creates an AWS Config Aggregator during landing zone deployment.
- Set up an Amazon Athena workgroup, following the Amazon Managed Grafana prerequisites
- Create a database with a table to support AWS Config data
- Create views to build dashboards within Amazon Managed Grafana
- Create an AWS Lambda function that invokes when new snapshots arrive in S3 and creates partitions in those snapshots
- Create a workspace in Amazon Managed Grafana
- Ensure Turn plugin management on is checked to install, update, or remove plugins using the Grafana plugin catalog
- Check Amazon Athena in the Data Sources section
- Install the Amazon Athena AWS Data Source by navigating to Apps from the Toggle menu and selecting AWS Data Source
- Configure the Amazon Athena data source and install any additional plugins needed for your dashboards
For more detailed instructions on setting up the environment for AWS Config data, please refer to the following posts: How to query your AWS resource configuration states using AWS Config and Amazon Athena, Visualizing AWS Config data using Amazon Athena and Amazon QuickSight, and Automate AWS Config data visualization with AWS Systems Manager.
Create dashboards with Amazon Managed Grafana
With the environment configured as expected, you can proceed to create an Amazon Managed Grafana dashboard that provides an accurate inventory of your resources. In this post, Attentive demonstrates inventory collection of AWS IAM resources, using data retrieved from AWS Config, as outlined in Figure 2. This includes detailed lists of IAM Roles grouped according to various parameters. To accomplish this, follow these steps:
1. After logging in to the Amazon Managed Grafana navigation pane, select the Toggle menu in the bottom left corner, then choose Dashboards.
2. On the Dashboards page, click New and select New Dashboard.
3. You are now ready to create the first panel; choose Add a new panel.
4. Choose the previously created Amazon Athena data source, and select the desired database (awsconfig) and table/view (aws_config_configuration_snapshot).
5. In the query pane, enter the following query:
SELECT resourcename AS Role,
accountid AS Account
FROM awsconfig.aws_config_configuration_snapshot
WHERE dt = 'latest'
AND resourcetype = 'AWS::IAM::Role'
ORDER BY role ASC;6. After obtaining results, select Table as the visualization and define desired options. By doing this, you have created the first panel.
7. Continue creating new panels on the dashboard. In the toolbar, you will see an Add panel icon. After clicking there, choose Add a new panel.
8. You can proceed with creating new panels. To display the number of IAM Roles, use Stat visualization and enter the following query:
SELECT COUNT(*) AS Total
FROM awsconfig.aws_config_configuration_snapshot
WHERE dt = 'latest'
AND resourcetype = 'AWS::IAM::Role';9. To visualize IAM Roles vs IAM Users, use the Pie chart visualization and enter the following query:
SELECT resourcetype,
COUNT(resourcetype)
FROM awsconfig.aws_config_configuration_snapshot
WHERE dt = 'latest'
AND (resourcetype = 'AWS::IAM::Role'
OR resourcetype = 'AWS::IAM::User')
GROUP BY resourcetype;10. To visualize IAM Roles by the “Team” tag, use the Bar chart visualization, and enter the following query:
SELECT element_at(tags, 'Team') AS Team,
COUNT(*) AS Total
FROM awsconfig.aws_config_configuration_snapshot
WHERE dt = 'latest'
AND resourcetype = 'AWS::IAM::Role'
GROUP BY element_at(tags, 'Team')
ORDER BY total DESC;11. A more advanced visualization may be to show all roles that have an inline policy which grants full access to S3. This can be useful if you are tracking an initiative to remove a specific permission from roles within your accounts:
SELECT resourcename,
accountid
FROM awsconfig.aws_config_configuration_snapshot
CROSS JOIN unnest(cast(json_extract(configuration, '$.rolePolicyList') AS array(json))) AS t(policy)
WHERE dt = 'latest'
AND resourcetype = 'AWS::IAM::Role'
AND url_decode(json_extract_scalar(policy, '$.policyDocument')) LIKE '%s3:*%';
Figure 2: Example AWS IAM inventory
By following the steps outlined in the previous section, you can create multiple dashboards using additional resources to enhance visibility into your environment. One such dashboard could be called the AWS Config compliance dashboard, which offers a centralized perspective on the compliance status of your AWS resources against defined rules and configurations, as outlined in Figure 3. This dashboard facilitates continuous monitoring, reporting, and management of resource compliance, enabling you to uphold desired configurations and best practices throughout your AWS environment.
Figure 3: Example AWS Config compliance dashboard
The post Use AWS Config inventory and compliance dashboards for a unified view of resource inventory and compliance introduces inventory and compliance dashboards for AWS Config aggregators. These dashboards offer a unified view of resource configurations and compliance status across AWS accounts, regions, or an organization, allowing you to easily assess resource optimization, security, audit, and compliance requirements.
The resource inventory and compliance dashboards provide significant business value by offering a unified view of resource configurations and compliance status across multiple AWS accounts, regions, or an entire organization. In daily operations, customers like Attentive use these dashboards to track misconfigurations, drive security initiatives, and ensure compliance at scale. For instance, security teams can quickly identify IAM roles with overly permissive policies, such as those granting full S3 access, allowing them to take immediate action to reduce risk. Infrastructure teams can monitor resource allocation and utilization trends, optimizing costs and capacity. Compliance officers can generate on-demand reports for audits, saving time and ensuring accuracy. By centralizing this critical information, organizations can proactively manage their AWS environment, make data-driven decisions, and maintain a strong security posture, ultimately leading to improved operational efficiency and reduced risk.
For more information on how to customize Grafana visualizations, refer to Visualization panels.
Cleanup
Upon completing the steps demonstrated in this post within your AWS account, it is advisable to clean up any created resources to avoid incurring unnecessary charges. This includes disabling AWS Config if you enabled it solely for the purpose of this blog post, and deleting the snapshot stored in the S3 bucket. Additionally, you should delete the Amazon Athena workgroup created specifically for Amazon Managed Grafana, as well as any views, tables, and databases created for it. Don’t forget to delete the AWS Lambda function you used to create partitions. Lastly, remember to delete the Amazon Managed Grafana workspace itself.
Conclusion
In this post, we explored building Amazon Managed Grafana dashboards for resource inventory and compliance monitoring within your AWS environment. Companies of any size operating in a single-account or multi-account AWS setup can benefit from these dashboards, providing a comprehensive resource overview crucial for teams such as infrastructure, security, and finance. By the end, you’ll have newfound visibility into your AWS resource inventory across accounts, enabling governance, security, and cost optimization efforts.
Explore the capabilities of Amazon Managed Grafana further by creating additional dashboards tailored to your needs, using your own data sources or publicly available datasets.