[SEO Subhead]
This Guidance demonstrates how to deploy Cloud Intelligence Dashboards in your AWS environment using AWS CloudFormation templates or command line tools. These pre-built dashboards enable you to drive financial accountability, optimize costs, and track usage goals across their AWS infrastructure. The Guidance also implements governance best practices and supports operational excellence through automated deployment, secure data handling, and efficient serverless architecture. By providing comprehensive visibility into cost and usage patterns, these dashboards can help you make informed decisions about your AWS resources while maintaining security and performance standards. These pre-built dashboards help customers drive financial accountability, optimize cost, track usage goals, implement best-practices for governance, and achieve operational excellence across all Well Architected pillars.
Please note: [Disclaimer]
Architecture Diagram
-
Foundational dashboards: deployment
-
Foundational dashboards: architecture
-
Advanced dashboards: deployment
-
Advanced dashboards: architecture
-
Foundational dashboards: deployment
-
This architecture diagram shows how to set up the foundation for cloud observability with Cloud Intelligence Dashboards.
Click to enlarge
Step 1
Deploy the AWS CloudFormation stack for AWS Data Exports to the Data Collection AWS account.Step 2
Deploy the AWS Data Exports CloudFormation stack to the Management (Payer) AWS accounts.Step 3
Deploy the Cloud Intelligence Dashboards CloudFormation stack to the Data Collection AWS account.Step 1
Deploy the AWS CloudFormation stack for AWS Data Exports to the Data Collection AWS account.Step 2
Deploy the AWS Data Exports CloudFormation stack to the Management (Payer) AWS accounts.Step 3
Deploy the Cloud Intelligence Dashboards CloudFormation stack to the Data Collection AWS account. -
Foundational dashboards: architecture
-
This architecture diagram shows the connection flow for foundational Cloud Intelligence Dashboards.
Click to enlarge
Step 1
AWS Data Exports delivers the AWS Cost & Usage Report (AWS CUR)—specifically CUR 2.0—daily to an Amazon Simple Storage Service (Amazon S3) bucket in the Management (Payer) Account.Step 2
An Amazon S3 replication rule copies export data from the S3 bucket in a Management (Payer) Account to the S3 bucket in the dedicated Data Collection Account automatically.Step 3
Amazon Athena allows querying data directly from the aggregated S3 bucket using an AWS Glue table schema definition.Step 4
Amazon QuickSight creates datasets from Athena, refreshes daily and caches in SPICE (Super-fast, Parallel, In-memory Calculation Engine) for QuickSight.Step 5
User teams (such as executives, FinOps, and engineers) can access Cloud Intelligence Dashboards in QuickSight. Access is secured through AWS Identity and Access Management (IAM), AWS IAM Identity Center, and optional row-level security.Step 1
AWS Data Exports delivers the AWS Cost & Usage Report (AWS CUR)—specifically CUR 2.0—daily to an Amazon Simple Storage Service (Amazon S3) bucket in the Management (Payer) Account.Step 2
An Amazon S3 replication rule copies export data from the S3 bucket in a Management (Payer) Account to the S3 bucket in the dedicated Data Collection Account automatically.Step 3
Amazon Athena allows querying data directly from the aggregated S3 bucket using an AWS Glue table schema definition.Step 4
Amazon QuickSight creates datasets from Athena, refreshes daily and caches in SPICE (Super-fast, Parallel, In-memory Calculation Engine) for QuickSight.Step 5
User teams (such as executives, FinOps, and engineers) can access Cloud Intelligence Dashboards in QuickSight. Access is secured through AWS Identity and Access Management (IAM), AWS IAM Identity Center, and optional row-level security. -
Advanced dashboards: deployment
-
This architecture diagram shows the setup for the foundational and advanced Cloud Intelligence Dashboards.
Click to enlarge
Step 1
Deploy the CloudFormation stack for AWS Data Exports to the Data Collection AWS account.
Step 2
Deploy the AWS Data Exports CloudFormation stack to the Management (Payer) AWS account(s).
Step 3
Deploy the Cloud Intelligence Dashboards CloudFormation stack to the Data Collection AWS account.
Step 4
Deploy the Advanced Data Collection Permissions CloudFormation stack to the Management (Payer) AWS account(s).
Step 5
The Permissions CloudFormation stack in the Management (Payer) Account also deploys Permissions stacks to each of Linked accounts using StackSets.Step 6
Deploy the Data Collection Stack to the Data Collection AWS account.
Step 7
Deploy the Advanced Dashboards using the Dashboard Plugin CloudFormation stack to the Data Collection AWS Account.
Step 1
Deploy the CloudFormation stack for AWS Data Exports to the Data Collection AWS account.
Step 2
Deploy the AWS Data Exports CloudFormation stack to the Management (Payer) AWS account(s).
Step 3
Deploy the Cloud Intelligence Dashboards CloudFormation stack to the Data Collection AWS account.
Step 4
Deploy the Advanced Data Collection Permissions CloudFormation stack to the Management (Payer) AWS account(s).
Step 5
The Permissions CloudFormation stack in the Management (Payer) Account also deploys Permissions stacks to each of Linked accounts using StackSets.Step 6
Deploy the Data Collection Stack to the Data Collection AWS account.
Step 7
Deploy the Advanced Dashboards using the Dashboard Plugin CloudFormation stack to the Data Collection AWS Account.
Step 8
After the required wait period, a Lambda function evaluates delete requests and removes only unused objects from Amazon S3.
Step 9
Events from core API functions are sent to Amazon EventBridge for subsequent reuse by other systems.Step 10
An optional Lambda function can process webhook requests to external systems according to the specification.
-
Advanced dashboards: architecture
-
This architecture diagram shows the connection flow for the foundational and advanced Cloud Intelligence Dashboards.
Click to enlarge
Step 1
AWS Data Exports delivers AWS CUR reports daily to the S3 bucket in the Management (Payer) Account.Step 2
An Amazon S3 replication rule copies Export data from the S3 bucket in a Management Account to the S3 bucket in the dedicated Data Collection Account automatically.
Step 3
Athena allows querying data directly from the S3 bucket using an AWS Glue table schema definition.
Step 4
QuickSight creates datasets from Athena, refreshes daily and caches in SPICE for QuickSight.
Step 5
User teams (such as executives, FinOps, and engineers) can access Cloud Intelligence Dashboards in QuickSight. Access is secured through IAM, IAM Identity Center, and optional row-level security.
Step 6
Optionally, the advanced data collection can be deployed to enable advanced dashboards based on AWS Trusted Advisor, AWS Health Events, and other sources. Additional data is retrieved from AWS Organizations or Linked Accounts. In this case, an Amazon EventBridge rule triggers AWS Step Functions for data collection modules on a configurable schedule.Step 7
The Account Collector AWS Lambda function in Step Functions retrieves linked account details using an Organizations API.Step 8
The Data Collection Lambda function in Step Functions assumes the role in each linked account to retrieve account-specific data using AWS SDK.Step 9
Retrieved data is stored in a centralized S3 bucket.
Step 10
Advanced Cloud Intelligence Dashboards leverage Athena and QuickSight for comprehensive data analysis.
Step 1
AWS Data Exports delivers AWS CUR reports daily to the S3 bucket in the Management (Payer) Account.Step 2
An Amazon S3 replication rule copies Export data from the S3 bucket in a Management Account to the S3 bucket in the dedicated Data Collection Account automatically.
Step 3
Athena allows querying data directly from the S3 bucket using an AWS Glue table schema definition.
Step 4
QuickSight creates datasets from Athena, refreshes daily and caches in SPICE for QuickSight.
Step 5
User teams (such as executives, FinOps, and engineers) can access Cloud Intelligence Dashboards in QuickSight. Access is secured through IAM, IAM Identity Center, and optional row-level security.
Step 6
Optionally, the advanced data collection can be deployed to enable advanced dashboards based on AWS Trusted Advisor, AWS Health Events, and other sources. Additional data is retrieved from AWS Organizations or Linked Accounts. In this case, an Amazon EventBridge rule triggers AWS Step Functions for data collection modules on a configurable schedule.Step 7
The Account Collector AWS Lambda function in Step Functions retrieves linked account details using an Organizations API.Step 8
The Data Collection Lambda function in Step Functions assumes the role in each linked account to retrieve account-specific data using AWS SDK.Step 9
Retrieved data is stored in a centralized S3 bucket.
Step 10
Advanced Cloud Intelligence Dashboards leverage Athena and QuickSight for comprehensive data analysis.
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
CloudFormation automates the deployment process in a dedicated account, adhering to best practices. This Guidance processes data from AWS CUR, and Trusted Advisor, storing it in Amazon S3. AWS Glue processes this information, which Athena then queries to update QuickSight visualizations. This streamlined workflow helps ensure you always have access to up-to-date cost insights through your dashboards.
-
SecurityRead the Security whitepaper
This Guidance employs role-based access controls and follows the principle of least privilege. Administrators can manage different user personas through a QuickSight and IAM integration or IAM Identity Center, while CloudFormation templates help ensure deployment with minimal required permissions. You can maintain data sovereignty by keeping all cost and usage information within your own S3 buckets, eliminating the need for third-party tools. Even if your organization has multiple Management Accounts, you can enhance security through row-level security, restricting access to only your owned linked accounts' data. Additionally, AWS CUR stored in Amazon S3 can be encrypted at rest for enhanced data protection.
-
ReliabilityRead the Reliability whitepaper
Amazon S3 is a durable storage service for cost and usage data, which AWS Glue and Lambda process before visualization through Athena and QuickSight. This serverless approach provides resilient data processing with automatic retries and failover capabilities. This Guidance also offers reliability insights, including information on single-Availability Zone virtual private cloud (VPC) endpoints and Trusted Advisor recommendations across the organization. By using a highly available and durable serverless architecture, this Guidance offers reliable visualization of service usage and billing information, minimizing the risk of downtime or data loss.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
This Guidance combines Athena views and QuickSight datasets to efficiently process and visualize large-scale data. This approach offers automatic scaling without compromising performance, adapting seamlessly to changing workload demands. By leveraging services like Amazon S3, AWS Glue, Athena, QuickSight, and Lambda, you can maintain high performance efficiency while scaling up or down as needed. This helps you to access and analyze your cost and usage data quickly and efficiently, regardless of the volume of information being processed.
-
Cost OptimizationRead the Cost Optimization whitepaper
QuickSight offers cost-effective visualization of AWS usage data. You can customize dashboards without incurring additional fees to maximize value. By primarily using serverless services like Amazon S3, AWS Glue, Athena, and Lambda, you pay only for the resources you use, aligning costs directly with actual usage and demand.
-
SustainabilityRead the Sustainability whitepaper
Built on serverless services, including Amazon S3, AWS Glue, Athena, QuickSight, and Lambda, this Guidance enables organizations to improve financial accountability, cost optimization, and sustainability initiatives. The serverless architecture consumes resources only as needed, supporting a sustainable approach, potentially reducing overall energy consumption and environmental impact.
Related Content
Cloud Intelligence Dashboards Framework
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between Amazon or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.