Powering serverless multi-account analytics with AWS Lambda, Amazon Aurora, and Amazon Quick Suite

As a large Amazon Web Services (AWS) public sector customer, you might need information about which accounts contributed to this month’s billing surge, which workloads generated the most security findings, or which services experienced the highest adoption rates. Although straightforward in single-account environments, finding such information becomes complex when organizations manage hundreds or thousands of AWS accounts.

Without comprehensive visibility across multiple accounts, public sector organizations lack the information needed to make sound decisions about the costs, security, and performance of their cloud workloads. As a result, they experience adverse impacts on decision-making speed and quality. In this post, we will discuss a comprehensive analytics solution we designed, which provides public sector decision makers with the insights necessary to accelerate data-driven decision-making through a serverless, event-driven architecture.

Understanding the multi-account analytics challenge

Organizations rely on multi-account AWS strategies for security boundaries, cost management, and compliance across business units. However, many organizations lack full access to their payer account or centralized AWS Organizations account due to organizational constraints or partner management arrangements. This distributed approach creates visibility gaps, making comprehensive cloud infrastructure oversight difficult. The need for centralized analytics becomes critical in public sector environments where compliance and security standards demand continuous monitoring. This implementation addresses these challenges through a hub-and-spoke architecture that aggregates multi-account data into a centralized analytics solution, enabling unified visibility while maintaining the security benefits of account separation.

The solution implements a serverless architecture built on proven AWS services. At its foundation, the solution uses AWS Lambda for data processing, Amazon Aurora PostgreSQL-Compatible Edition for data storage, and Amazon Quick Suite for visualization and analytics. This serverless approach is cost-effective and provides the scalability needed to handle data from hundreds of AWS accounts.

The architecture consists of two primary account types. The Analytics Account serves as the central hub that receives, processes, and aggregates data from all sender accounts. The Sender Accounts act as spokes that collect and transmit their infrastructure data to the analytics account. Each sender account runs a lightweight Lambda function that gathers comprehensive data from over 20 AWS services and securely uploads it to the central Amazon Simple Storage Service (Amazon S3) bucket.

The data flow follows a six-step process that begins with Amazon EventBridge triggering sender Lambda functions on a daily schedule. These functions retrieve data collection scripts from the analytics account, gather comprehensive infrastructure data, and upload daily and monthly JSON files to Amazon S3. Amazon S3 events then trigger receiver Lambda functions that process the JSON data and load it into Aurora PostgreSQL-Compatible, where it becomes available for analysis through Amazon Quick Suite dashboards. The following diagram illustrates this architecture.

Figure 1: Architecture overview

Data collection capabilities

This solution collects data from over 20 AWS services, delivering comprehensive cloud infrastructure visibility to organizations. The solution aggregates security and compliance data from AWS Security Hub, AWS Config, Amazon GuardDuty, AWS Identity and Access Management (IAM), and AWS CloudTrail. These services track user activity, evaluate resource configurations, and provide threat detection across AWS accounts.

This solution leverages AWS Cost Explorer, AWS Systems Manager, and AWS Trusted Advisor  for comprehensive cost and resource management—providing detailed analysis, usage metrics, and optimization guidance. For security posture assessment, it gathers security inventory data from AWS Key Management Service (AWS KMS), AWS WAF, AWS Secrets Manager, and AWS Certificate Manager.

This data collection creates unified dashboards that deliver technical and business insights for decision-making. The solution aggregates data from multiple sources into a single view, addressing multi-account AWS environment challenges through integrated monitoring capabilities.

Security architecture

Multilayered security protection maintains data integrity and access control. The Amazon Aurora database operates in private subnets without internet access, following AWS best practices where database instances in private subnets can’t access the internet directly. Virtual private cloud (VPC) endpoints deliver secure connectivity without internet gateways by creating private connections that don’t traverse the public internet.

The solution enforces least privilege access through service-specific IAM roles and automatic database credential rotation through AWS Secrets Manager. IAM policies assign permissions that determine who can manage Amazon Relational Database Service (Amazon RDS) resources, and database engine security features control database login access.

Cross-account access operates through controlled S3 bucket policies that restrict data uploads to authorized sender accounts only. Security groups manage VPC and Amazon Quick Suite access, functioning as instance-level virtual firewalls alongside network access control lists (ACLs) that provide subnet-level protection. Network-level security works with application-level access controls to create defense-in-depth protection.

This security model maintains strict security standards while delivering centralized analytics capabilities through multiple security control layers across all infrastructure components.

The technical specifications use AWS serverless technologies, including Lambda functions running Python 3.13 with 10 GB memory allocation and 15-minute timeout configurations. The Amazon Aurora Serverless v2 database delivers Multi-AZ deployment with Data API enabled for performance and availability. The VPC architecture uses a 10.0.0.0/16 network with three private subnets distributed across Availability Zones for high availability and fault tolerance.

Analytics and visualization through Amazon Quick Suite

The solution uses the powerful analytics capabilities of Amazon Quick Suite to transform raw infrastructure data into actionable insights. The solution establishes secure VPC connectivity to Aurora PostgreSQL through VPC endpoints and therefore sensitive data never traverses the public internet. Amazon Quick Suite support for multiple data sources—including Amazon S3, Amazon Aurora, and traditional databases—provides flexibility for organizations with diverse data requirements.

The integration with Amazon Quick Suite enables natural language querying capabilities. Users can ask questions about their infrastructure in plain English and receive immediate, data-driven answers. This democratization of analytics access means that both technical and business stakeholders can benefit from the solution’s insights without requiring deep technical expertise.

The solution features comprehensive dashboards that present critical insights across multiple domains of AWS infrastructure management. These dashboards visualize key metrics for account status, cost management, security assessments, configuration details, operational metrics, application health, and system resilience. With these analytics capabilities, teams can maintain continuous visibility of their compliance requirements, analyze cost trends, detect potential security issues, and fine-tune resource allocation throughout their AWS environment.

Figure 2: Sample overview screen of the dashboard

Implementation and deployment considerations

Successful deployment requires careful planning and adherence to best practices. Organizations should begin with a phased rollout, starting with a pilot group of accounts before expanding to full deployment. By following this approach, teams can validate the data collection process, test dashboard functionality, and refine access controls before scaling to production environments.

Sender account configuration remains minimal, requiring only the deployment of lightweight Amazon CloudFormation templates and the configuration of cross-account permissions. This streamlined approach makes it possible to add new accounts to the analytics solution with minimal technical expertise. For implementation details, refer to this GitHub repository.

Best practices and operational excellence

Organizations deploying this implementation must establish data governance policies that include data retention schedules and access control procedures. Continuous security monitoring and alerting protects the solution while monitoring the connected account security posture. Performance optimization through AWS Lambda monitoring and Amazon Aurora utilization tracking maintains the solution’s performance when data volumes increase.

Cost management requires implementing Amazon S3 lifecycle policies to control storage costs and optimizing resource usage based on utilization patterns. Organizations must establish backup strategies, implement CloudWatch alarms for critical components, and develop data validation processes for data quality and reliability.

The architecture handles hundreds of sender accounts and processes large data volumes efficiently. The serverless architecture scales automatically with organizational growth, and multi-Region deployment capabilities support geographically distributed organizations. API integration plans deliver RESTful access to analytics data for programmatic integration with existing enterprise systems.

Future enhancements include machine learning (ML) integration for predictive analytics and anomaly detection, extended data source support for additional AWS services and third-party integrations, and advanced visualization capabilities with custom widgets and dashboard functionality. The architecture follows AWS best practices by taking a data-driven approach to building high-performance systems.

Conclusion

In this post, we introduced a serverless multi-account analytics solution that provides public sector organizations with a single view across their AWS accounts, using AWS Lambda, Amazon Aurora PostgreSQL, and Amazon Quick Suite for comprehensive infrastructure insights.

To learn more about implementing similar solutions in your organization, explore AWS best practices for multi-account strategies and consider how centralized analytics can enhance your cloud governance and compliance efforts.

Ready to implement your own solution? Access the deployment template hosted in AWS Samples GitHub repositories today.