Operational excellence is the focus on running and monitoring systems effectively, and we accomplish that here with the help of Step Functions, which is used to visualize the overall workflow when the API report is created and downloaded. Additionally, Amazon CloudWatch can be used in this Guidance to help you visualize metrics and logs across all services used. Lastly, Amazon SNS lets you know when reports have succeeded or failed.

AWS Identity and Access Management (IAM), CloudWatch, AWS KMS, and Secrets Manager collectively protect data, systems, and assets in ways that improves your security posture. First, IAM policies are scoped down to provide least privilege access required for the services to function properly. By scoping IAM policies to the minimum permission required, you limit unauthorized access to resources. Second, AWS KMS provides encryption for both data at-rest and data in-transit using Pretty Good Privacy (PGP) encryption of data files. Third, Secrets Manager provides secure storage for API keys and hashes. AWS KMS and Secrets Manager are used to ensure that data and sensitive information is secured properly using encryption and hashing. And fourth, CloudWatch monitors logs and metrics across all services used to detect unexpected and unwanted behaviour.

Lambda, Amazon S3, and DynamoDB help ensure your workloads perform their intended functions correctly and consistently, as well as help your workloads recover quickly from failure. With Lambda, you can initiate code without managing infrastructure. It also runs your function in multiple Availability Zones to ensure that it is available to process events in case of a service interruption in a single zone. Amazon S3 stores multiple copies of data across Availability Zones and provides 99.999999999% (11 9's) of durability of the data stored within the buckets. Furthermore, DynamoDB provides a serverless data store and stores multiple copies of data across Availability Zones while maintaining consistent and fast performance.

Lambda and AWS Glue help you structure and streamline your allocation of IT and computing resources. Lambda supports parallelization and concurrency with automatic scaling and can handle many concurrent processes to support the ingestion of multiple data pipeline workflows. AWS Glue jobs can scale both horizontally and vertically by letting you set the number of workers and the worker type to support small and large extract, transform, and load (ETL) processes. 

AWS Glue is a serverless ETL service and only incurs cost when an AWS Glue job is occurring. When no AWS Glue job occurs, AWS Glue does not incur cost. And, AWS Glue scales both vertically and horizontally, allowing you to tune your ETL processes to be the most cost efficient. AWS Glue is cost efficient, resilient, highly performant, and scales to support any dataset transformation size. In addition, Amazon S3 stores reporting data and allows you to choose how data is stored into different tiers. You can configure Amazon S3 Lifecycle Policies to ensure older data is stored on storage tiers with less cost.

Lambda offers autoscaling, as well as the ability to scale from zero to peak demands to ensure only the required capacity is utilized. Also, DynamoDB removes unused resources from the database based on time to live (TTL), set by you, so only necessary storage is utilized. DynamoDB supports both provisioned and on-demand capacity to ensure only the necessary capacity is used to support the workload. Data stored in DynamoDB is compressed so minimal storage space is utilized. These services help ensure the lowest footprint of compute and data storage while lowering overall carbon generation, as services are only utilized as-needed with the most performant resources available.