Every service has built-in observability, with metrics published to CloudWatch, where dashboards and alarms are then configured. 

IAM policies are created using the least-privilege access, so every policy is restricted to the specific resource and operation. Secrets, keys, and configuration items are centrally managed and secured using the AWS KMS service. The data at rest in the Amazon S3 bucket is encrypted using AWS KMS keys. File transfers into Amazon S3 are secured using Pretty Good Privacy (PGP) encryption and tunnel level TLS 1.2 encryption for API calls. Data transfer through API calls are encrypted using TLS 1.2.

Every service or technology for each architecture layer is fully managed by AWS, making the overall architecture elastic, highly available, and fault-tolerant. Incremental data processing is not included in the solution. This solution is built using a multi-tier architecture, where every tier is independently scalable, deployable, and testable.

Using serverless technologies, you only provision the exact resources you use. The serverless architecture reduces the amount of underlying infrastructure you need to manage, allowing you to focus on solving your business needs. All components of the solution are collocated in a single region and uses a serverless stack, which avoids the need for you to make infrastructure location decisions apart from the region choice. You can use automated deployments to deploy the solution components into any region quickly, providing data residence and reduced latency.  Experiments and tests can be performed against different load levels, configurations, and services.

This Guidance utilizes managed services for cost optimization. As the data ingestion velocity increases and decreases, the costs align with usage. When AWS Glue is performing data transformations, you only pay for infrastructure while the processing is occurring. In addition, through a tenant solution model and resource tagging, you can automate cost usage alerts and measure costs specific to each tenant, application module, and service. IAM policies are created using the least-privilege access, such that every policy is restricted to the specific resource and operation.