AWS Cloud Operations Blog

Service Quota Observability Across Regions and Accounts

Customers often need to launch workloads in new accounts and regions. You could be developing an application in a development account, and looking to launch it in a production account, following AWS multi-account best practices on separating production and non-production workloads. You could also be launching a second instance of your payment processing application in the US East (N. Virginia) region to the Asia Pacific (Singapore) for lower latency to your end customers in Asia. One of the factors to consider before migration is that the majority of service quotas are regional, meaning they vary by region. This means that any prior service quotas increases for a specific region will not apply to other regions.

When launching workloads in new accounts or regions, you will need to review any differences in service quotas between regions or accounts due to prior approved service quota increases. This will prevent any ServiceQuotaExceededException errors during and after deployment. In this blog post, you will learn about a solution that allows you to programmatically list any differences in service quotas between accounts and regions due to prior approved service quota increases.

Additionally, some of the default values for service quotas differ by region. For example, for Amazon Simple Notification Service (Amazon SNS), the PublishMessage quota in US East (N. Virginia) is 30,000 messages per second (mps) by default, however for US East (Ohio) region, the default is 1,500 mps. For the US West (Oregon) region, the default PublishMessage quota is 3,000 mps. It is important to be able to programmatically list all the different default service quotas between regions before beginning your migration.

Overview of solution

This solution identifies differences in service quotas across regions and accounts by leveraging the AWS Command Line Interface (AWS CLI). The solution will identify and list differences in service quotas whether it is due to a prior increase or a difference in the default values. This post will also review how to use AWS Organizations to automatically request service quota increases for new accounts created in your organization.

Prerequisites

For this walkthrough, you should have the following prerequisites:

Identifying Service Quota Differences Across Regions and Accounts

Let’s discuss how to identify service quota differences across regions and accounts by using the AWS CLI.

Steps

  1. Set up your AWS CLI with the necessary ServiceQuotasReadOnlyAccess managed policy.
  2. Create an AWS CLI profile using the aws configure --profile <profilename> command. You can also use aws configure sso --profile <profilename>
  3. If you are comparing service quotas across accounts, be sure to create profiles for each account.
  4. Download the compareQuotas.sh file and cd (change directory) into the file’s directory.
  5. Run the following commands to compare quotas across regions or accounts.

chmod +x compareQuotas.sh

./compareQuotas.sh <profile1> <region1> <profile2> <region2>

Note that the first command (chmod) only needs to be run once in order to make the file executable. If you are comparing service quotas for the same account (profile) across regions, simply specify the same profile twice. For example:

chmod +x compareQuotas.sh

./compareQuotas.sh dev us-east-1 dev us-east-2

If you are comparing service quotas across accounts (profiles), specify the profiles.

chmod +x compareQuotas.sh

./compareQuotas.sh dev us-east-1 prod us-east-1

You can also compare service quotas across accounts and regions simultaneously. For example:

chmod +x compareQuotas.sh

./compareQuotas.sh dev us-east-1 prod us-east-2

At a high level, this script calls the Service Quotas API to list services, then feeds it into a list service quotas command. The result is that it lists all service quotas for all services. It then performs additional operations to remove unnecessary rows and columns, and then sorts the rows.

Figure 1 – Snippet of list of service quotas in a region

Figure 1 – Snippet of list of service quotas in a region

Above you can see that each line represents a specific service quota, including the name of quota, the service it belongs to, the unit of the service quota, and the applied value. However this only applies to one region. The script will generate another file for the second region you specify too, and then compare the two files generated by using a diff and output the differences to a quota-differences-<profile1>-<region1>-<profile2>-<region2>.txt file

6. Let’s review some diff syntax:

| — Line is different in the second file (the limits are different between the two files)

> — Line is additional in the second file (the service quota does not exist in Region 1)

< — Line has been deleted from the second file (the service quota does not exist in Region 2)

A potential reason for a service quota not existing in another region is if the service has not launched yet for that region, or if the service quota is global rather than regional.

Figure 2 - Service Quotas Diff Text File

Figure 2 – Service Quotas Diff Text File

Taking a look at a sample quota-differences.txt file generated, you can see that in the first two lines the separator | is shown meaning the service quotas are different in the two regions. In us-east-1 (left), the quota for Amazon Athena Active DDL queries is 21, whereas for us-east-2 (right), the quota for Amazon Athena Active DDL queries is 20. This is due to a prior approved service quota increase. On the second line, the quotas for Active DML queries are different. For lines 3-8 you can see the separator < is shown meaning the service quota does not exist in us-east-2 (right). This is because at the time of the blog posts writing, Amazon Braket is not supported in us-east-2.

Creating AWS Organizations Request Templates

To make provisioning new accounts even easier, you can use AWS Organizations to create service quota request templates. Quota request templates automatically request up to 10 quota increases for newly-created accounts in your organization.

Cleaning up

The only resources created during this post are text files, which you may choose to delete or store in Amazon Simple Storage Service (Amazon S3) using the Put-Object API and write permissions to a bucket.

Conclusion

You now have the ability to identify differences in Service Quotas across regions and across accounts. Review these differences when you are planning to migrate workloads across regions and across accounts to mitigate against ServiceQuotaExceededException errors. You can request quota increases for quotas you would like to match across accounts and regions, and create templates for up to 10 quotas to automatically request increases in new accounts.

Testimonial

SailPoint helps businesses automate the governance and management of identities across their organization, securing the right access to the right identity at the right time. When expanding their products to new regions, the compare quotas script automated what was previously manual work for SailPoint and allowed them to launch to two new regions in a fraction of the time.

About the authors:

Yassin Abouel Seoud

Yassin Abouel Seoud is an Enterprise Support Lead at Amazon Web Services. He works with customers on improving their workloads’ performance and resiliency, and resolving blockers. Prior to AWS, Yassin completed his Bachelor of Engineering in Mechanical Engineering at McGill University, and worked on expanding Alexa’s understanding and answering internationally.

Vincent Nicotra

Vincent Nicotra is a Senior Software Development Engineer with Service Quotas. Vincent has been with Amazon 9 and a half years and has been developing software for over 40 years.

Brad Hein

Brad Hein manages the DevOps FedRAMP and security teams at SailPoint. He has over 20 years of experience in the industry. At SailPoint, he started as a DevOps engineer then transitioned to build and manage the team bringing up SailPoint SaaS products for FedRAMP authorization. Prior to SailPoint, he wrote production software applications for storage management, architected then implemented Kubernetes clusters when Kubernetes was new, and migrated those clusters from data centers into AWS with no service downtime. He gives Yassin a hard time with all his AWS questions.