How to manage cost overruns in your AWS multi-account environment – Part 1

AWS provides a flexible and secure environment where you can experiment, innovate, and scale more quickly. As you build and deploy your workloads, you need mechanisms to isolate your resources (for example, a resource container). You can use multiple AWS accounts for this purpose. An AWS account provides natural security, access, and billing boundaries for your AWS resources. Although you might start your AWS journey with a single account, AWS recommends that you set up multiple accounts using AWS organizations as your workloads grow in size and complexity.

When you set up a multi-account environment, you need a plan for cloud cost management. Cloud computing offers lower total cost of ownership (TCO), but you still need an effective cost control mechanism to make sure you only pay for what you need.

When you set up your cost control system, we recommend that you focus on the following core principles:

• Budget your spend with custom thresholds.
• Monitor and analyze how your costs progress toward limits.
• Take action to reduce unintended costs.

AWS Budgets gives you the ability to set custom budgets that alert you when your costs or usage (actual or forecasted) exceed your budgeted amount. With the recent launch of AWS Budget actions, you can now preconfigure actions that can trigger the implementation of AWS Identity and Access Management (IAM) policies or service control policies (SCPs). In addition, you can stop target Amazon Elastic Compute Cloud (Amazon EC2) or Amazon RDS instances in your account.

In a multi-account AWS environment, depending on your organization’s governance structure, there are two patterns for managing the budget:

• Centralized budget management, where the budget is set by the management account for all its member accounts.
• Decentralized budget management, where the budget is set for individual member accounts by its owners.

In this two-part blog series, we will share approaches that can stop cost overruns using AWS Budgets alerts no matter which budget management pattern you use. These approaches are better suited for accounts with non-production or lower environment applications and is not intended for production application accounts without proper considerations as they can be potentially disruptive.

Centralized budget pattern

In this budget pattern, a central team manages the total cloud spend and divides it among departments in the company. For this scenario, you can have two approaches to control cost overruns:

• Use budget actions to restrict SCPs on an account or accounts to prevent new resource deployment. This approach is more native and turn-key, is fairly straightforward, and easy to implement. However, it will impact other accounts in the OU after you apply the restrictive SCP. It is suitable for cases where there is a one-to-one mapping between an OU and an account.
• Use restrictive SCPs to quarantine the account or accounts to an AWS Organizations organizational unit (OU). This approach enables to apply specific restrictions per account basis. However, you’ll have to predefine an OU where the account should be moved after a budget condition breach.

We will cover approach specific considerations in detail in their respective sections below.

Use AWS Budgets actions to apply restrictive SCPs from the management account

AWS Budgets provides the capability to configure cost-saving controls, or actions, that run either automatically on your behalf or by using a workflow approval process. You can use actions to define an explicit response that you want to take when a budget exceeds its action threshold. You can trigger these alerts on actual or forecasted cost and usage budgets. The workflow for this approach includes the following steps:

1. The management account sets the budget and threshold for the member account using budget filters.
2. When the budget threshold is breached, a budget action applies a restrictive SCP on the OU.

Figure 1 illustrates this workflow:

Figure 1: Using AWS Budgets actions to apply restrictive SCPs from the management account

Prerequisites and assumptions

• You need to set a cost budget for your AWS account. Follow the instructions in Creating a cost budget in the AWS Billing and Cost Management User Guide.
• AWS Organizations setup in all-feature mode with access to the management account.
• Create an SCP with restrictive permissions that can be applied to the OU to restrict the creation of resources to avoid cost overruns. Follow the instructions in Creating, updating, and deleting service control policies and Attaching and detaching service control policies in the AWS Organizations User Guide.

Here is an example of an SCP that restricts the creation of EC2 instances:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": [
        "ec2:Run*",
        "ec2:Start*",
        "ec2:Create*"
      ],
      "Resource": "*"
    }
  ]
}

•  Create an IAM role and attach identity-based policies (IAM policies) for Billing and Cost Management to allow AWS Budgets to execute an action.

Deploy this approach

1. To configure a budget action, open the Billing and Cost Management console, and in the navigation pane, choose Budgets.
2. Complete the fields as shown in Figure 2.

Figure 2: Set your budget page of the Billing and Cost Management console

      3. In Set threshold based on, choose the one of the following options:

1. • Actual cost: This creates a notification for your actual spend.
   • Forecasted cost: This creates a notification for your forecasted spend.

4. Under Alert threshold, enter the amount as an absolute value or percentage. For example, if your budget is $200 and you want to be notified at $160, enter 160 as your absolute value or 80 as your percentage. From the dropdown, choose either Absolute amount or % of budgeted amount:

1. • Absolute amount: You are notified when the threshold amount has exceeded.
   • % of budgeted amount: You are notified when the threshold percentage of the budget has exceeded.

5. Configure notification settings for your action. For information, see Creating an Amazon SNS topic for budget notifications. Under Amazon Simple Notification Service, enter the Amazon Resource Name (ARN) for your Amazon SNS topic, and then choose Verify.

Figure 3: Define your budget threshold

6. In Choose your budget action:

• Choose an IAM role to allow AWS Budgets to perform an action on your behalf.
• For Which action type should be applied when the budget threshold has been exceeded, choose Service Control Policy.
• For Select the SCP you want to apply, choose the SCP you created in the prerequisites.
• For Choose the organizational units you want this policy to apply to, choose an OU to which you want to apply the restrictive SCP (for example, Development).
• For Do you want to automatically execute this action when this threshold is exceeded, choose Yes.

Figure 4: Choose your budget action

Considerations

• The AWS Budgets actions work when the selected AWS Budgets period within your budgets configuration is monthly, quarterly, or annually.
• To prevent any unintended effects of restrictive SCPs on other AWS accounts, create separate budgets for individual linked accounts or combine multiple linked accounts in the same OU using budget filters.
• Once the root cause for the cost overrun has been identified and resolved, the restrictive SCP from the OU can be manually removed or you can reverse the budget action and choose the reset option. Also note, if no remediation action is taken, the restrictive SCPs will be automatically detached at the start of the next budget period.

Use AWS Budgets alerts to move the AWS account to an OU with restrictive SCPs

In this approach, the management account moves the member account to another restrictive OU after the budget threshold for the member account is met. You can adjust the restrictive policies in this new OU according to your needs. In this blog post, we use a read-only SCP for EC2 shown here:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": [
        "ec2:Run*",
        "ec2:Start*",
        "ec2:Create*"
      ],
      "Resource": "*"
    }
  ]
}

The workflow for this approach includes the following steps:

1. The management account sets the budget and threshold for the member account.
2. An SNS notification is sent upon breach of the budget threshold.
3. The SNS notification triggers a Lambda function.
4. The Lambda function moves the member account from its current OU to an OU with more restrictive policies. In this case, the SCP on this new OU is shown in the preceding example.
5. The Lambda function sends an email to notify recipients (AWS account owners or administrators) of this move.

Figure 5 illustrates this architecture:

Figure 5: Using AWS Budgets actions to move an AWS account to an OU

Prerequisites and assumptions

• Create an OU with restrictive SCPs. The member accounts will be moved to this OU when the budget threshold is breached.
• You must have already set budgets for the member accounts.

To deploy this approach using AWS CloudFormation

1. Choose this button to launch the AWS CloudFormation template. The template will create a stack the us-east-1 Region.

2. Enter a unique name for the stack (for example, budget-control-stack).

3. In Parameters:

• For AccountList, enter a comma-separated list of member account numbers that you want to move to a restrictive OU. Enclose the account numbers in quotation marks (“).
• For DeploymentType, choose Centralized from the dropdown.
• For NotificationEmailAddress, enter an email address where the notification should be sent following an account move.
• For QuarantineOU, enter the ID of the OU where you want to move the account to.

4. In Capabilities and transforms, select the checkboxes to acknowledge that AWS CloudFormation will create IAM resources, AWS CloudFormation might create IAM resources with custom names, and AWS CloudFormation might require the following capability: CAPABILITY_AUTO_EXPAND.

5. Choose Create stack.

Figure 6: Stack name and parameters

6. After the stack is created, copy the SNS topic ARN for the key BudgetNotificationSnsTopic displayed in Outputs section.

7. Use the ARN value copied in the previous step to update the Amazon Simple Notification Service parameter in budget threshold configuration.

Figure 7: Configure thresholds

When the budget threshold is breached for one of the member accounts entered in the AccountList parameter, the member account will be moved to the restrictive OU specified in the QuarantineOU parameter. You’ll also get an email to notify you of the move, as shown in Figure 8.

Figure 8: Notification the budget threshold has been reached

After you have determined the reason for the cost increase, you can manually move the member account to its original OU.

Considerations

• In the approach, the account is moved from its current OU to a quarantine OU. To avoid any inconsistency, you may want ensure that the quarantine OU’s SCP largely resembles the source account OU’s SCP.
• Once the root cause for the cost overrun has been identified and resolved, the account has to be manually moved to the original OU.

Conclusion

Stay tuned! In the second post in this two-part series, we show how you can control cost overruns in a decentralized budget management pattern.

About the authors