AWS Management & Governance Blog

How to manage cost overruns in your AWS multi-account environment – Part 2

In the first post of this two-part series, we showed you two approaches for preventing cost overruns in a centralized budget management pattern:

  • Applying a restrictive service control policy (SCP) to an organizational unit (OU).
  • Moving the account to another OU with restrictive SCPs.

In this post, we share how you can prevent cost overruns in a decentralized budget management pattern.

Decentralized budget pattern

In a decentralized budget management pattern, the budget is set by the owners of member accounts and not the management account. In this scenario, the owners of the member account manage their own cloud spend for their individual accounts. There are two approaches for controlling cost overruns in a decentralized budget management pattern:

  • Use AWS Budgets actions to restrict the IAM policy of the individual account to prevent new resource deployment. This approach gives you more granular control over a group of users, groups, or roles, but it does not prevent someone with more permissions in the member account from overriding this change.
  • Quarantine the account to an organization unit in AWS Organizations with restrictive SCPs. This approach enables to apply specific restrictions per account basis. However, you’ll have to predefine an OU where the account should be moved after a budget condition breach.

We will cover approach specific considerations in detail in their respective sections below.

Use AWS Budgets actions to apply restrictive IAM policies in the member account

We will walk you through the process for using AWS Budget actions to apply a restrictive managed policy (AmazonEC2ReadOnlyAccess) to an IAM group, user, or role to prevent cost overruns. You can adjust the IAM policy according to your needs.

The workflow for this approach includes the following steps:

  1. The member account sets an AWS account budget for itself.
  2. Budget actions are triggered when the budget threshold is breached.
  3. Budget actions apply a restrictive IAM policy (in this case, an EC2 read-only policy) to an IAM group that prevents the users in that group from deploying EC2 resources. Preventing further EC2 resource deployment prevents cost overruns.

Figure 1 illustrates this workflow:

In step 1, the budget is set in member accounts. In step 2, when the threshold is breached, a budget action is triggered and a restrictive IAM policy is applied to the role, user, or group.

Figure 1: Using AWS Budgets actions to apply restrictive IAM policies in the member account

 

Prerequisites and assumptions

  • You need to set a cost budget for your AWS account. Follow the instructions in Creating a cost budget in the AWS Billing and Cost Management User Guide.

Deploying this approach

  1. To configure a budget action, follow the instructions in Editing a budget.
  2. In the Billing and Cost Management console, choose Configure thresholds.

The Set your budget page includes fields for the budget name (in this example, Monthly EC2 budget), period (Monthly), budget effective dates (Recurring budget), start month (Feb 2021), and budgeted amount ($100).

Figure 2: Set your budget page of the Billing and Cost Management console

 

3. In Set threshold based on, choose the one of the following options:

    • Actual cost: This creates a notification for your actual spend.
    • Forecasted cost: This creates a notification for your forecasted spend.
  1. Under Alert threshold, enter the amount as an absolute value or percentage. For example, if your budget is $200 and you want to be notified at $160, enter 160 as your absolute value or 80 as your percentage. From the dropdown, choose either Absolute amount or % of budgeted amount:
    • Absolute amount: You are notified when the threshold amount has exceeded.
    • % of budgeted amount: You are notified when the threshold percentage of the budget has exceeded.
  1. Choose either Absolute amount or % of budgeted amount:
    • Absolute amount – You are notified when the threshold amount has passed.
    • % of budgeted amount – You are notified when the threshold percentage of the budget has passed.

6. Configure notification settings for your action. For information, see Creating an Amazon SNS topic for budget notifications. Under Amazon Simple Notification Service, enter the Amazon Resource Name (ARN) for your Amazon SNS topic, and then choose Verify.

The Define your budget threshold section includes fields for setting a threshold based on actual or forecasted cost. There is a section for setting up notifications using Amazon SNS or AWS Chatbot.

Figure 3: Define your budget threshold

 

7. In Choose your budget action:

    • Configure notification settings for your action. This defaults to the same notification settings created in Creating an Amazon SNS topic for budget notifications.
    • Choose an IAM role to allow AWS Budgets to perform an action on your behalf.
    • For Which action type should be applied when the budget threshold has been exceeded, choose IAM Policy.
    • For Select the IAM policy you want to apply, choose the sample AmazonEC2ReadOnlyAccess policy.
    • For Choose the user, group, or role you want this action applied to, choose an appropriate user, group, or role. In Figure 4, we use an IAM group named testgroup.
    • For Do you want to automatically execute this action when this threshold is exceeded, choose Yes.

The Choose your budget action section contains fields completed as described in the body of the post.

Figure 4: Choose your budget action

 

8. Review your budget and then choose Done.

Considerations

  • The AWS Budgets actions work when the selected AWS Budgets period within your budgets configuration is monthly, quarterly, or annually.
  • This approach only applies to the selected set of IAM users/groups/roles and does not restrict any new IAM users/groups/roles created after the fact.
  • Once the root cause for the cost overrun has been identified and resolved, the restrictive IAM policy applied to the IAM users/groups/roles has to be manually removed.

Use AWS Budgets alerts in the member accounts to move the AWS account to an OU with restrictive SCPs

This approach is similar to the quarantine approach we described in part one of this blog series. After the budget threshold for the member account is met, the management account moves the member account to another restrictive OU. The key difference is that the budget for the member account is set by the owners of the member account. The restrictive policies on this new OU can be adjusted according to your needs. In this blog post, we use a read-only SCP for EC2 shown here:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": [
"ec2:Run*",
"ec2:Start*",
"ec2:Create*"
],
"Resource": "*"
}
]
}

The workflow for this approach includes the following steps:

  1. The owner of the member account sets the budget and threshold for the account.
  2. When the budget threshold is reached, an SNS notification is sent.
  3. The SNS notification triggers a Lambda function.
  4. The Lambda function parses the budget breach notification and places that event on the management account’s event bus.
  5. The management account’s event bus triggers the Amazon CloudWatch Events rule.
  6. The rule invokes a Lambda function.
  7. The Lambda function moves the member account to an OU with more restrictive policies. In this case, the SCP on this new OU is shown in the preceding example.
  8. The Lambda function sends an email to notify recipients of this move.

Figure 5 illustrates this architecture:

The workflow in the diagram is described in the body of the post.

Figure 5: Using AWS Budgets actions to move an AWS account to an OU

 

Prerequisites

To deploy this approach using AWS CloudFormation

  1. Choose this button to launch the AWS CloudFormation template. The template will create a stack the us-east-1 Region. Clickable launch stack button
  2. Enter a unique name for the stack (for example, budget-control-stack-member-accounts).
  3. In Parameters:
    • For AccountList, enter a comma-separated list of member account numbers that you want to move to a restrictive OU. Enclose the account numbers in quotation marks (“).
    • For DeploymentType, choose Decentralized from the dropdown.
    • For NotificationEmailAddress, enter an email address where the notification should be sent following an account move.
    • For QuarantineOU, enter the ID of the OU where you want to move the account to.
  4. In Capabilities and transforms, select the checkboxes to acknowledge that AWS CloudFormation will create IAM resources, AWS CloudFormation might create IAM resources with custom names, and AWS CloudFormation might require the following capability: CAPABILITY_AUTO_EXPAND.
  5. Choose Create stack.

The stack is configured with values used in the body of the post.

Figure 6: Stack name and parameters

  1. After the stack is created, make a note of the ARN for the output key BudgetNotificationSnsTopic. You’ll need it later when you update your budget notifications.
  2. From the navigation pane, choose StackSets.
  3. Choose the stack set named MemberAccountStackSet-<YourStackName>, and from Actions, choose Add new stacks to StackSet.

StackSets page includes an Actions dropdown. Add new stacks to StackSet is highlighted

Figure 7: StackSets page in the CloudFormation console

  1. In Set deployment options, choose Deploy to organizational units (OUs). The member account you entered earlier should be part of this OU.
  2. Under Specify regions, choose an AWS Region, and then choose Next.

Set deployment options page includes sections for deployment targets, regions, and deployment options.

Figure 8: Set deployment options

  1. In Specify overrides, from Edit override value, choose Override StackSet Value.

On the Specify overrides page, there is a table with columns for name, StackSet value, and override value. The Override StackSet value is selected from the Edit override value menu.

Figure 9: Specify overrides

  1. On Override StackSet parameter values, enter the account ID of the management account, and then choose Save changes.

The Override StackSet parameters values page displays the management account ID for configuring the event bus ARN in the member accounts.

Figure 10: Override StackSet parameter values

  1. Choose Next on the Specify overrides screen. Review your selections and then choose Submit. This deploys the required resources in the member accounts listed in AccountList.

The Specify overrides page lists the name (ManagementAccount), StackSet value, and override value.

Figure 11: Specify overrides

  1. After the stack sets are successfully deployed, update the budgets of the member accounts with the SNS topic ARN you noted in step 6.

After the budget threshold is breached for one of the member accounts entered in the AccountList parameter, this solution will move that member account to the restrictive OU specified in the QuarantineOU parameter. You’ll also get an email notifying you of this move as shown below.

The notification from AWS Notifications says the budget threshold for the account has been reached and the account has been moved from Developer OU to the Read Only OU to prevent incurring additional charges.

Figure 12: Notification the budget threshold has been reached

After you have determined the reason for the cost increase, you can manually move back the member account back to its original OU.

Considerations

  • In the approach, the account is moved from its current OU to a quarantine OU. To avoid any inconsistency, you may want ensure that the quarantine OU’s SCP largely resembles the source account OU’s SCP.

Conclusion

In this two-part series, we shared you two approaches in a centralized budget pattern and two approaches in a decentralized budget management pattern that you can use to prevent cost overruns. Each approach has its own benefits and considerations.

The centralized budget management pattern with an SCP override approach we described in part 1 is fairly straightforward and easy to implement, but it will impact other accounts in the OU after you apply the restrictive SCP. It is suitable for cases where there is a one-to-one mapping between an OU and an account. This impact can be mitigated if you use the quarantine approach for this pattern, but you’ll have to predefine an OU where the account should be moved after a budget condition breach.

The decentralized budget management pattern with IAM override approach we described in this post gives you more granular control over a group of users, groups, or roles, but it does not prevent someone with more permissions in the member account from overriding this change. You can use the quarantine account approach in the decentralized budget management pattern instead, but again you’ll need to predefine the OU where the account should be moved after a budget condition breach.

For more information, see AWS Cost Management and AWS Budgets.

About the authors

Siva Rajamani

Siva Rajamani is a Boston-based Enterprise Solutions Architect at AWS. He enjoys working closely with customers and supporting their digital transformation and AWS adoption journey. His areas of focus are serverless, application integration, and security. Outside of work, he enjoys outdoor activities and watching documentaries.

Sudhanshu Malhotra

Sudhanshu Malhotra is a Boston-based Enterprise Solutions Architect for AWS. He’s a technology enthusiast who enjoys helping customers find innovative solutions to complex business challenges. His areas of focus are DevOps, machine learning, and security. When he’s not working with customers on their journey to the cloud, he enjoys reading, hiking, and exploring new cuisines.

Pawan Matta

Pawan Matta is a Solutions Architect with AWS whose focus is games and enterprise governance. In his free time, Pawan loves watching cricket and playing video games with friends.