AWS Cloud Operations Blog

Supercharge Multi-Account Management with AWS CloudFormation

As your use of Amazon Web Services evolves, you will probably outgrow your first account, and need to move into a multi-account model.

There are plenty of benefits to using more than one AWS account:

  • An administrative boundary: I can choose how permissive or restrictive my policies are based on the account type. Separating user authority within an account can be complicated and error prone. Using separate accounts is often the answer.
  • A workload boundary: I can choose to peer (or not to peer) various workloads together within accounts, ensuring that my ‘blast radius’ for a poorly behaved application is minimized.
  • A billing entity: Detailed bills are generated at an account level. An account has higher ‘resolution’ than is afforded by billing tags, and can be easier to implement.

However, management complexities increase using multiple accounts. How do you manage the administrative boundaries? Who can log in and how? How do you manage your baseline infrastructure, such as VPCs and CloudTrail? Is it possible to deploy these by hand and not make potentially critical mistakes?

In this blog post I’ll explore ways to manage deployments in your multi-account AWS environment.

The broad answer to the questions above is to use infrastructure as code, and tools like AWS CloudFormation. CloudFormation gives you a language to describe your desired state, and have it implemented programmatically. For complex multi-account deployments, this can get difficult to manage quickly, especially for complex subjects such as IAM users, groups, polices, and roles.

The recent release of AWS Organizations has certainly helped with this complexity. Using policies, you can broadly permit or deny service use within your AWS account organizational structure. Layering these capabilities with consistently deployed IAM elements gets the most value from a multi-account model.

A tool to help

I’d like to introduce a tool I wrote while working with multiple AWS customers as a Professional Services Consultant. This tool allows you to describe complex multi-account IAM elements using simple YAML, and Jinja2 templates. Running a build produces a CloudFormation template per account that can be deployed in your preferred fashion.

Some customers have even built a pipeline so that when the configuration YAML is changed, a build is run and deployed automatically. They’re managing multiple accounts completely hands-off, from a central source of truth!

I’ll discuss the tool a bit. First, the tool assumes that you’re using a ‘parent’ account model where everyone logs in, or federates, into a single account. Then, users assume roles in the ‘child’ accounts for which they have permissions. This model was explored in more detail by an AWS customer with hundreds of accounts. For more information, see (SEC315) AWS Directory Service Deep Dive re:Invent session video.

I’ve included a simplification of the model for reference. Your child accounts can be one or many.

Tool walk-through

Prerequisites

To go through this walkthrough, you need the following:

  • A bash environment
  • A Python 2.7 interpreter
  • At least two AWS accounts that you can experiment with. You can use AWS Organizations to create them programmatically
  • The troposphere and the jinja2 libraries installed

sudo pip install troposphere
sudo pip install jinja2

Download the code

Clone or download the project from /awslabs/ on GitHub:


git clone git@github.com:awslabs/aws-iam-generator.git

Build your configuration file

Start with a simple configuration and grow it over time, as you would naturally evolve your AWS environment.

Create a file called config.yaml in the downloaded projects root directory. Cut and paste the example configuration below into the file. Substitute the account IDs with the account IDs that you’re using for this walkthrough.


---
accounts:
  central:
    # Change to the ID of your central parent account.
    id: 012345678910
    parent: true
  lab:
    # Change to the ID of a child account for specific roles.
    id: 109876543210

policies:
  AssumeLabPowerUser:
    description: Allow assumption of the PowerUser Role in the lab account
    assume:
      roles:
        - PowerUser
      accounts:
        # Use the name you've given your account in the 'accounts' section.
        - lab
    in_accounts:
      # Use the keyword 'parent' to match the account that has 'parent: true'
      - parent

roles:
  PowerUser:
    trusts:
      - parent
    managed_policies:
      - arn:aws:iam::aws:policy/PowerUserAccess
    in_accounts:
      # Use the keyword 'children' to match all accounts except the one marked 'parent: true'
      - children

groups:
  LabUsers:
    managed_policies:
      - arn:aws:iam::aws:policy/IAMSelfManageServiceSpecificCredentials
      - arn:aws:iam::aws:policy/IAMUserChangePassword
      - arn:aws:iam::aws:policy/IAMUserSSHKeys
      - arn:aws:iam::aws:policy/IAMReadOnlyAccess
      - AssumeLabPowerUser
    in_accounts:
      - parent

users:
  TestUser:
    groups:
      - LabUsers
    in_accounts:
      - parent

What happens now?

Because you have two accounts, expect two CloudFormation templates to be created.

The CloudFormation template for your central account should contain:

  • A managed policy called AssumeLabPowerUser. The policy document permits sts::AssumeRole on your child account.
  • A group called LabUsers. This group has built-in AWS policies attached to it to allow self service credential management. It also has the AssumeLabPowerUser policy attached that you created under the policies: section.
  • A user called TestUser who is a member of the LabUser group.

The CloudFormation template for your lab account should contain:

  • A role called PowerUser. This role has a trust document that allows it to be assumed from the central account. The roles policy document is modeled from the AWS built-in PowerUserAccess policy document.

Run the build

Executing the build.py script reads the contents of the config.yaml file and creates CloudFormation templates in your output_templates/ directory. If the build is successful, you won’t see any output. When a build fails, the output should produce an meaningful error to help figure out what’s wrong. Most of the time, the problem is in the yaml formatting (usually, indentation).


python build.py

If your build ran, you should see output in the output_templates/ directory


ls -l output_templates/
-rw-r--r-- 1 user user Users 3367 Apr 10 10:14 central(012345678910)-IAM.template
-rw-r--r-- 1 user user Users 1680 Apr 10 10:14 lab(109876543210)-IAM.template

You can see both of your CloudFormation templates. They are named so that you can easily identify the account in which they should be run. Also notice that your central template is about double the size of the lab template. This is expected as central contains a lot more information.

Explore the output

View the central template in your favorite viewer.

Your managed policy is there, with the sts::AssumeRole correctly formed:


...
        "AssumeLabPowerUser": {
            "Properties": {
                "Description": "Allow assumption of the PowerUser role in the lab account",
                "Groups": [],
                "PolicyDocument": {
                    "Statement": [
                        {
                            "Action": "sts:AssumeRole",
                            "Effect": "Allow",
                            "Resource": "arn:aws:iam::109876543210:role/PowerUser"
                        }
                    ],
                    "Version": "2012-10-17"
                },
                "Roles": [],
                "Users": []
            },
            "Type": "AWS::IAM::ManagedPolicy"
        },
...

You can also see the group that uses this managed policy:


....
        "LabUsersGroup": {
            "Properties": {
                "GroupName": "LabUsers",
                "ManagedPolicyArns": [
                    "arn:aws:iam::aws:policy/IAMSelfManageServiceSpecificCredentials",
                    "arn:aws:iam::aws:policy/IAMUserChangePassword",
                    "arn:aws:iam::aws:policy/IAMUserSSHKeys",
                    "arn:aws:iam::aws:policy/IAMReadOnlyAccess",
                    {
                        "Ref": "AssumeLabPowerUser"
                    }
                ],
                "Path": "/",
                "Policies": []
            },
            "Type": "AWS::IAM::Group"
        }
...

Finally, your user is here as a member of the group you’ve created:


...
        "TestUserUser": {
            "Properties": {
                "Groups": [
                    "LabUsers"
                ],
                "ManagedPolicyArns": [],
                "Path": "/",
                "Policies": [],
                "UserName": "TestUser"
            },
            "Type": "AWS::IAM::User"
        }
...

When you view the lab template, you can see the PowerUser role that you’re permitted to assume, with the trust set correctly to the parent.


...
        "PowerUserRole": {
            "Properties": {
                "AssumeRolePolicyDocument": {
                    "Statement": [
                        {
                            "Action": "sts:AssumeRole",
                            "Effect": "Allow",
                            "Principal": {
                                "AWS": "arn:aws:iam::012345678910:root"
                            }
                        }
                    ],
                    "Version": "2012-10-17"
                },
                "ManagedPolicyArns": [
                    "arn:aws:iam::aws:policy/PowerUserAccess"
                ],
                "Path": "/",
                "Policies": [],
                "RoleName": "PowerUser"
            },
            "Type": "AWS::IAM::Role"
        }
...

Deploy in your accounts

Deploy the CloudFormation templates in your favorite way, either through the console or the AWS CLI. I’ve included example CLI commands.

To use multiple accounts from the AWS CLI, configure profiles. For more information about setting up profiles, see AWS CLI.

Deploy in your central account. Substitute your –template-body –profile and –region accordingly.


aws cloudformation create-stack \
--stack-name CentralIAMPolicies \
--template-body 'file://./output_templates/central(012345678910)-IAM.template' \
--capabilities CAPABILITY_NAMED_IAM \
--profile central \
--region us-east-2

Deploy in your lab account. Substitute your –template-body –profile and –region accordingly.


aws cloudformation create-stack \
--stack-name CentralIAMPolicies \
--template-body 'file://./output_templates/lab(109876543210)-IAM.template' \
--capabilities CAPABILITY_NAMED_IAM \
--profile lab \
--region us-east-2

Explore the results

Sign into the central account with an existing role or user and set the password for the TestUser user. You could have set a password in the CloudFormation template, but then it would be in plaintext!

Sign into the central account as TestUser. Verify that you have read-only access. You should be able to self-manage your credentials though (set your own password, etc.).

Now, switch roles to the PowerUser role in the lab account. Switching roles can be done through the console. For more information, see Switching to a Role (AWS Management Console).

Confirm that you can now perform expected activities in the lab account. You should be able to create resources, as per the PowerUserAccess reference policy.

Add a new account

A new line of business has decided to use AWS and has asked for a new lab environment. You used AWS Organizations to create a new account for them. Now, bring their IAM policies up to the central standard.

Edit config.yaml and add the new account:


---
accounts:
  central:
    # Change to the ID of the central parent account.
    id: 012345678910
    parent: true
  lab:
    # Change to the ID of the child account.
    id: 109876543210
  newlab:
    # Change to the ID of the new account created using Organizations.
    id: 543210123456
...

This automatically creates the PowerUser role definition in the newlab account because you’ve used a keyword of children in the in_accounts section of the PowerUser role definition. This is a great way to assure that roles are kept consistent across all accounts. Only the users who can assume those roles change.

You also need to create a managed policy, group, and a test user in the central account to make use of the new account.

For reference your config.yaml should now look like this:


---
accounts:
  central:
    # Change to the ID of the central parent account.
    id: 012345678910
    parent: true
  lab:
    # Change to the ID of the child account.
    id: 109876543210
  newlab:
    # Change to the ID of the new account created using Organizations.
    id: 543210123456

policies:
  AssumeLabPowerUser:
    description: Allow assumption of the PowerUser role in the lab account
    assume:
      roles:
        - PowerUser
      accounts:
        - lab
    in_accounts:
      - parent

  # The new managed policy lets users assume PowerUser in the newlab account.
  AssumeNewLabPowerUser:
    description: Allow assumption of the PowerUser role in the newlab account
    assume:
      roles:
        - PowerUser
      accounts:
        # Here is the new account added in the accounts section
        - newlab
    in_accounts:
      - parent

roles:
  PowerUser:
    trusts:
      - parent
    managed_policies:
      - arn:aws:iam::aws:policy/PowerUserAccess
    in_accounts:
      # Because you use the keyword 'children' here, the newlab account gets this role
      - children

groups:
  LabUsers:
    managed_policies:
      - arn:aws:iam::aws:policy/IAMSelfManageServiceSpecificCredentials
      - arn:aws:iam::aws:policy/IAMUserChangePassword
      - arn:aws:iam::aws:policy/IAMUserSSHKeys
      - arn:aws:iam::aws:policy/IAMReadOnlyAccess
      - AssumeLabPowerUser
    in_accounts:
      - parent
  # This group services the newlab account users.
  NewLabUsers:
    managed_policies:
      - arn:aws:iam::aws:policy/IAMSelfManageServiceSpecificCredentials
      - arn:aws:iam::aws:policy/IAMUserChangePassword
      - arn:aws:iam::aws:policy/IAMUserSSHKeys
      - arn:aws:iam::aws:policy/IAMReadOnlyAccess
      # This matches our new policy above.
      - AssumeNewLabPowerUser
    in_accounts:
      - parent

users:
  TestUser:
    groups:
      - LabUsers
    in_accounts:
      - parent
  NewTestUser:
    groups:
      - NewLabUsers
    in_accounts:
      - parent

Build and deploy

Run the build:


python build.py

Now, you see a new set of CloudFormation templates in output_templates/:


ls -l output_templates/
-rw-r--r-- 1 user user Users 6307 Apr 10 10:59 central(012345678910)-IAM.template
-rw-r--r-- 1 user user Users 1680 Apr 10 10:59 lab(109876543210)-IAM.template
-rw-r--r-- 1 user user Users 1683 Apr 10 10:59 newlab(543210123456)-IAM.template

View one of the templates, and notice that the build number has changed:


...
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Build 2017-07-25Z19:35:41 - IAM Users, Groups, Roles, and Policies for account central (012345678910)",
...

The build number is the datestamp when the build was executed. It will be the same across all of the templates generated during that build. Use it to assure the environment remtains consistent. You can audit all of your accounts to assure this number is identical to assure deployment conssitency.

Again, deploy using your favorite mechanism or use the AWS CLI commands below.

In central this is an update which creates the new managed policy, group, and user.


aws cloudformation update-stack \
--stack-name CentralIAMPolicies \
--template-body 'file://./output_templates/central(012345678910)-IAM.template' \
--capabilities CAPABILITY_NAMED_IAM \
--profile central \
--region us-east-2

In lab, this is a simple version increment. Nothing else changes but keeping the version consistent ensures that your environment is consistently deployed.


aws cloudformation update-stack \
--stack-name CentralIAMPolicies \
--template-body 'file://./output_templates/lab(109876543210)-IAM.template' \
--capabilities CAPABILITY_NAMED_IAM \
--profile lab \
--region us-east-2

In newlab, this is the first deployment.


aws cloudformation create-stack \
--stack-name CentralIAMPolicies \
--template-body 'file://./output_templates/newlab(543210123456)-IAM.template' \
--capabilities CAPABILITY_NAMED_IAM \
--profile newlab \
--region us-east-2

Explore the results

You’ve enabled a new account and brought it in line with centrally managed IAM policies, with a few changes to a configuration yaml file.

As before, create a password for NewTestUser and assume the PowerUser Role in the newlab account. As a test step, confirm that you cannot assume the PowerUser role in the lab account from this user because you didn’t permit that in the config.yaml file.

Corral users into a specific region

As your use of AWS has grown, you’ve noticed that your users are creating EC2 instances in multiple regions. This is becoming difficult to manage, and you’d like to restrict their region use to us-east-2.

There is no ‘off the shelf’ managed policy for this, so you need to create one. Going through the IAM Policy Generator, create a policy that looks like the following:


{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyAllButOhio",
      "Action": "ec2:*",
      "Effect": "Deny",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "ec2:Region": "us-east-2"
        }
      }
    }
  ]
}

Update the configuration

To support the region restrictions, create a file called policy/restrictEc2Region.j2, and copy the policy json contents above into it.

Edit the config.yaml file and make some changes.

Define the new managed policy to restrict regions based on the template in the policy/ directory.


...
policies:
...
  restrictEc2Region:
    description: Prevent EC2 actions outside of us-east-2
    # Get the policy json content from the file in the policy/ directory
    policy_file: restrictEc2Region.j2
    in_accounts:
      - children
...

Modify the PowerUser role to include this region restriction policy:


...
roles:
  PowerUser:
    trusts:
      - parent
    managed_policies:
      # Add our new managed policy
      - restrictEc2Region
      - arn:aws:iam::aws:policy/PowerUserAccess
    in_accounts:
      - children
...

For reference your config.yaml should now look like this:


---
accounts:
  central:
    # Change to the ID of the central parent account.
    id: 012345678910
    parent: true
  lab:
    # Change to the ID of the child account.
    id: 109876543210
  newlab:
    # Change to the ID of the new account created using Organizations.
    id: 543210123456

policies:
  AssumeLabPowerUser:
    description: Allow assumption of the PowerUser role in the lab account
    assume:
      roles:
        - PowerUser
      accounts:
        - lab
    in_accounts:
      - parent

  # The new managed policy lets users assume PowerUser in the newlab account.
  AssumeNewLabPowerUser:
    description: Allow assumption of the PowerUser role in the newlab account
    assume:
      roles:
        - PowerUser
      accounts:
        # Here is the new account added in the accounts section
        - newlab
    in_accounts:
      - parent

  # A policy document taken from a jinja template
  restrictEc2Region:
    description: Prevent EC2 actions outside of us-east-2
    # Get the policy json content from the file in the policy/ directory
    policy_file: restrictEc2Region.j2
    in_accounts:
      - children

roles:
  PowerUser:
    trusts:
      - parent
    managed_policies:
      # Add our new managed policy
      - restrictEc2Region
      - arn:aws:iam::aws:policy/PowerUserAccess
    in_accounts:
      # Because you use the keyword 'children' here, the newlab account gets this role
      - children

groups:
  LabUsers:
    managed_policies:
      - arn:aws:iam::aws:policy/IAMSelfManageServiceSpecificCredentials
      - arn:aws:iam::aws:policy/IAMUserChangePassword
      - arn:aws:iam::aws:policy/IAMUserSSHKeys
      - arn:aws:iam::aws:policy/IAMReadOnlyAccess
      - AssumeLabPowerUser
    in_accounts:
      - parent
  # This group services the newlab account users.
  NewLabUsers:
    managed_policies:
      - arn:aws:iam::aws:policy/IAMSelfManageServiceSpecificCredentials
      - arn:aws:iam::aws:policy/IAMUserChangePassword
      - arn:aws:iam::aws:policy/IAMUserSSHKeys
      - arn:aws:iam::aws:policy/IAMReadOnlyAccess
      # This matches our new policy above.
      - AssumeNewLabPowerUser
    in_accounts:
      - parent

users:
  TestUser:
    groups:
      - LabUsers
    in_accounts:
      - parent
  NewTestUser:
    groups:
      - NewLabUsers
    in_accounts:
      - parent

Build and deploy

Build the CloudFormation templates:


python build.py

View the lab CloudFormation template. You can see the new managed policy:


...
        "restrictEc2Region": {
            "Properties": {
                "Description": "Prevent EC2 actions outside of us-east-2",
                "Groups": [],
                "PolicyDocument": {
                    "Statement": [
                        {
                            "Action": "ec2:*",
                            "Condition": {
                                "StringNotEquals": {
                                    "ec2:Region": "us-east-2"
                                }
                            },
                            "Effect": "Deny",
                            "Resource": "*",
                            "Sid": "DenyAllButOhio"
                        }
                    ],
                    "Version": "2012-10-17"
                },
                "Roles": [],
                "Users": []
            },
            "Type": "AWS::IAM::ManagedPolicy"
        }
...

You can also see that PowerUser definition now includes the following:


...
                "ManagedPolicyArns": [
                    "arn:aws:iam::aws:policy/PowerUserAccess",
                    {
                        "Ref": "restrictEc2Region"
                    }
                ],
...

If you look at newlab, you see the same set of definitions are there too. If you had tens or even hundreds of child accounts, those would all reflect the new policy and updated role definitions as well. This scales very nicely!

Now, deploy to your three accounts.

In central, nothing actually changes. It’s just a deployment version update.


aws cloudformation update-stack \
--stack-name CentralIAMPolicies \
--template-body 'file://./output_templates/central(012345678910)-IAM.template' \
--capabilities CAPABILITY_NAMED_IAM \
--profile central \
--region us-east-2

In lab and newlab, this deploys the new managed policy, and update the PowerUser role.


aws cloudformation update-stack \
--stack-name CentralIAMPolicies \
--template-body 'file://./output_templates/lab(109876543210)-IAM.template' \
--capabilities CAPABILITY_NAMED_IAM \
--profile lab \
--region us-east-2


aws cloudformation update-stack \
--stack-name CentralIAMPolicies \
--template-body 'file://./output_templates/newlab(543210123456)-IAM.template' \
--capabilities CAPABILITY_NAMED_IAM \
--profile newlab \
--region us-east-2

Conclusion

I hope you find this tool as useful as I have. The tool is capable of much more complex deployments as well, which leverage the power of Jinja2 templating. Explore the contents of sample_configs/config-complex.yaml and the policy examples in the sample_policy/ directory in the project.

The README.md file has more detailed explanations of functionality and syntax usage.

About the Author


Adam McLean is a Cloud Infrastructure Architect with AWS Professional Services. Adam enjoys helping his Canadian enterprise customers achieve success on their cloud journey. In his spare time, Adam enjoys spending time with his family and three small children.