Managing AWS Organizations using the open source org-formation tool — Part 2

This article is a guest post from Olaf Conijn, the creator of org-formation.

• Part 1: Managing AWS Organizations resources using infrastructure as code
• Part 2: Integrating management of resources across accounts using task files
• Part 3: Deploying CloudFormation resources to multiple accounts using Organization Bindings
• org-formation on GitHub

In the first part of this series on managing AWS Organizations using org-formation, we covered the basics of how to use org-formation, how to initialize an organization.yml file on disk, and how to make changes using the update command. In Part 2, we will take managing Organizations to the next level by describing how the update process can be automated using AWS CodePipeline.

Creating a CodePipeline using org-formation

Because Infrastructure as Code (IaC) is particularly useful when stored in source control, and applied automatically upon commit (or merge), org-formation has a command to set up such a pipeline in AWS. Running the init-pipeline command is a lot like the init command, but instead of creating a file on disk, it creates AWS CodeCommit, AWS CodeBuild, and CodePipeline resources. It also creates an initial check-in that contains the organization.yml file for the organization and all other files needed to deploy changes to this file automatically.

\> org-formation init-pipeline organization.yml --region eu-central-1

This command will create an initial commit to the CodeCommit repository containing the following files:

<repository root>
├── templates
│   └── org-formation-build.yml
├── buildspec.yml
├── organization-tasks.yml
└── organization.yml

• organization.yml is the file we created on our local disk using the init command.
• buildspec.yml contains instructions to run org-formation upon every check-in to main. Running org-formation using the perform-tasks command allows us to run any number of tasks from within a tasks file.
• organization-tasks.yml is an org-formation tasks file that contains two tasks: a task of type update-organization, which is used to apply all changes made to the organization.yml file (if any); and an update-stacks task that can be used to change the pipeline itself.
• templates/org-formation-build.yml contains the AWS CloudFormation template used to create the AWS resources and can be used to modify these.

Contents of the organization-tasks.yml file, as generated by the init-pipeline command:

OrganizationUpdate:
  Type: update-organization
  Template: ./organization.yml

OrganizationBuild:
  Type: update-stacks
  Template: ./templates/org-formation-build.yml
  StackName: organization-formation-build
  Parameters:
    stateBucketName: organization-formation-111222333444
    resourcePrefix: orgformation
    repositoryName: organization-formation
  DefaultOrganizationBindingRegion: eu-central-1
  DefaultOrganizationBinding:
    IncludeMasterAccount: true

Automating deployments using task files

New AWS accounts within an organization typically also come with a basic set of resources created within these accounts. Therefore, updating our organization is likely a process with multiple steps. To do this, org-formation has a command called perform-tasks. We can run perform-tasks to execute tasks we would like to be part of the organization build pipeline.

The task file needs to contain at least one update-organization task that will be executed before all other tasks. If other tasks reference an organization.yml file, this file must always be the same file specified in the update-organization task.

A task file can contain the following task types:

• update-organization, a task used to update the organization resources in the main account
• update-stacks, a task used to create/update CloudFormation templates in the accounts that are part of the organization
• include, a task used to include another tasks file
• update-cdk, a task used to execute an AWS CDK project in the accounts that are part of the organization
• update-serverless.com, a task used to execute a Serverless.com project in the accounts that are part of the organization
• copy-to-s3, a task used to copy a local file to Amazon S3

An example of a task file may look like the following:

OrganizationUpdate:
  Type: update-organization
  Template: ./organization.yml

UpdateStack:
  Type: update-stacks
  Template: ./templates/mytemplate.yml
  StackName: my-stack-name
  DefaultOrganizationBindingRegion: eu-west-1
  DefaultOrganizationBinding:
    Account: '*'

IncludeOther:
  DependsOn: UpdateStack
  Type: include
  Path: ./included.yml

CdkWorkload:
  Type: update-cdk
  DependsOn: UpdateStack
  Path: ./workload/
  RunNpmInstall: true
  RunNpmBuild: true
  OrganizationBinding:
    Account: !Ref AccountA
    Region: eu-central-1

We can specify all tasks with the following attributes:

• Use DependsOn to have a task run only after the task(s) specified here have executed successfully.
• Use Skip to skip the execution of a task (when set to true). Tasks that depend on this using DependsOn will be skipped (unless explicitly unskipped).
• Use TaskRoleName to specify the name of the AWS IAM role that will be assumed in the target account when performing the task.
• Use LogVerbose to print debug-level logging when executing a particular task.

Note that the perform-tasks command has options to run multiple tasks concurrently. It also has options to specify a tolerance for failures on both tasks and stacks. If we are into speeding up our deployment, try adding the option --max-concurrent-stacks 10 when executing perform-tasks. If we want the perform-tasks to continue even after a number of tasks have failed, we can add the option --failed-tasks-tolerance 5. Tasks that depend on tasks that have failed will not be executed and considered failed. Both options can also be specified on a task with type include.

Organization Bindings

A concept at the core of org-formation is the Organization Binding. The Organization Binding allows us to specify a number of target accounts (and regions) and update them all at once. Annotated CloudFormation templates can use multiple Organization Bindings and specify exactly where to deploy needed resources.

An Organization Binding always specifies both the target accounts and target regions. The targets that are used are all the possible combinations of regions and accounts. For example, an Organization Binding with two regions and three accounts will have six targets, but an Organization Binding with zero regions and six accounts will not have any targets.

Because Annotated CloudFormation templates can have multiple bindings, there is the option to specify a default set of regions using DefaultOrganizationBindingRegion. This prevents us from forgetting to specify a region and not having the resources deployed anywhere.

An Organization Binding can have the following attributes:

• Region used to specify the region(s) for which this binding needs to create targets.
• Account used to include a specific account, or list of accounts, that this binding needs to create targets for. We can also use * to specify all accounts, except for the main account.
• IncludeMasterAccount used to include the MasterAccount in the targets (when value is true).
• OrganizationalUnit used to include accounts from an Organizational Unit (or list of Organizational Units).
• AccountsWithTag used to include all accounts that declare a specific tag in the organization file.
• ExcludeAccount used to exclude a specific account (or list of accounts) from the targets.

All references use the logical names as declared in the organizational.yml file, and accounts that are not part of the organizational model are not used to create a target for.

Examples of Organization Bindings

Simple list of accounts in eu-west-1:

OrganizationBinding:
  Region: eu-west-1
  Account:
    - !Ref Account1
    - !Ref Account2

All accounts in an organization (including the main account) in both eu-west-1 and eu-central-1:

OrganizationBinding:
  Region:
    - eu-west-1
    - eu-central-1
  Account: '*'
  IncludeMasterAccount: true

All accounts part of the development OU, except for the SandboxAccount:

OrganizationBinding:
  Region: eu-west-1
  OrganizationalUnit: development
  ExcludeAccount: !Ref SandboxAccount

All accounts that declare a subdomain tag:

OrganizationBinding:
  Region: eu-central-1
  AccountsWithTag: subdomain

Variables and Parameters in the task file

From within the task file, it is possible to reference attributes from the organization.yml using !Ref, !GetAtt, and !Sub (or any combination). This can be useful if we want to parameterize the tasks (or Parameters of a task) using information in our organization.

For example:

SomeTemplate:
  Type: update-stacks
  Template: ./cloudtrail.yml
  StackName: variables-example
  StackDescription: !Sub
  - 'CloudTrail implementation ${account} with events persisted to ${persistanceAccount}'
  - { account: !Ref CurrentAccount, persistanceAccount: !Ref ComplianceAccount}
  DefaultOrganizationBindingRegion: eu-west-1
  DefaultOrganizationBinding:
    IncludeMasterAccount: true
  Parameters:
    accountForPersistance: !Ref ComplianceAccount
    enable: !GetAtt CurrentAccount.Tags.enableCloudTrail

We can refer to any account in the organization.yml by its logical name. Refer to the account that is part of the current task and target (when executing the task) by CurrentAccount. We can declare custom parameters in a top-level Parameters attribute in the task file. Parameters can have default values specified in the template and be overwritten by adding a --parameters option to the perform-tasks command.

Declaring and specifying parameter values when running the perform-tasks command:

\> org-formation perform-tasks organization-tasks.yml --parameters stackPrefix=test includeMasterAccount=false
Parameters:
  stackPrefix:
    Description: will be used as a prefix for stack names.
    Type: String
    Default: my

  includeMasterAccount:
    Description: if true the bucket template will be deployed to the main account
    Type: Boolean
    Default: true

OrganizationUpdate:
  Type: update-organization
  Template: ./organization.yml

BucketTemplate:
  Type: update-stacks
  Template: ./bucket.yml
  StackName: !Sub ${stackPrefix}-scenario-stack-parameters
  DefaultOrganizationBindingRegion: eu-west-1
  DefaultOrganizationBinding:
    IncludeMasterAccount: !Ref includeMasterAccount

IncludeOther:
  DependsOn: BucketTemplate
  Type: include
  Path: ./included-task-file.yml
  Parameters:
    stackPrefix: other-prefix

In the previous example, we demonstrated how the parameters are:

• Used in a StackName attribute to avoid colliding stack names when re-using or testing the tasks file.
• Used in an OrganizationBinding to conditionally include the MasterAccount.
• Passed down to an include task. If nothing is specified in the Parameters attribute of the include task, parameter values from the parent task file are passed down to included task files. In the example, we assigned the parameter stackPrefix a specific value in the included task file. However, the value from includeMasterAccount will remain the same.

In addition to organization attributes and parameters, CloudFormation exports can be queried using the !CopyValue function. As opposed to CloudFormations native !ImportValue function, the stack (and the resources within the stack) that declares the output can also be deleted after the value was copied from the export. We can use!CopyValue cross account and cross region, however !ImportValue only works within the same account and region.

The following four examples demonstrate how a task (called PolicyTemplate) uses a value exported by another task (called BucketTemplate) and assigns it to a parameter.

BucketTemplate:
  Type: update-stacks
  Template: ./bucket.yml
  StackName: scenario-export-bucket
  DefaultOrganizationBindingRegion: eu-west-1
  DefaultOrganizationBinding:
    IncludeMasterAccount: true

PolicyTemplate:
  DependsOn: BucketTemplate
  Type: update-stacks
  Template: ./bucket-policy.yml
  StackName: scenario-export-bucket-role
  DefaultOrganizationBindingRegion: eu-west-1
  DefaultOrganizationBinding:
    IncludeMasterAccount: true
  Parameters:
    bucketArn: !CopyValue 'BucketArn'
    bucketArn2: !CopyValue ['BucketArn', !Ref MasterAccount]
    bucketArn4: !CopyValue ['BucketArn', !Ref MasterAccount, 'eu-west-1']
    bucketArn3: !CopyValue ['BucketArn', 123123123123, 'eu-west-1']

The !CopyValue function can declare up to three arguments:

• ExportName, the first argument, must contain the name of the export of which the value needs to be resolved.
• AccountId, the second argument will, if specified, contain the account ID of the account that declares the export. This can be either a hard coded AccountId (12 digits), or !Ref to a logical account name in the organization file that will resolve to the account ID when processing the task file.
• Region, the third argument will, if specified, contain the region that declares the export.

If we do not specifyAccountId and/or Region, the account and region of the target are used. If we have an Organization Binding with six targets and do not specify AccountId or Region, we will find the export in all six targets (Account/Region combinations).

Protecting critical resources

There are several ways we can protect critical resources deployed by org-formation. The update-stacks tasks allows us to set the TerminationProtection attribute to true to prevent a template from being deleted; and sets the UpdateProtection attribute to true, thus preventing any of the resources within the template from updating using CloudFormation.

The following is an example of using TerminationProtection and UpdateProtection attributes:

CriticalResourcesTemplate:
  Type: update-stacks
  Template: ./bucket.yml
  StackName: critical-stack
  TerminationProtection: true
  UpdateProtection: true
  DefaultOrganizationBindingRegion: eu-west-1
  DefaultOrganizationBinding:
    Account: !Ref Production1

TerminationProtection will cause any call to delete the stack to fail (through the CloudFormation console or org-formation). The UpdateProtection will cause updates of any resource using CloudFormation to fail. This feature uses a CloudFormation StackPolicy that can also be specified explicitly using the StackPolicy attribute.

TerminationProtection, UpdateProtection, and StackPolicy only apply to changes made using CloudFormation. The resources can still be modified directly in the console or using an API. If we want to ensure resources in the accounts remain unchanged, we can specify this as a service control policy in the organization.yml file.

An example service control policy that prevents modifying an IAM role called ProtectedRole:

RestrictUpdatesOnIAMRoles:
  Type: OC::ORG::ServiceControlPolicy
  Properties:
    PolicyName: RestrictUpdatesOnIAMRoles
    PolicyDocument:
      Version: '2012-10-17'
      Statement:
        - Sid: RestrictIamChanges
          Effect: Deny
          Action:
          - 'iam:Update*'
          - 'iam:Put*'
          - 'iam:Delete*'
          - 'iam:Attach*'
          - 'iam:Detach*'
          Resource: 'arn:aws:iam::*:role/ProtectedRole'

The previous example policy will prevent anyone in the organization (including root) from changing the ProtectedRole resource in any of the applicable accounts. If we only want to allow the Organization Build to change these resources, we can add a Condition to the service control policy:

RestrictUpdatesOnIAMRoles:
  Type: OC::ORG::ServiceControlPolicy
  Properties:
    PolicyName: RestrictUpdatesOnIAMRoles
    PolicyDocument:
      Version: '2012-10-17'
      Statement:
        - Sid: RestrictIamChanges
          Effect: Deny
          Action:
          - 'iam:Update*'
          - 'iam:Put*'
          - 'iam:Delete*'
          - 'iam:Attach*'
          - 'iam:Detach*'
          Resource: 'arn:aws:iam::*:role/ProtectedRole'
          Condition:
            StringNotLike:
              aws:PrincipalARN: arn:aws:iam::*:role/OrganizationAccountAccessRole

Conclusion

In Part 2 of this series, we learned how to create a continuous deployment pipeline for changes to our AWS Organizations. We also learned what task files are, how they can be parameterized, and what we can do to prevent resource modification from outside the pipeline.

In the final installment of this series, we will learn about annotation we can add to the CloudFormation language, parameterizing templates with attributes defined in our AWS Organizations, and creating references between resources across different AWS accounts and regions.

• Part 1: Managing AWS Organizations resources using infrastructure as code
• Part 2: Integrating management of resources across accounts using task files
• Part 3: Deploying CloudFormation resources to multiple accounts using Organization Bindings
• org-formation on GitHub

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.