AWS DevOps & Developer Productivity Blog

Exploring Fn::ForEach and Fn::FindInMap enhancements in AWS CloudFormation

AWS CloudFormation, an Infrastructure as Code (IaC) service that lets you model, provision, and manage AWS and third-party resources, recently released a new language transform that enhances the core CloudFormation language. Today, we’ll be covering two more enhancements we’ve added since our initial release: Fn::FindInMap enhancements and a new looping function – Fn::ForEach.

These new language extensions are the result of open discussions with the larger CloudFormation community via our Request For Comments (RFC) proposals for new language features at our Language Discussion GitHub repository. We want to collaborate with the community to better align features and incorporate early feedback into the development cycle to meet the community’s needs. We invite you to participate in new RFCs to help shape the future of the CloudFormation language.

In this post, I’ll dive deeper into the new enhancements for Fn::FindInMap as well as explore the new Fn::ForEach looping mechanism and provide some examples.

Prerequisites

To use these new language features, you must add AWS::LanguageExtensions to the transform section of your template.

---
AWSTemplateFormatVersion: 2010-09-09
Transform: 'AWS::LanguageExtensions'

If you have a list of transforms, then we recommend having AWS managed transforms at the end, and AWS::LanguageExtensions must be listed before AWS::Serverless.

---
AWSTemplateFormatVersion: 2010-09-09
Transform: 
  - 'AWS::LanguageExtensions'
  - 'AWS::Serverless-2016-10-31'

This transform will cover all of the existing and future language extensions.

FindInMap enhancements

We have updated the language extension transform for CloudFormation to support Fn::FindInMap enhancements, that extend the existing functionality of the Fn::FindInMap intrinsic function so that now you can:

  • use an optional, default value in Fn::FindInMap parameters, if a given key in a Mappings section is not found, and
  • use a number of additional intrinsic functions in the parameters of Fn::FindInMap; for more information, see Supported functions.

Let’s see an example use case where Fn::FindInMap enhancements can help you simplify the business logic of your template, and make it more readable and easier to maintain. Let’s suppose you create a CloudFormation template that describes an Amazon Elastic Compute Cloud (Amazon EC2) instance, and you need to use smaller EC2 instance types for pre-production environments, and a larger EC2 instance type for production for cost savings. In this example, you choose a t2.micro instance type for the dev environment, t2.medium for the qa environment, and t2.large for the prod environment, that you start to describe as follows:

---
AWSTemplateFormatVersion: "2010-09-09"

Description: 'Sample template that describes usage for `Fn::FindInMap` enhancements'

Parameters:
  Environment:
    Description: Lifecycle environment.
    Type: String
    AllowedValues:
      - sandbox
      - dev
      - qa
      - prod
    Default: dev

  LatestAmiId:
    Description: Region-specific image to use.
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2

Mappings:
  LifecycleEnvToInstanceType:
    dev:
      InstanceType: t2.micro
    qa:
      InstanceType: t2.medium
    prod:
      InstanceType: t2.large

You described instance types for each of your 3 lifecycle environments in the Mappings section, and you engineered your template to read environment names as input data from Environment in the Parameters section. Looking closer at Environment, you define another allowed value: sandbox, that in this example is an environment for developers to use for prototype testing only: you choose not to include this environment in the mapping you created, with the intent to do the same for any other non-formal environment (for example, a contributor’s personal environment). Next, powered by the new enhancements toFn::FindInMap, you assign a default value for environment names that are different than dev, qa, and prod; this way, the only change you’ll need to make in this context is a new value(s) to AllowedValues in Environment. You describe this business logic in your template, to which you add the sample code shown next:

Transform: AWS::LanguageExtensions

Resources:
  Ec2Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref 'LatestAmiId'
      InstanceType: !FindInMap
        - LifecycleEnvToInstanceType
        - !Ref 'Environment'
        - InstanceType
        - DefaultValue: t2.micro
      Tags:
        - Key: test
          Value: test

In the snippet above, you have declared the AWS::LanguageExtensions transform, and described your configuration for an EC2 instance in the Resources section. For InstanceType, you chose to use Fn::FindInMap enhancements, and pass DefaultValue as an additional parameter with t2.micro as its value. When the user uses this template to create a stack, and chooses sandbox for Environment, the !Ref 'Environment' reference to the value for Environment will evaluate to sandbox, which is not present in the mapping you created: in this case, t2.micro will be used as a value for InstanceType.

These new enhancements also allow you to use more intrinsic functions inside of Fn::FindInMap. Let’s say you have received requirements to use -env as a suffix to environment names. You choose to make a minimal set of changes to your template, and start with the Parameters section as follows:

Parameters:
  Environment:
    Description: Lifecycle environment.
    Type: String
    AllowedValues:
      - sandbox-env
      - dev-env
      - qa-env
      - prod-env
    Default: dev-env

Next, instead of modifying all of your keys in the Mappings section, you choose to only change the second parameter to Fn::FindInMap enhancements as follows: first, you use the Fn::Split intrinsic function to split the user-selected environment value string (for example, dev-env) into a list of values using the ‘-‘ character as a delimiter, and next you use the Fn::Select intrinsic function to choose the first element (that is, 0) of that list:

Resources:
  Ec2Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref 'LatestAmiId'
      InstanceType: !FindInMap
        - LifecycleEnvToInstanceType
        - !Select
          - 0
          - !Split
            - '-'
            - !Ref 'Environment'
        - InstanceType
        - DefaultValue: t2.micro
      Tags:
        - Key: test
          Value: test

With the updated code above, if the user selects the dev-env value for Environment, Fn::FindInMap enhancements will use dev as the second parameter when looking up values in the Mappings section.

Fn::ForEach intrinsic function

Another enhancement to the language extensions is the addition of native looping inside of CloudFormation with Fn::ForEach. Imagine you have a situation where you need three EC2 instances that look exactly the same. Currently, you would have to copy and paste each instance as a separate resource with CloudFormation:

---
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::LanguageExtensions

Resources:
  FirstInstance:
    Type: AWS::EC2::Instance
    Properties: 
      # ..removed for brevity..

  SecondInstance:
    Type: AWS::EC2::Instance
    Properties: 
      # ..removed for brevity..

  ThirdInstance:
    Type: AWS::EC2::Instance
    Properties: 
      # ..removed for brevity..

If you encounter the need to update one property (the AMI ID, for example), you will have to update all three separately. While this is trivially easy for our example, templates that extend into the hundreds of resources quickly becomes difficult to maintain.

With the Fn::ForEach language extension, we’re able to group all of these items together into an easy-to-manage snippet:

---
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::LanguageExtensions

Resources:
  Fn::ForEach::Instances:
    - InstanceLogicalId
    - [FirstInstance, SecondInstance, ThirdInstance]
    - ${InstanceLogicalId}:
        Type: AWS::EC2::Instance
        Properties: 
          # ..removed for brevity..

This results in the following output YAML, which is identical to our previous example:

---
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::LanguageExtensions

Resources:
  FirstInstance:
    Type: AWS::EC2::Instance
    Properties: 
      # ..removed for brevity..

  SecondInstance:
    Type: AWS::EC2::Instance
    Properties: 
      # ..removed for brevity..

  ThirdInstance:
    Type: AWS::EC2::Instance
    Properties: 
      # ..removed for brevity..

To break down the syntax, the Fn::ForEach function requires:

  • A Logical ID for the looping function directly following the Fn::ForEach call. In our case, we named it Instances
  • The variable name we’ll be referencing in our snippet below
  • The collection of strings we’ll be iterating over. You can write these inline, or pass them as parameters or mappings.
  • A section of the template we’ll be iterating over using the variable name above. This is standard CloudFormation JSON/YAML.

These must be listed in an array immediately following the Fn::ForEach intrinsic function and in this exact order.

We can use our key to reference values found elsewhere in the template. This adds additional flexibility when combined with the aforementioned FindInMap enhancements. Imagine a similar scenario where each instance needs a specific instance type, dictated by which instance it is and which environment we’re in. As described before, we would add our parameters and mappings to our template:

---
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::LanguageExtensions

Parameters:
  Environment:
    Description: Lifecycle environment.
    Type: String
    AllowedValues:
      - sandbox
      - dev
      - qa
      - prod
    Default: dev

  LatestAmiId:
    Description: Region-specific image to use.
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2

Mappings:
  dev:
    FirstInstance:
      InstanceType: t2.micro
    SecondInstance:
      InstanceType: t2.micro
    ThirdInstance:
      InstanceType: t2.micro
  qa:
    FirstInstance:
      InstanceType: t2.medium
    SecondInstance:
      InstanceType: t2.medium
    ThirdInstance:
      InstanceType: t2.large
  prod:
    FirstInstance:
      InstanceType: t2.large
    SecondInstance:
      InstanceType: t2.xlarge
    ThirdInstance:
      InstanceType: t2.2xlarge

Given this configuration, we have different environment values as Parameters and a Mapping section that details our sizing requirements for our instance. With this, we can then use our new Fn::ForEach functionality and FindInMap enhancements:

Resources:
  Fn::ForEach::Instances:
    - InstanceLogicalId
    - [FirstInstance, SecondInstance, ThirdInstance]
    - ${InstanceLogicalId}:
        Type: AWS::EC2::Instance
        Properties: 
          ImageId: !Ref LatestAmiId
          InstanceType: !FindInMap
            - !Ref Environment
            - !Ref InstanceLogicalId
            - InstanceType
            - DefaultValue: t2.micro

This results in the following output:

Resources:
  FirstInstance:
    Type: AWS::EC2::Instance
    Properties: 
      ImageId: !Ref LatestAmiId
      InstanceType: !FindInMap
        - !Ref Environment
        - FirstInstance
        - InstanceType
        - DefaultValue: t2.micro

  SecondInstance:
    Type: AWS::EC2::Instance
    Properties: 
      ImageId: !Ref LatestAmiId
      InstanceType: !FindInMap
        - !Ref Environment
        - SecondInstance
        - InstanceType
        - DefaultValue: t2.micro

  ThirdInstance:
    Type: AWS::EC2::Instance
    Properties: 
      ImageId: !Ref LatestAmiId
      InstanceType: !FindInMap
        - !Ref Environment
        - ThirdInstance
        - InstanceType
        - DefaultValue: t2.micro

This looping feature can be used to create more than just resources – say we want to reference outputs from these EC2 instances as we create them. We can modify our above template to add an Output section and iterate over it in the same way, as well as exporting the instance ID. We can even express more than one Output per iteration. We’ll also move the instances list to a parameter for increased clarity.

Parameters:
  InstancesToManage:
    Type: CommaDelimitedList
    Description: Instances to be managed
    Default: FirstInstance,SecondInstance,ThirdInstance

Outputs:
  Fn::ForEach::InstanceOutputs:
    - InstanceLogicalId
    - !Ref InstancesToManage
    - "${InstanceLogicalId}Id": 
        Export: 
          Name: !Sub ${AWS::AccountId}-${InstanceLogicalId}Id
        Value: !Ref 
          Ref: InstanceLogicalId
        
      "${InstanceLogicalId}AvailabilityZone":
        Value:
          Fn::GetAtt: 
            - !Ref InstanceLogicalId
            - AvailabilityZone

This outputs to:

Outputs:
  FirstInstanceId: 
    Export: 
      Name: !Sub ${AWS::AccountId}-FirstInstanceId
    Value: !Ref FirstInstance
  
  FirstInstanceAvailabilityZone:
    Value:
      Fn::GetAtt:
        - FirstInstance
        - AvailabilityZone
  
  SecondInstanceId: 
    Export: 
      Name: !Sub ${AWS::AccountId}-SecondInstanceId
    Value: !Ref SecondInstance
  
  SecondInstanceAvailabilityZone:
    Value:
      Fn::GetAtt:
        - SecondInstance
        - AvailabilityZone

  ThirdInstanceId: 
    Export: 
      Name: !Sub ${AWS::AccountId}-ThirdInstanceId
    Value: !Ref ThirdInstance
  
  ThirdInstanceAvailabilityZone:
    Value:
      Fn::GetAtt:
        - ThirdInstance
        - AvailabilityZone

In this snippet, we iterated over our collection and created multiple outputs. For each output, we concatenated our key with some other string. In this case, both Id and AvailabilityZone were concatenated with the key to create a unique output name based on the stack name and the logical ID of the resource.

Finally, loops can be nested inside other loops. Combined with the ability to concatenate values and do lookups inside of the Mapping section, we’re able to significantly simplify complex CloudFormation templates. Imagine an example where we are tasked with creating a Virtual Private Cloud (VPC) with three private subnets and three public subnets. This is a common configuration our customers have and we can configure it simply with looping.

---
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::LanguageExtensions

Parameters:
  AvailabilityTypes:
    Type: CommaDelimitedList
    Description: Types of subnets availability - public, private, or both
    AllowedValues:
      - Public
      - Private
    Default: Public,Private

Mappings:
  SubnetOne:
    Public: 
      Cidr:  10.215.0.0/24 
    Private: 
      Cidr:  10.215.1.0/24 
  SubnetTwo:
    Public: 
      Cidr:  10.215.2.0/24
    Private: 
      Cidr:  10.215.3.0/24
  SubnetThree:
    Public: 
      Cidr:  10.215.4.0/24
    Private: 
      Cidr:  10.215.5.0/24

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.215.0.0/16
      EnableDnsSupport: true
      EnableDnsHostnames: true

  Fn::ForEach::Subnets:
    - SubnetIdentifier
    - - SubnetOne
      - SubnetTwo
      - SubnetThree
    - Fn::ForEach::SubnetAvailabilityType:
      - AvailabilityType
      - !Ref AvailabilityTypes
      - "${SubnetIdentifier}${AvailabilityType}":
          Type: AWS::EC2::Subnet
          Properties:
            VpcId: !Ref VPC
            CidrBlock: !FindInMap
              - !Ref SubnetIdentifier
              - !Ref AvailabilityType
              - Cidr

which outputs the following resource section:

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCidr
      EnableDnsSupport: true
      EnableDnsHostnames: true

  SubnetOnePublic:
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !FindInMap
        - SubnetOne
        - Public
        - Cidr

  SubnetOnePrivate:
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !FindInMap
        - SubnetOne
        - Private
        - Cidr

  SubnetTwoPublic:
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !FindInMap
        - SubnetTwo
        - Public
        - Cidr

  SubnetTwoPrivate:
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !FindInMap
        - SubnetTwo
        - Private
        - Cidr

  SubnetThreePublic:
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !FindInMap
        - SubnetThree
        - Public
        - Cidr

  SubnetThreePrivate:
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !FindInMap
        - SubnetThree
        - Private
        - Cidr

Combining everything we’ve learned so far, this created six subnets total, three public, three private, and attached them to the respective VPC with a relevant CIDR block.

We’re excited to share this functionality with our community, and we invite you to share feedback on future enhancements to the looping functionality here. A few enhancements we’re discussing are:

  • Iterating over a key/value pair
  • Iterating over a list of lists
  • Support in other template sections
  • And more!

Please head over and let us know what you think!

Conclusion

In this post, we walked through the new CloudFormation additions to the language extensions transform, how to enable them in your templates, and how to engage in future language extensions via our open language discussion repository. Leave us your feedback at our Language Discussion GitHub repository to help shape the future of the CloudFormation language. We look forward to hearing from you!

About the Author:

Dan Blanco

Dan is a senior AWS Developer Advocate based in Atlanta for the AWS IaC team. When he’s not advocating for IaC tools, you can either find him in the kitchen whipping up something delicious or flying in the Georgia sky. Find him on twitter (@TheDanBlanco) or in the AWS CloudFormation Discord server.