Overview

Learn about the best practices for introducing changes to configurations of AWS Services. Changes include updating s CloudFront Function code, changing the runtime of a Lambda@Edge function, adding a new route (i.e. cache behavior in CloudFront), updating WAF rules, and enabling a new release CloudFront feature such as HTTP/3. Customers who have relatively static and simple configuration, and prefer a UI to manually manage AWS Edge services, can rely on the AWS Console for this purpose. Otherwise, customers can use their CI/CD pipelines to deploy changes to their configurations and code in a controlled way.

CI/CD pipeline

Infrastructure as a code (IaaC)

CI/CD pipelines improve software release cycles by increasing development velocity, delivering higher code quality, and reducing human error with automation. AWS Edge services such as CloudFront, and edge functions can be managed in a CI/CD pipeline, with their configuration as code (IaaC). You can deploy edge resources using APIs (e.g. CloudFront's API), using the AWS CLI, or using tools with higher level of abstraction such as CloudFormation, Terraform and CDK.

CDK based IaaC

CDK, is based on CloudFormation, but provides you with the highest level of abstraction. It allows you to deploy AWS resources using the power of a programming language. For example, the following 3 lines of code allows you to deploy an S3 bucket, with CloudFront as origin. Consider this portal for CDK constructs that you can reuse in your CDK code.

const myBucket = new s3.Bucket(this, 'myBucket');
new cloudfront.Distribution(this, 'myDist', {
  defaultBehavior: { origin: new origins.S3Origin(myBucket) },
});

Deployment and orchestration

In your CI/CD pipeline, you need a repository such as AWS CodeCommit to store your code (CDK code, edge functions code), a tool such as AWS CodeDeploy to deploy your infrastructure, and a tool such as AWS CodePipeline to orchestrate the steps of your CI/CD pipeline. This blog is an example of using AWS to AWS developer tools to implement a CI/CD pipeline for CloudFront and Lambda@Edge, however, you can also use your preferred CI/CD tooling. Note that this blog is outdated, and it is only used for illustration. Consider the following limitation when you are building your CI/CD pipeline for AWS Edge services:

  • CloudFront, CloudFront Functions, and regional AWS WAF WebACLs can be deployed from any AWS Region
  • Lambda@Edge, and WAF WAF WebACLs for CloudFront can only be deployed from us-east-1 region in N Virginia.
  • Firewall Manager policies must be deployed in the Firewall Manager administrator AWS Account

Testing and staging

Your edge configuration can be tested at different levels.

First, you can replicate your infrastructure, such as a CloudFront distribution with a AWS WAF WebACL in testing accounts. This could be done during the development phase or for running automated functional tests before moving to next step in production. It's important to note that you can't use two CloudFront distributions configured with the same CNAME. As a consequence, you need to differentiate the CNAME configuration in the deployment of your CloudFront resource according to the CI/CD phase. For example, you can use CloudFront's domain name (xyz.cloudfront.net) for development phase, then a dedicated CNAME for staging (staging.www.example.com) and finally the public CNAME for deploying to production (www.example.com). Note that you can also differentiate your security controls by stage, such as restricting access to specific IPs using AWS WAF for staging phase.

After your new configuration passes testing steps, you can implement a canary release approach using CloudFront's continuous deployment feature to test the change in production on a small portion of your traffic. Using this feature, you can create a different configuration of your production distribution, and only send a portion of your global traffic to it. You have two possibilities to route traffic to the new configuration: Using a a specific request header, which is usefull for synthetic testing, or using a weighted policy to route a configurable percentage of traffic to the new configuration, with the option of enabling stickiness. In the current version of this feature, you can only test changes on a subset parameters of the CloudFront configuration. Read the documentation to better understand what you can test using this feature.

Dynamic configuration

Consider the case when you have multiple teams introducing changes to the configuration of CloudFront, but you have a single team managing the CI/CD pipeline for all teams. An example would be the scenario where different teams manage different microservices, all exposing their APIs on the main domain hosted by a single CloudFront distribution. If you allow each team to access the full configuration of the CloudFront distribution, you increase the blast radius of a single bad change. On the opposite side, if you only allow the CI/CD team to make changes to the CloudFront distribution, you decrease the development velocity and introduce a bottleneck in your release lifecycle. Instead, you can generate configuration dynamically based on partial configurations owned by each micro-service team. In a simple implementation, each team manages the configuration of their own route (caching, edge functions, etc..) in a text-based file they own, hosted in an S3 bucket. In your CI/CD pipeline, you can introduce an additional step to create dynamically the final CloudFront configuration using the different partial configuration files.

WAF and Shield deployment

Shield and WAF services can be deployed using the same tools described earlier, in a CI/CD pipeline. Note that Shield is not supported natively in CloudFormation, but using a Custom Resource, you can deploy Shield using CloudFormation.

However, it's recommended to use AWS Firewall Manager for deploying WAF rules and Shield protections for the following advantages:

  • It facilitates the enforcement of a central security governance, (e.g. deployment of central rules in combination of rules managed by application teams)
  • It is faster to deploy, which is important for patching critical vulnerabilities using AWS WAF
  • It simplifies the deployment across accounts and across heterogeneous CI/CD pipelines (e.g. inherited from acquisitions) However, with this approach, you need to manage drift if you are using a CI/CD pipeline with drift detection.
    In addition, you can use, like OLX, a CI/CD pipeline to make changes to your Firewall Manager policies in the administrator account, which will be deployed across your AWS organization by Firewall Manager.

Deploying WAF rules safely to your production can be done using different strategies. You can add new rules to an existing WebACL in count mode, and then change it to block mode. However, with this approach, you need to pay attention to the maximum WCU of your WebACL. Another approach would be deploying changes to a staging WebACL, and test the changes in a staging environment.

Resources

Was this page helpful?