AWS Edge Services - Change management

This document outlines best practices for introducing changes to the configurations of AWS Edge Services, such as CloudFront, Lambda@Edge, and AWS WAF. These changes may include updating CloudFront Function code, modifying the runtime of a Lambda@Edge function, adding a new cache behavior in CloudFront, updating WAF rules, or enabling new CloudFront features like HTTP/3. If you have a relatively static and simple configurations and prefer a UI for manual management, you can rely on the AWS Console. However, for more complex setups, it is recommended to leverage CI/CD pipelines to deploy configuration changes and code updates in a controlled manner.

CI/CD pipelines enhance software release cycles by increasing development velocity, delivering higher code quality, and reducing human error through automation. AWS Edge services such as CloudFront, and edge functions can be managed within a CI/CD pipeline using Infrastructure as Code (IaC). Edge resources can be deployed via APIs (e.g., CloudFront's API), the AWS CLI, or higher-level abstraction tools like CloudFormation, Terraform, and the AWS Cloud Development Kit (CDK).

The CDK, based on CloudFormation, provides the highest level of abstraction by allowing you to deploy AWS resources using a programming language. For example, the following three lines of JavaScript code can deploy an S3 bucket with CloudFront as the origin:

Check out the CDK constructs library for reusable constructs to include in your CDK code.

In your CI/CD pipeline, you'll need a repository like AWS CodeCommit to store your code (CDK code, edge functions code), a tool such as AWS CodeDeploy to deploy your infrastructure, and an orchestration tool like AWS CodePipeline to manage the pipeline steps. This blog post demonstrates using AWS developer tools to implement a CI/CD pipeline for CloudFront, but you can also use your preferred CI/CD tooling. When building your CI/CD pipeline for AWS Edge services, consider the following limitations:

Your edge configuration can be tested at different levels. First, you can replicate your infrastructure, such as a CloudFront distribution with an AWS WAF WebACL, in testing accounts. This can be done during the development phase or for running automated functional tests before moving to production. Note that you cannot use two CloudFront distributions with the same CNAME. Consequently, you need to differentiate the CNAME configuration for your CloudFront resource based on the CI/CD phase. For example, you can use CloudFront's domain name (e.g., xyz.cloudfront.net) for the development phase, a dedicated CNAME for staging (e.g., staging.www.example.com), and the public CNAME for production deployment (e.g., www.example.com). You can also differentiate security controls by stage, such as restricting access to specific IPs using AWS WAF for the staging phase.

After your new configuration passes testing, you can implement a canary release approach using CloudFront's continuous deployment feature to test the change in production on a small portion of your traffic. With this feature, you can create a different configuration for your production distribution and send only a portion of your global traffic to it. You have two options to route traffic to the new configuration: using a specific request header (useful for synthetic testing) or using a weighted policy to route a configurable percentage of traffic to the new configuration, with the option of enabling stickiness. In the current version of this feature, you can only test changes on a subset of the CloudFront configuration parameters. Refer to the documentation to better understand what you can test using this feature.

Consider a scenario where multiple teams introduce changes to the CloudFront configuration, but a single team manages the CI/CD pipeline for all teams. For example, different teams may manage separate microservices, all exposing their APIs on the main domain hosted by a single CloudFront distribution. If you allow each team to access the full configuration of the CloudFront distribution, you increase the blast radius of a single poisonous change. Conversely, if you only allow the CI/CD team to make changes to the CloudFront distribution, you decrease development velocity and introduce a bottleneck in your release lifecycle. Instead, you can dynamically generate the configuration based on partial configurations owned by each microservice team. In a simple implementation, each team can manage the configuration of their own route (caching, edge functions, etc.) in a text-based file hosted in an S3 bucket. In your CI/CD pipeline, you can introduce an additional step to dynamically create the final CloudFront configuration using the different partial configuration files.

Shield and WAF services can be deployed using the same tools described earlier in a CI/CD pipeline.

However, it's recommended to use AWS Firewall Manager for deploying WAF rules and Shield protections for the following advantages:

Additionally, you can use a CI/CD pipeline to make changes to your Firewall Manager policies in the administrator account, which will then be deployed across your AWS organization by Firewall Manager, as demonstrated by OLX.

Deploying WAF rules safely to your production environment can be done using different strategies. You can add new rules to an existing WebACL in count mode, and then change it to block mode. However, with this approach, you need to pay attention to the maximum WCU of your WebACL. Another approach is to deploy changes to a staging WebACL and test the changes in a staging environment.