Moving towards DevOps CI/CD approach to configure and manage AWS networking resources

Introduction

Organizations are moving from traditional monolithic data center networks to an agile application programming interface (API) driven cloud network. As a result, customers are looking for an efficient and reliable way to make changes to their cloud network infrastructure. They want to adopt a pipeline driven approach to make any network changes following DevOps best practices and with infrastructure as code.

Infrastructure as code avails writing custom scripts to automate tasks. In essence, you leverage the same tools that you used to develop software to now manage network configurations, its provisioning and deployments. You can apply version control, perform testing and continuously integrate and deploy networking changes using pipelines.

For example, it is usually a time-consuming and difficult task if you have to manually perform changes to security groups across many accounts, virtual private clouds (VPCs), environments and regions. The manual process is often error prone and does not have of easy failback mechanisms. The devops approach overcomes these limitations and offers benefits such cost reduction: in terms of less time spent by people and effort. Speed: in terms of automation enabled that provides fast executions with more visibility. Less risk: less chance of human error or manual misconfiguration which increases reliability, decreases downtime and increases security.

In this post, I show how you can adopt a DevOps approach to carry out common networking tasks across multiple AWS Regions and accounts using AWS native services like AWS Code Commit, AWS Code Pipeline and AWS Cloudformation. I show you how to set up a CI/CD pipeline that helps carry out these networking changes in a seamless, fast, and efficient manner. Our deployment process consists of security guardrails that help you catch issues early, which subsequently saves time for network deployments and minimizes the potential downtime for your applications. I also provide you with a walkthrough example of our solution for managing security groups. The solution can be extended to carry out any networking tasks like VPC changes, transit gateway changes, private hosted zone DNS changes, route table changes, and network access control list changes etc.

Architecture

Our architecture consists of two AWS accounts; 1) A shared services account and, 2) any other AWS customer account (e.g., test accounts or production accounts).

The Shared Services account is a centralized account for creating and managing the pipeline infrastructure such as CodeCommit and CodePipeline. This centralized account is managed by the networking DevOps team. The spoke accounts are the accounts where the networking resources, like security groups, virtual private cloud, transit gateway attachments, network access control list, and route tables reside. These resources can reside in multiple AWS accounts and regions. They are centrally managed by the pipeline.

Prior to the DevOps model, if you had to make a networking change in your environment, like a change to security groups, you had to log into every account or region to add or remove the rules. If these changes broke an application, you had to leverage VPC flow logs to identify and troubleshoot the problem and fail back manually. With a DevOps approach, you can leverage testing tools, such as VPC reachability analyzer or transit gateway route analyzer as a part of your pipeline. For example, the VPC reachability analyzer can perform connectivity testing between a source and a destination and validate the proposed rule changes for a security group in test accounts. You can roll back your changes in case of any issues using the CI/CD pipeline. This minimizes the overall time and effort required by your network team to manage AWS network infrastructure.

Our pipeline mainly consists of four stages as illustrated below:

Figure 1: Different stages in CI/CD pipeline

Stage 1 – Source stage:

“Infrastructure as Code (IaC)” principles within cloud gives us the ability to quickly and efficiently launch fresh instances of the network infrastructure required. It limits the variations for different application environments to the bare minimum. The code is source controlled and is the source of truth for how your network is configured within AWS. This is the stage where you develop the code in the form of cloudformation templates for various networking resources, such as transit gateway, VPC, route53, network access control list, route tables, security groups etc. A commit to the central repository (such as one hosted in CodeCommit or GitHub) triggers the next stage in the code pipeline.

Stage 2 – Build stage:

The next stage is the build stage which entails linting checks on our CloudFormation templates developed in the previous step. During this stage, we can also validate the templates against any security vulnerabilities using cfn-nag and have guardrails using cfn-guard.

Stage 3 –Validation stage:

The next stage in our pipeline is the validation or test stage which examines the proposed changes from the previous stage for your networking resources to ensure that they do not cause any downtime when deployed. The proposed configurations are deployed to test /staging account(s) that mimics a production or QA environment. We can leverage tools like VPC reachability analyzer, transit gateway route analyzer, or debugging tool for networking connectivity to determine the impact of proposed changes. The blog posts listed in the additional resources section at the end of this post provides detailed information of how these tools can be used in debugging networking issues.

Stage 4 – Deploy stage:

Once you complete a successful deployment in test accounts, you have to manually approve within the pipeline in order to push the changes to the production accounts across different VPCs and/or regions. This is the final stage called the deploy stage where the network configuration changes are carried out to the production environment using CloudFormation Stacksets.

The CI/CD platform is built in a Shared Services Account using the following components:

• AWS CodeCommit: A fully-managed source control service that hosts secure Git-based repositories. CodeCommit makes it easy for teams to collaborate on code in a secure and highly scalable ecosystem. Our solution uses CodeCommit to create a repository to store the deployment codes.
• AWS CodeBuild: A fully managed continuous integration service that compiles source code, runs tests, and produces software packages that are ready to deploy. Our solution uses CodeBuild to build and test the code, which we deploy later.
• AWS CodePipeline: A fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates. Our solution uses CodePipeline to create an end-to-end pipeline that fetches the application code from CodeCommit, builds and tests using CodeBuild, and finally deploys using AWS CloudFormation.
• Security Token Service: A service that enables you to request temporary, limited-privilege credentials for AWS Identity and Access Management (IAM) users or for users that you authenticate (federated users). Our solution leverages Security Token Service (STS) with Assume Roles to access resources across cross account boundaries.
• Parameter Store: is part of Amazon EC2 Systems Manager. It provides a centralized, encrypted store to manage your configuration data, whether it is plain text data (to store resource ID strings) or secure strings and secrets (such as passwords or API keys). This could be used to store cross account resource IDs which can then be referenced by the automated pipeline.
• AWS Organizations: Our solution leverages AWS Organizations, which is an account management service that lets you consolidate multiple AWS accounts into an organization that you create and centrally manage. You can organize these accounts into groups and attach policy-based controls. Our solution uses AWS Organizations to manage your provisioned accounts.
• Simple Notification Service: Our solution leverages Amazon Simple Notification Service (Amazon SNS), which is a managed service that provides message delivery from publishers to subscribers (also known as producers and consumers). Our solution uses SNS to publish updates for all infrastructure networking changes that are deployed via the code pipeline.

Figure 2: A multi-account, multi-region pipeline setup to manage networking resources

AWS CloudFormation is used to setup the pipeline infrastructure and the networking resources required to deploy any network infrastructure changes. IAM roles are used to securely access resources across multiple AWS accounts. Static and security code analysis are performed using tools like cfn-lint and cfn-nag. This is shown in figure 2 above.

Prerequisites

The following requirements must be met:

• A repository in CodeCommit with read/write access to DevOps networking team.
• Enable AWS Organizations to centrally manage your accounts.
• An AWS account for shared services and at least one member account in which resources need to be provisioned or managed.
• A copy of the sample source code from GitHub in your repository.

Deployment Steps

In this section, I show an example of how to create a security group, update it in different accounts and across regions that exist under an AWS organization. The pipeline for this example consists of four stages:

1.The source stage consists of CloudFormation templates for creating or managing security groups and its rules.

2.The build stage consists of performing linting checks on the cloudformation template to catch any syntax issues.

3.The validate stage consists of deploying the stacks using cloudformation templates in the test accounts and ensure its able to create or update the cloudformation stack successfully,

4.The deploy stage is the final stage. This is a stage where I approve the changes and promote the changes to be deployed to production accounts and VPCs.A screenshot of the stages is shown:

Figure 3: A screenshot of the stages in the pipeline for managing networking resources

Step-by-step instructions for setting up the pipeline to manage networking resources are outlined below:

Infrastructure setup for pipeline :

1. Create a service role in shared service account using AWS Organization. This role provides trust relationship that is required to provision resources in target accounts.
   • To do this, navigate to the CloudFormation service in the AWS console.
   • Select Stacksets. At the top center pane, select Enable Trusted Access.By default, this creates a role with name AWSServiceRoleForCloudFormationStackSetsOrgAdmin. Please refer to the document Enable Trusted Access to Cloudformation for further details.
2. Log on to the AWS Management Console of shared services account. Select CloudFormation service.
3. Create a stack using the macro template. This will provision s3 buckets required for multi-region deployments and set up replication between them.
4. Create a stack using pipeline template. Provide the code repository details for storing the configuration artifacts, S3 bucket prefix, Organization ID, email addresses for manual approval and regions for deployment. An example is shown below:
   

   Figure 4: An example of parameters for cloudformation template that provisions a pipeline to manage networking resources in a multi-region, multi-account setup.

5. Next, acknowledge the permissions required to provision IAM roles, and cross account permissions granted via Organization policies and create the stack. The template provisions a CodeBuild project, and fully managed AWS Code pipeline in the organization shared services account.
6. Select the Resources tab in the center pane. Validate that all the resources are CREATE_COMPLETE and the stack is in CREATE_COMPLETE state.

Source Stage:

1. Next, navigate to the CodeCommit service in the AWS console. Create a repository with a name and a suitable description.
2. Navigate to IAM service in the AWS console. Add permissions to the IAM users who require access to the code commit repository. To do this,
   • Click the Users. Click on the user who require code commit repository details.
   • In the center pane, navigate to security credentials.
   • For SSH access, upload your public key in “SSH keys for AWS CodeCommit”.
   • For HTTPS access, generate credentials in “HTTPS Git credentials for AWS CodeCommit”.
     Please refer to the document git-remote-codecommit for further details.
3. On your local machine, use any IDE like cloud9 or intellij. Create a directory with suitable name to store the networking configurations as code. Navigate into that directory.
   Next, clone the code commit repository created in step 6 using git utility. If you don’t have git installed, please follow the instructions outlined here. For example, if you created a code commit repository by name IAC-Networking-Resources in step 6 and have configured HTTPS access, run the command:
   git clone https://git-codecommit.us-west-2.amazonaws.com/v1/repos/IAC-Networking-Resources
4. Enter the credentials for HTTPS access provided in Step7. In the cloned directory, add the files and folders from the repository. Please check the readme.md for details on the folder structure and details of individual files.
5. Modify the files
   • accounts-dev.txt – edit the file and add the OU for your dev accounts.
   • accounts-prod.txt – edit the file and add the OU for your prod accounts.
   • Under config folder, navigate to code
     i.sg.csv – edit the file and add the security group rules you wish to deploy/change.
6. Commit and push the changes to the repository using the following commands:

git add .
git commit -m "adding cloudformation template to deploy security groups"
git push -u origin main

Build Stage:

Commit the change to CodeCommit. A CodePipeline will be triggered using a CloudWatch event which then builds a CodeBuild project that deploys security group rule changes to all the dev accounts.

Validate Stage:

Upon successful deployment of security group rules in dev accounts, a SNS notification is sent to the email address provided in step-4 of infrastructure setup for pipeline.

Deploy Stage:

A manual approval is then required to push the changes to production accounts. To perform this action, check the AWS Notifications email by logging into the email account provided in step-4 of infrastructure setup for pipeline. Click the hyperlink and approve. This will deploy the security group changes to the production accounts across the regions specified in step-4 of infrastructure setup for pipeline.

The above setup can be extended to deploy transit gateway changes, private hosted zone dns changes, and VPC route table or network acl changes.

Conclusion

In this post, I have shown how to leverage CI/CD solution using AWS Code Commit, AWS Code Pipeline and AWS Cloudformation to carry out networking tasks on AWS. I removed the undifferentiated heavy lifting for setting up CI/CD pipeline by providing templates to provision the infrastructure required. I also provided an example of how you can manage security groups. There is more to come! Future posts will cover deep dives on how to carry out network testing using VPC Reachability Analyzer leveraging the CI/CD.

I hope that you’ve found this post informative and look forward to hearing how you use this feature!

Additional Resources

1. VPC Reachability Analyzer
2. Diagnosing traffic disruption using AWS Transit Gateway Network Manager Route Analyzer 

About the Author