Updating AWS CloudFormation Stacks Without Service Disruption to Support Rapid Business Innovation

By Aniket Dehnikar, Associate Engineer, Cloud & DevOps – Successive Cloud
By Tushar Kinhikar, Lead Technology, Cloud & DevOps – Successive Cloud
By Evan Elezaj, Solutions Architect – AWS

Successive Cloud

As a DevOps engineer, how often have you been asked to make changes or add a new resource to the value of a running Amazon Web Services (AWS) stack? On top of that, how many times have you had to apply the changes without disturbing the running services or downtime?

As DevOps engineers, it’s our job to support rapidly-changing business decisions or technical requirements without disrupting business continuity and innovation initiatives.

Thankfully, you can rely on AWS CloudFormation to update or add new resources to an application. AWS CloudFormation provides two methods for updating the stack: direct update or change set.

In this post, we will show you how to update the value and add new resources to your running Amazon Relational Database Service (Amazon RDS) deployment. We’ll also share screenshots and architecture diagrams to help you see how the system behaves when you apply the changes via CloudFormation template.

Successive Cloud is an AWS Advanced Tier Services Partner and AWS Marketplace Seller that enables customers to take full advantage of all that AWS has to offer and accelerate their journey to the cloud.

Background

We all know the hassle and downtime experienced when it comes to updating or adding resources to an application. It can lead to disrupting application resiliency, reliability, and performance. It’s never been easier, though, to keep the application running to accommodate customer needs.

Successive Cloud faced a similar challenge when one of its clients wanted to update the infrastructure resources of an already established application that was built using AWS services. Since the stakeholder’s requirements changed abruptly and required Successive Cloud to alter some existing values and add some resources, the team looked to a solution using AWS CloudFormation best practices.

Successive Cloud followed the best methods, modified some of the running stack’s values, and added resources without recreating the existing infrastructure. The team picked up a running Amazon RDS service and performed the tests to validate the solution. First, they updated the value of storage of an RDS service; next, they added new resources to test how reliably the system performs or which behavior changes impact the whole application.

Let’s have a look at the steps followed and how the system has responded to the updates performed.

Solution Overview

Here, we’ll take the example of MySQL and show how to increase the storage value of a running MySQL instance by implementing instance requirements through an AWS CloudFormation template.

Refer to the AWS documentation about updating behaviors of stack resources and AWS::RDS:DBInstance to see how a system behaves when you submit an update through CloudFormation for more information on AWS services.

If you want to learn more about the impact of value updates in the RDS DB cluster, see the AWS documentation for more details.

For reference, the CloudFormation template used in this post is available in GitHub.

Prerequisites

You must have the following:

AWS account ID or AWS account with AWS Identity and Access Management (IAM) permissions to access Amazon RDS and AWS CloudFormation to set up AWS services and deploy the template.
Prepare your CloudFormation template either in YAML or JASON with the new infrastructure requirements, or you can use the CloudFormation template linked above.
Set up a (demo) environment and deploy RDS service with this CloudFormation template.

Walkthrough

This solution will complete the following steps:

Value change in running Amazon RDS
- In a demo environment, we’ll analyze the current state of the MySQL in the AWS Management Console (Figure 1) and its stack specification (Figure 2).
- Once the new configuration is pushed through CloudFormation, we’ll observe the system behavior change in the AWS console (Figure 3) to understand the impact of direct updates (Figures 4 and 5).
Adding new resources to a running RDS service
- We’ll add a new database in a new AWS Availability Zone (AWS) as a backup to a running database to ensure security. Even when a replica database is deleted, a backup database will be available to support business continuity.
- Since a running RDS service is important and needs to be available at all times, implementing the requirement with a change set allows previewing changes before applying them. It also ensures RDS instances stay running smoothly and efficiently.

Value Change in Running RDS Service

Step 1: Setting Up the Infrastructure with RDS

We have already deployed a MySQL instance with 16GB RAM and 100 GB storage on AWS to run an application. As you can see in the screenshot below, the console shows the resource status available.

Figure 1 – Deployed RDS status.

To verify the specifications of your deployed instance, just click on the instance link. The console will redirect you to the instance’s overview which includes configuration, instance class, storage, performance insights, multi-Availability Zone, and other options.

Figure 2 – Values of RDS instance (values of RDS storage).

Step 2: Change Analysis of Deployed RDS State

Once the mentioned RDS template with new configuration requirements is pushed, you can see a change in the status of RDS service as “Modifying” in the console.

Figure 3 – Status changed to modifying.

Once the stack status was confirmed to be “Available,” we checked the values of the newly-deployed RDS instance. Since we programmed to update storage from 100GB to 200GB, we can see a new value was prompted without hampering the system performance and creating issues with other values.

Figure 4 – Value updated in RDS (value change in storage).

The above process is an example of a direct update in stack value; as mentioned in the behavior table, we observed “No Interruption” in system availability and performance.

However, we successfully updated our RDS service’s storage value without experiencing downtime. We also received a message in our console as “Error” that there is a five-hour wait time when we can change the storage value again. Pay attention to details when attempting to create a solution for your project.

Figure 5 – RDS change error (time block of five hours).

Adding Resources in Running Stack

We wanted to add a new RDS service as a backup database in the new Availability Zone to ensure business continuity even in scenarios when replicas fail. As RDS is a critical service to any application, we went through the change set method recommended by AWS.

Change sets provide a preview of the changes CloudFormation will make to the stack. It allows you to check the change in your console, and then you can decide whether to apply those changes.

The process of updating a stack with change sets is depicted in the diagram below.

Figure 6 – AWS service workflow when updating stack with change set.

Change sets allow you to evaluate how proposed changes to a stack can affect your currently-running resources. Since there was no transparency earlier, for example, whether the implemented changes would add or replace any crucial resources, it was important to have visibility and complete control over the system the change set method offered.

Successive Cloud noticed in its proof of concept (PoC) that going with change set allows CloudFormation to create a new resource in the stack parallel to the old one and ask the user permission to execute the change set. Let’s check the steps followed to achieve the results.

To create a change set for a running stack, you can refer to the AWS documentation.

Figure 7 – Status of a deployed instance as “Available.”

We can see in Figure 8 below that once we run a CloudFormation template with an updated RDS request, it creates a second RDS instance in a different Availability Zone. The change will reflect instantly on the AWS console, and the new RDS deployment will happen within seconds without disturbing the existing RDS resource.

Figure 8 – CloudFormation creates a new resource parallel to the old one.

In this example, CloudFormation created a new resource parallel to the old resource in the running stack. Once the new resource is created, accept both resources as we have added the new one to support the system performance.

With “No Interruption,” CloudFormation with the change set method added new resources to meet the requirements and offered convenience to focus on innovation, without worrying about system reliability and scalability requirements.

Conclusion

In this post, we described the key impact of updating value and adding resources in a running stack, and how to use AWS CloudFormation to perform such changes.

With the CloudFormation template, users can easily implement changes in the stack as needed. CloudFormation leverages all modern paradigms, policies, and settings to automate infrastructure deployment with a single click on AWS.

With this PoC, Successive Cloud tested and helped its client achieve the following benefits:

Avoid downtime
Scaling resources efficiently (in a single click)
Achieve high availability

We expect the community to leverage this capability to frequently innovate and back up their changing requirements without worrying about the deployed infrastructure. To learn more, we recommend you review these additional resources:

.

.

Successive Cloud – AWS Partner Spotlight

Successive Cloud is an AWS Partner that enables customers to take full advantage of all that AWS has to offer and accelerate their journey to the cloud.

Contact Successive Cloud | Partner Overview | AWS Marketplace

AWS Partner Network (APN) Blog