Achieve continuous delivery with blue/green deployments using Amazon DocumentDB database cloning and AWS DMS

In the fast-paced realm of digital transformation, organizations are navigating a landscape of unprecedented change and opportunity. As businesses evolve to meet the demands of modern consumers and markets, the need for agile delivery of enhancements to existing applications is becoming more and more essential. Zero downtime during application deployment is one of the key requirements for continuous delivery.

In response to this demand, new techniques have emerged to address the challenge. In complex computing environments, replicating an identical staging environment to production is often impractical. This is due to reasons like incorrect data, the need to duplicate terabytes of data, and configuring identical infrastructure. This time-consuming process diverts valuable engineering resources away from developing new features and enhancements. Consequently, it becomes difficult to conduct an exhaustive staging process that instills complete confidence in the new application version.

Blue/green deployment is a deployment strategy used in software development to minimize downtime and risks during updates or releases. It involves maintaining two separate but identical environments. One environment (blue) is running the current application version, and one environment (green) is running the new application version. Using a blue/green deployment strategy increases application availability and reduces deployment risk by simplifying the rollback process if a deployment fails. When testing is complete on the green environment, live application traffic is directed to the green environment and the blue environment is deprecated.

With the Amazon DocumentDB (with MongoDB compatibility) cloning feature, you can now implement blue/green deployment within a few minutes. Amazon DocumentDB is a scalable, highly durable, and fully managed database service for operating mission-critical JSON workloads for enterprise workloads. Amazon DocumentDB simplifies your architecture by providing built-in security best practices, continuous backups, and native integrations with other AWS services.

In this post, we explore the implementation of blue/green deployment using the Amazon DocumentDB fast database cloning feature and data synchronization between databases across two environments using AWS Database Migration Service (AWS DMS).

Solution overview

Blue/green deployments have two identical production environments: one is labeled blue and the other is labeled green. Only one of the environments is ever live, and deployments are always done to the inactive environment. For example, if blue is the live environment, deployment is done to the green (inactive) environment. After verification, a switchover happens, which makes the green environment the live environment, and the blue environment inactive.

The following diagram shows the blue/green deployment architecture for Amazon DocumentDB at a high level.

Our blue/green deployment solution is based on Amazon DocumentDB fast cloning and AWS DMS change data capture (CDC) features.

The blue environment represents the current Amazon DocumentDB cluster serving production traffic. You create the green environment by cloning the database from the blue environment. You also create an AWS DMS replication instance and configure a CDC task to replicate ongoing changes from the blue environment to the green environment. For more information about using AWS DMS, see Migrating from MongoDB to Amazon DocumentDB.

Next, you apply changes like minor version upgrade, major version upgrade, creating or updating indexes, renaming databases or collections, and minor transformations to the staging environment (green) that mirrors the production environment (blue). During this period, the blue environment continues to serve production traffic and the data changes are replicated to the green environment with AWS DMS CDC. If there are any modifications to the schema or if the database or collection names are changed, table mapping and transformation rules may be required.

After the green cluster is upgraded and replication is fully in sync, you switch over the production traffic to the green cluster (new production cluster). To achieve this, you can use the DNS routing functionalities offered by Amazon Route 53. You can effortlessly switch between the blue and green environments by updating DNS records, seamlessly routing traffic to the desired environment.

Next, you set up a reverse AWS DMS CDC migration task to apply all the changes from the green cluster to the blue cluster (old production cluster). This makes sure no data loss will occur on the blue database, should a rollback be necessary at a later time. Again, table mapping and transformation rules may be required for reverse replications.

There are several advantages to this approach. Rollbacks are just a matter of switching from green to blue or blue to green. When switchovers occur, they are seamless because the code has already been running and tested and there is no need to wait for warmup. Changes are verified in production without any user traffic impacting the code, which reduces the risk in the deployments.

Amazon DocumentDB fast cloning

The Amazon DocumentDB fast database cloning feature takes advantage of the underlying distributed storage engine. Amazon DocumentDB uses a copy-on-write protocol to create a clone. This mechanism uses minimal additional space to create an initial clone. When the clone is first created, Amazon DocumentDB keeps a single copy of the data that is used by the source database cluster (blue) and the new cloned cluster (green). Additional storage is allocated only when changes are made to the data by the source Amazon DocumentDB cluster or the Amazon DocumentDB cluster clone. To learn more about the copy-on-write protocol, see How Amazon DocumentDB cloning works.

The following diagram shows the blue and green implementation using the fast cloning feature.

Let’s say you make a change to your source Amazon DocumentDB cluster (blue) that results in a change to data held on page 1. Instead of writing to the original page 1, Amazon DocumentDB creates a new page, 1[A]. The Amazon DocumentDB cluster (blue) now points to page 1[A], 2, 3, and 4, while the clone (green) still references the original pages.

On the clone cluster (green), a change is made to page 4 on the storage volume. Instead of writing to the original page 4, Amazon DocumentDB creates a new page, 4[B]. The clone now points to pages 1, 2, 3, and 4[B], while the source cluster(blue) continues pointing to 1[A], 2, 3, and 4.

Set up a blue/green environment and apply changes

The following diagram outlines the necessary steps for configuring a blue/green deployment environment for Amazon DocumentDB.

The implementation steps are as follows:

1. Create a new green stack that mirrors the current blue stack, with the exception of the database tier. For example, the new green environment can include Route 53 for testing routing entries, an application tier like a load balancer, Amazon Elastic Compute Cloud (Amazon EC2) instances, or containers—but not the database tier.
2. Create an Amazon DocumentDB clone cluster in the green environment using fast database cloning from the source cluster in the blue environment.
3. Enable change streams for the source (blue) cluster, which are required for CDC. To perform a minimal downtime migration, AWS DMS requires access to the cluster’s change streams. Amazon DocumentDB change streams provide a time-ordered sequence of update events that occur within the cluster’s collections and databases. Reading from the change stream enables AWS DMS to perform CDC and apply incremental updates to the target cluster.
4. Apply changes like adding indexes or data reorganization and major version upgrades on the clone cluster in the green environment.
5. Create an AWS DMS CDC-only migration task to enable replication of data changes from the source (blue) to the target (green) databases. The replication task should be configured with a custom CDC start time matching when the green clone database cluster was originally created. This custom start time makes sure data replication occurs accurately from the moment the green environment clone was created. Table mapping and transformation rules may be required based on the changes made to the cloned cluster in Step 4.
6. Make any application code deployments on the green environment corresponding to the database changes.
7. Perform necessary tests on the green environment using a separate Route 53 URL than the current production URL.

Cut over from the blue to the green environment

After functional and performance testing is complete on the green environment, you can redirect your production traffic to the green environment. Before initiating the cutover, refer to the following checklist as a reference guide (add additional checks as required to suit your environment):

• The green environment becomes the new production database after cutover. Therefore, avoid destructive tests on the green environment that cause data changes or data inconsistency between environments.
• Make sure that all users and roles are set up with appropriate permissions in the green environment.
• Verify that the green environment is configured with the correct instance type, custom database parameter groups, and security groups.
• Make sure required read replicas are created for the Amazon DocumentDB cluster in the green environment for high availability and read scaling requirements of your production databases.
• Conduct necessary tests to verify that the application is working correctly.

The following diagram outlines the steps to redirect production traffic from the blue environment to the green environment.

The detailed steps are as follows:

1. Stop the user traffic and shut down the applications on the blue environment. You can stop traffic to the blue environment by updating the alias record in an Route 53 DNS hosted zone.
2. Reverse the AWS DMS database replication task, so that green becomes the source and blue becomes the target. This makes sure there is no data loss on the blue database if a rollback is required later on. Again, table mapping and transformation rules may be required for reverse replications based on the changes applied to the green database cluster.
3. Update the production Route 53 routing records that direct the end-user traffic to the green environment. The system is now available for the client applications.

The downtime for end-users is the duration between Steps 1 and 3 and does not depend on the size of the database, complexity of the changes, or duration of testing.

Post-cutover activities

After the stability of the green environment has been verified with the production traffic, you can stop the AWS DMS task and decommission the blue stack.

Conclusion

In this post, we showed how to implement Amazon DocumentDB blue/green deployment by employing the Amazon DocumentDB fast database cloning feature and AWS DMS CDC. The fast database cloning feature allows you to quickly create a copy-on-write clone of your database to minimize costs. Additionally, you can reduce application downtime using AWS DMS CDC.

With the blue/green deployment strategy, you can make changes to the database in the staging environment without impacting the production environment. For instance, you can upgrade major or minor database engine versions, make changes to the index, and adjust database parameter groups. Once these changes are complete, you can promote the staging environment to serve as the new production database environment with near-zero downtime.

If you have feedback about this post, share it in the comments section.

About the Author

Gururaj S Bayari is a Senior Amazon DocumentDB Specialist Solutions Architect at AWS. He enjoys helping customers adopt Amazon’s purpose-built databases. He helps customers design, evaluate, and optimize their internet-scale and high-performance workloads powered by NoSQL or relational databases.