Automate AWS account provisioning and server migration using AWS Service Catalog and CloudEndure from AWS Marketplace

If you’re involved in a migration project for your company that is moving to the AWS Cloud, you’ll likely go through a variety of stages, such as migration preparation, portfolio discovery, planning, and design. In most cases, the rubber hits the road after these stages, and you begin migrating your physical, virtual, or cloud-based infrastructur­­­­­e workloads to AWS. AWS customers use tools such as CloudEndure (now an AWS company) to automate application migration, disaster recovery, or backup of their legacy infrastructure to AWS.

One of the challenges customers face during migration is managing and moving servers into a hierarchical account structure that consists of hundreds or thousands of AWS accounts. In this blog post, you’ll learn how to automate how to set up a new CloudEndure migration project, as well as how to automate this process every time you vend a new account in your environment using an “account vending machine”.

Introduction

CloudEndure helps you simplify, expedite, and automate large-scale migration and disaster recovery deployments to AWS. Continuous data replication takes place in the background, without application disruption or performance impact, which ensures that data is synced in real time and minimizes cutover/failover windows. When cutover/failover is initiated, CloudEndure executes a highly automated machine conversion and orchestration process. This enables even the most complex applications and databases to run natively in AWS, without compatibility issues and with minimal IT skills required. You can deploy CloudEndure from AWS Marketplace.

To create this account vending machine, you use additional native AWS services, including AWS Service Catalog, AWS Lambda, and AWS Organizations. You also use API integration with CloudEndure to set up a new project after an account is created. Additionally, you can extend this sample reference solution by configuring additional AWS services such as AWS Direct Connect, Amazon Kinesis Data Firehose, and Amazon S3 Transfer Acceleration in the vended accounts to support your migration.

Services in this solution

Here’s a quick review of the services that you use for this solution:

AWS Service Catalog

AWS Service Catalog allows organizations to create and manage catalogs of IT services that are approved for use on AWS. These IT services can include everything from virtual machine images, servers, software, and databases to complete multi-tier application architectures. AWS Service Catalog enables you to centrally manage commonly deployed IT services and helps you achieve consistent governance and meet your compliance requirements, while enabling users to quickly deploy only the approved IT services they need.

In this solution, the account vending machine exists as an AWS Service Catalog product, which you can launch with user parameters to create new accounts.

AWS Lambda

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume. There’s no charge when your code isn’t running.

With Lambda, you can run code for virtually any type of application or backend service, all with zero administration. Just upload your code, and Lambda takes care of everything required to run and scale it with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.

In our case, AWS Service Catalog launches Lambda to perform the compute necessary to create new accounts and to integrate with CloudEndure.

AWS Organizations

AWS Organizations helps you centrally govern your environment as you grow and scale your workloads on AWS. With AWS Organizations, you can automate account creation, create groups of accounts to reflect business needs, and apply policies for these groups for governance. You can also simplify billing by setting up a single payment method for all of your AWS accounts.

In this solution, Lambda works with AWS Organizations to create accounts and interacts with CloudEndure to automatically set up a migration project for the newly vended account.

Amazon S3

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, Internet of Things (IoT) devices, and big data analytics.

In this solution, we will use Amazon S3 to store the AWS CloudFormation template required to create an AWS Identity and Access Management (IAM) user for CloudEndure access in the new account.

Solution overview

At its core, this solution creates a new account using the AWS Organizations API, deletes the default virtual private cloud (VPC) in the new account, and bootstraps the account with an IAM user for CloudEndure with the permissions required. To achieve this, you need to launch the account vending machine product from AWS Service Catalog, which launches an AWS CloudFormation template. This template triggers a Lambda function for performing the API calls that set up a new account with AWS Organizations and integrates it with CloudEndure.

The following diagram shows the architecture of the account creation process.

The process follows these steps:

1a. AWS Service Catalog launches the account vending machine product, which triggers an AWS CloudFormation template, with required parameters for account creation and CloudEndure integration.

1b. AWS CloudFormation triggers a custom Lambda resource defined for performing API calls for account creation and CloudEndure integration.

1c. Lambda downloads an AWS CloudFormation template called AccountBaseline.yml from an S3 bucket.

1d. The Account Vending Lambda function calls the AWS Organizations API to create a new account. After a new account is created, the Lambda function performs the following functions:

• Deletes the default VPC for the new account in all the AWS Regions
• Launches the AccountBaseline.yml template to create a new IAM user for CloudEndure access.

1e. The Account Vending Lambda function triggers the CloudEndure Lambda function and passes the new account ID to it.

1f. The CloudEndure Lambda function makes an API call to CloudEndure with the required parameters to set up a new project for the newly vended account.

At this point, you have a newly vended account with a corresponding project associated in CloudEndure. The following steps aren’t included in the sample solution, but would be your logical next steps:

2a. Connect the workloads to be migrated from your data center to CloudEndure.

2b. Begin migration/replication of your on-premises workloads using the projects already established in CloudEndure.

For more information about automating CloudEndure migration to AWS, see Facilitating a Migration to AWS with CloudEndure by Leveraging Automation on the AWS Partner Network (APN) Blog.

Deploying the account vending machine with CloudEndure integration

Before you start:

• You need access to a master payer account of an organization in AWS Organizations.
• You need a CloudEndure subscription, which you can get for migration or disaster recovery in AWS Marketplace. Then you need the CloudEndure user name and password to set up this solution.

You can also modify this implementation to enable logging in using the CloudEndure API key, which is the preferred authentication method for issuing CloudEndure API calls.

Note

This is a sample solution to show you the art of the possible and not for production usage.

Setting up the master account

1. Sign in to the AWS Management Console with your master account and switch to the US East (N. Virginia) Region.
2. Click on the Launch Stack button to launch setting up of your account vending machine infrastructure, and click Next.

3. On the Specify Details page, enter the following information:

• Stack name – The name of the stack created for the account vending machine’s infrastructure. Enter avm-ce-infrastructure-setup.
• AccountAdministrator – The administrator who is responsible for launching the AWS Service Catalog product for new account creation with CloudEndure. Enter the ARN value (arn:aws:iam::…) of the IAM user, role, or group you want to grant access for launching the account vending machine.
• CloudEndureUserName – The user name of your CloudEndure account.
• CloudEndurePassword – The password of your CloudEndure account.
• SourceBucket – The Amazon S3 bucket name of all the files required for this solution. Keep the default value.
• SourceTemplate – The URL of the AWS CloudFormation template to create the account vending machine product in AWS Service Catalog. Keep the default value.

The following image shows the Specify Details page.

4. Choose Next, and then Next on the Options page, and then Create.

Creating the stack generates two outputs, AccountLambda and CloudEndureLambda, as shown in the following image. Have these values ready for the next step.

Launching the account vending machine

1. Open the AWS Service Catalog console. Make sure that you’re still in the US East (N. Virginia) Region.

2. Choose Launch product, enter the provisioned product name, and choose Next.
3. On the Parameters page, enter the following information:

• MasterLambdaArn – The ARN of the Lambda function AccountCreationLambda. Use the value from the outputs of the stack creation in the previous section.
• AccountEmail – Enter a unique email ID for account creation.
• OrganizationalUnitName – Enter the name of the organizational unit in AWS Organizations where the new account will be moved.
• AccountName – Enter the name of the account to create.
• StackName – Enter the name of the AWS CloudFormation stack to be launched in the newly vended account. This stack creates baseline resources in the new created account.
• StackRegion – Enter the Region where the baseline resources will be launched in the new account.
• SourceBucket – The name of the Amazon S3 bucket holding the Accountbaseline.yml template. Keep the default value.
• BaselineTemplate – The name of the baseline template to be launched. Keep the default value.
• CloudEndureLambdaArn – The ARN of the Lambda function CloudEndureLambda. Use the value from the outputs of the stack creation in the previous section.
• CloudEndureProjectType – Choose either Migration or DR, depending on the type of CloudEndure project that you want to create.

The following image shows the Parameters page.

4. Choose Next on the TagOptions, Notifications, and Review pages, and then choose Launch.Launching the account vending machine generates the outputs shown in the following image.

Finding the new account on the CloudEndure portal

At this point, you can log in to your CloudEndure portal and check for a new project with the corresponding account ID generated in the previous section. The credentials for the new account appear on the AWS Credentials view of the Setup & Info pane, as shown in the following image.

Conclusion

In this blog post, we have addressed a key migration challenge of automatically integrating newly vended accounts with CloudEndure from AWS Marketplace. This approach can help you reduce the time-consuming manual steps involved in setting up CloudEndure for each of your accounts. It can also help you accelerate your move to key post-migration activities such as validation and ultimately operating your production workloads in the AWS Cloud.

CloudEndure is available in AWS Marketplace, a digital catalog with thousands of software listings from independent software vendors that makes it easy to find, test, buy, and deploy software that runs on AWS. View the CloudEndure products in AWS Marketplace, or ask a question in the AWS Marketplace support forum.

About the authors

Sagar Khasnis is a Partner Solutions Architect focusing on AWS Marketplace and AWS Service Catalog. He is passionate about building innovative solutions using AWS services to help customers achieve their business objectives.

Carmen Puccio is an AWS Solutions Architect. He spent two years helping Consulting and Technology Partners to mass migrate their workloads to AWS at scale and now enables container partners to help their customers adopt container technologies as part of their modernization effort.