Migrate from self-managed Redis to Amazon ElastiCache

Introduction

In this lesson, you migrate a self-managed Redis cluster to a fully managed cluster using Amazon ElastiCache. First you learn why you would want to use ElastiCache to manage your Redis cluster. Then you work through the steps to migrate an existing Redis cluster to ElastiCache. At the end of this lesson, you should feel confident in your ability to migrate an existing cluster to ElastiCache.

Time to complete: 30–45 minutes

Move to Managed Databases - Migrate from Redis to Amazon ElastiCache (15:29)
Why use ElastiCache?

ElastiCache is a fully managed service for running in-memory data stores with Amazon Web Services (AWS). In-memory data stores such as Redis and Memcached are used for a variety of workloads including query caching and session stores. In-memory data stores enable high-velocity and low-latency workloads with submillisecond latency.

With ElastiCache, your cluster operations are managed by AWS, leaving your team free to focus on innovation. In this lesson, you use ElastiCache for Redis, which handles cluster scaling, instance failover, data backups, and software updates.

Lesson contents

In this lesson, you learn how to migrate a self-managed Redis cluster to a fully managed cluster on ElastiCache. This lesson has four steps.

  • 1. (Optional) Create a source Redis cluster in Amazon EC2

    If you need to create a source database for this migration walkthrough, navigate to the Amazon EC2 console. Choose Launch instance to start the Amazon EC2 instance creation wizard.

    Choose "Launch instance" to start the Amazon EC2 instance creation wizard
    (click to zoom)

    The first step in the Amazon EC2 instance creation wizard is to choose your Amazon Machine Image (AMI). Use the Amazon Linux 2 AMI with the default x86 architecture and choose Select.

    Use the Amazon Linux 2 AMI with the default x86 architecture and choose "Select"
    (click to zoom)

    On the next page, choose the instance type for your Amazon EC2 instance. For this walkthrough, you can use the default t2.micro instance type, which falls in the AWS Free Tier. Choose Review and Launch to continue.

    Choose the instance type for your Amazon EC2 instance
    (click to zoom)

    The next page shows the default options for the rest of your Amazon EC2 settings. The default options should be fine for this use case. Choose Launch to continue.

    Choose "Launch" to continue
    (click to zoom)

    After you choose Launch, choose a key pair to allow SSH access to your new Amazon EC2 instance. From the dropdown, choose Create a new key pair to create a new key pair for this walkthrough. Then name your key pair elasticache-migration and choose Download Key Pair to download the key pair to your computer. Finally, choose Launch Instances to create your instance.

    Choose a key pair to allow SSH access to your new Amazon EC2 instance
    (click to zoom)

    You should see a page confirming that your instance is launching. Choose View Instances to see your Amazon EC2 instance.

    Choose "View Instances" to see your Amazon EC2 instance
    (click to zoom)

    As your instance is initialized, it shows an Instance State of pending Wait until the Instance State shows running.

    Wait until the "Instance State" shows "running"
    (click to zoom)

    When the Instance State shows running, you can SSH into your instance.

    When the "Instance State" shows "running," you can SSH into your instance
    (click to zoom)

    Copy the IPv4 Public IP value for your instance, and then run the following commands in your terminal to SSH into your instance.

    chmod 600 /path/to/elasticache-migration.pem
    ssh -i /path/to/elasticache-migration.pem ec2-user@<IPv4PublicIP>

    Be sure to use the proper values for the path to your elasticache-migration.pem file that you downloaded and use the correct IPv4 Public IP for your instance.

    If you have difficulty connecting to your Amazon EC2 instance, see Connecting to your Linux instance using SSH.

    If you connected successfully, your terminal should look like the following.

    Copy the &quot;IPv4 Public IP&quot; value for your instance, and then run these commands in your terminal to SSH into your instance
    (click to zoom)

    To install Redis, execute the following commands in your terminal.

    sudo yum -y install gcc make
    wget http://download.redis.io/releases/redis-5.0.6.tar.gz
    tar xvzf redis-5.0.6.tar.gz
    cd redis-5.0.6
    make
    sudo make install

    Installing Redis takes a few minutes because the system downloads the Redis source and compiles Redis for your machine.

    After Redis has been compiled, start your Redis cluster by running the following command in your terminal.

    redis-server --daemonize yes --protected-mode no

    Note: This is running with protected mode disabled only because it is for a demonstration. In a production use case, you should not disable protected mode.

    You should see output indicating that the Redis cluster is starting. You can test your Redis cluster by running a ping command in your terminal.

    redis-cli PING

    This command should return a response of PONG.

    If you get the PONG response, you have started your Redis cluster on your Amazon EC2 instance.

    Finally, load some sample data into your Redis cluster to test the migration. In your terminal, run the following command to insert 10 keys into Redis.

    redis-cli DEBUG POPULATE 10 test 10000

    You should see a response of OK in your terminal. This loads 10 keys into your Redis cluster. Each key has a prefix of test.


    In this module, you created a self-managed source Redis cluster from which you can test performing a migration to ElastiCache. In the next module, you will create an ElastiCache cluster.

  • 2. Create an ElastiCache cluster

    In this module, you create an ElastiCache cluster. This is your primary cluster after you copy your existing data into it by using ElastiCache online migration.


    To get started, navigate to the ElastiCache console. On the Redis page, choose Create to create a new cluster.

    Choose &quot;Create&quot; to create a new cluster
    (click to zoom)

    In the ElastiCache cluster creation wizard, choose Redis as your Cluster engine. Do not enable Cluster Mode because online migration does not work with Cluster Mode.

    Give your Redis cluster a name and description, and choose the instance type for your cluster nodes.

    Choose &quot;Redis&quot; as your &quot;Cluster engine&quot;
    (click to zoom)

    In Advanced Redis settings, create a subnet group for your Redis cluster. A subnet group refers to a set of network subnets in your Amazon Virtual Private Cloud (Amazon VPC) where your Redis cluster is placed.

    In &quot;Advanced Redis settings&quot;, create a subnet group for your Redis cluster
    (click to zoom)

    In the Security section, use the same security group that you used for your source Redis cluster on Amazon EC2. Do not enable encryption at rest or encryption in transit because these are not supported during online migrations.

    In the &quot;Security&quot; section, use the same security group that you used for your source Redis database on Amazon EC2
    (click to zoom)

    The other default settings work for this migration. Scroll to the bottom and choose Create.

    Scroll to the bottom and choose &quot;Create&quot;
    (click to zoom)

    After you choose Create, you should see your cluster in the ElastiCache dashboard. It shows a Status of creating while it is initializing.

    While your cluster is initializing, it has a &quot;Status&quot; of &quot;creating&quot;
    (click to zoom)

    While you are waiting for your ElastiCache cluster, navigate to the Security Groups section of the Amazon EC2 console. Find the security group that you used for your Amazon EC2 instance and ElastiCache cluster, and choose it.

    Find the security group that you used for your Amazon EC2 instance and ElastiCache instance, and choose it
    (click to zoom)

    When looking at your security group details, choose Edit inbound rules.

    Choose &quot;Edit inbound rules&quot;
    (click to zoom)

    Add an additional inbound rule to your security group that allows TCP traffic on port 6379 for all instances in your security group. This allows your ElastiCache cluster to access your self-managed Amazon EC2 instance.

    Your screen should look like the following.

    After you have added the rule, choose Save rules to confirm your changes.

    Add an additional inbound rule that allows for TCP traffic on port 6379 for all instances in your security group
    (click to zoom)

    Navigate back to the ElastiCache console. You should see the Redis cluster you created. When this cluster has a Status of available, you are ready to move on to the next module.

    When your cluster has a &quot;Status&quot; of &quot;available&quot;, you are ready to move on to the next module
    (click to zoom)

    In this module, you created a fully managed Redis cluster using ElastiCache. You also updated the inbound rules in your security group to allow your ElastiCache nodes to access your self-managed Redis cluster on Amazon EC2.

    In the next module, you will use ElastiCache to perform an online migration of your existing data.

    If you are using ElastiCache in production, it is best to use AWS CloudFormation to manage your resources using principles of infrastructure as code. You can create an ElastiCache cluster, an ElastiCache subnet group, a security group, and the related networking infrastructure by creating this CloudFormation stack.

  • 3. Perform an online migration from an existing Redis cluster to your ElastiCache cluster

    In this module, you perform an online migration of your Redis data from your existing, self-managed Redis cluster to your fully managed Redis cluster on ElastiCache.

    Online migration configures one of your ElastiCache nodes to be a replica for your self-managed Redis cluster. When the replication is complete, you can switch your application to use the ElastiCache cluster directly and stop the migration.


    Before you start your migration, you need the private IP address of your self-managed Redis cluster on Amazon EC2. Navigate to the Amazon EC2 console and find the instance that is hosting your Redis cluster. Copy the Private IPs value and save it for the next step.

    Find the instance that is hosting your Redis cluster and copy the &quot;Private IPs&quot; value
    (click to zoom)

    Navigate to the ElastiCache console to begin your online migration. Choose your ElastiCache cluster, and then choose Migrate Data from Endpoint in the Actions dropdown.

    Choose your ElastiCache cluster, and then choose &quot;Migrate Data from Endpoint&quot;
    (click to zoom)

    In the configuration window that is displayed, enter the private IP address of your Amazon EC2 instance as the Source Redis Endpoint. Then choose Start Migration.

    Enter the private IP address of your Amazon EC2 instance as the &quot;Source Redis Endpoint&quot;
    (click to zoom)

    If you prefer to use the AWS Command Line Interface (AWS CLI) to initiate an online migration, you can use the start-migration command.

    aws elasticache start-migration \
        --replication-group-id <replicationGroupId> \
        --customer-node-endpoint-list ‘[
                {
                            “Address”: “<privateIPOfEC2Instance>”,
                            “Port”: 6379
                }
        ]’

    Be sure to replace <replicationGroupId> and <privateIPOfEC2Instance> with the values for your configuration.

    As your Redis cluster starts the migration, it shows a Status of modifying to indicate it is being added to your existing cluster.

    Your Redis cluster shows a &quot;Status&quot; of &quot;modifying&quot; to indicate it is being added to your existing cluster
    (click to zoom)

    After your ElastiCache cluster has successfully connected to your existing Redis cluster, you should see a Status of migrating.

    After your ElastiCache cluster has successfully connected to your existing Redis instance, you should see a &quot;Status&quot; of &quot;migrating&quot;
    (click to zoom)

    There are a few ways you can check the status of your ElastiCache online migration.

    First, you can look at the metrics for your primary node. Choose your ElastiCache cluster, which should bring up the nodes for your cluster. Choose the primary node to see the Amazon CloudWatch metrics for the node in the ElastiCache dashboard.

    Scroll to the Master Link Health Status chart. When the value for this chart is at 1, your ElastiCache cluster is caught up with replicating items from your self-managed Redis cluster.

    Choose the primary node to see the Amazon CloudWatch metrics for the node in the ElastiCache dashboard
    (click to zoom)

    A second way you can check the status of your ElastiCache online migration is to connect to your ElastiCache cluster itself.

    Find the primary endpoint of your ElastiCache cluster in the ElastiCache console. Copy the hostname but do not copy the port number at the end.

    Copy the hostname of the primary endpoint but do not copy the port number at the end
    (click to zoom)

    If you are still connected to your self-managed Amazon EC2 instance via SSH, you can use that instance to connect to your ElastiCache cluster.

    Type the following into your terminal while connected to that cluster.

    redis-cli -h <yourPrimaryEndpoint>

    Be sure to replace <yourPrimaryEndpoint> with the hostname of your Redis cluster from the previous step. This connects you to your ElastiCache Redis cluster.

    Then execute the following command while connected to the Redis cluster.

    KEYS *

    You should see the 10 test keys you inserted into your self-managed Redis cluster.

    The following is an image that shows the terminal output after you connect to your Redis cluster and execute the KEYS command.

    You should see the 10 test keys you inserted into your self-managed Redis cluster
    (click to zoom)

    If the online migration is not copying the keys to your fully managed ElastiCache cluster, you can debug the process by choosing Events in the navigation pane.

    You can debug the migration process by choosing &quot;Events&quot; in the navigation pane
    (click to zoom)

    The Event page shows you information about your migration, including when it started, when it completed, and if it failed. If your migration failed, this page includes information about the failure.

    The &quot;Event&quot; page shows you information about your migration, including when it started, when it completed, and if it failed
    (click to zoom)

    In this module, you started an online migration in ElastiCache to copy data from your self-managed Redis cluster in Amazon EC2 to your fully managed Redis cluster in ElastiCache. This is a safe, reliable way to migrate your data to ElastiCache.

    In the next module, you will complete the migration and clean up resources.

  • 4. Complete the migration and clean up resources

    If you have followed all the steps in this lesson, you have created a new, fully managed Redis cluster in ElastiCache and migrated your existing data from your self-managed Redis cluster to your new Redis cluster. In this final module, you learn the steps to complete your migration and clean up your resources.


    When your initial migration is complete and all data is synced to your new Redis cluster, you are ready to use your new Redis cluster in your application. In your application, change the configuration to use your ElastiCache endpoint rather than your existing Amazon EC2 instance.

    You can find the Primary Endpoint for your ElastiCache cluster in the ElastiCache console.

    You can find the &quot;Primary Endpoint&quot; for your ElastiCache cluster in the ElastiCache console
    (click to zoom)

    When you have switched the configuration to your new ElastiCache cluster and are confident in the results, you can stop the migration from your existing Redis cluster.

    To do that, choose your ElastiCache cluster in the console, and then choose Stop Data Migration in the Actions dropdown.

    Choose your ElastiCache cluster in the console, and then choose &quot;Stop Data Migration&quot;
    (click to zoom)

    This removes your ElastiCache node as a replica from your self-managed Redis cluster.

    When your ElastiCache node has been removed as a replica, it shows a Status of available.

    When your ElastiCache node has been removed as a replica, it shows a &quot;Status&quot; of &quot;available&quot;
    (click to zoom)

    Next, you can delete your Amazon EC2 instance that is hosting your self-managed Redis cluster. You should do this only when you are comfortable that the migration has been successful and you no longer need your existing instance.

    Navigate to the Amazon EC2 console. Choose your Amazon EC2 instance, and then choose Instance State > Terminate from the Actions dropdown.

    Choose your Amazon EC2 instance, and then choose &quot;Instance State&quot; &gt; &quot;Terminate&quot;
    (click to zoom)

    If you were using the ElastiCache cluster only for this walkthrough, you can terminate that also. Navigate to the ElastiCache console. Choose your ElastiCache cluster, and then choose Delete in the Actions dropdown.

    After a few minutes, your ElastiCache cluster should be deleted.

    Choose your ElastiCache cluster, and then choose &quot;Delete&quot;
    (click to zoom)

    In this module, you learned how to migrate your application to use your new, fully managed Redis cluster. You also saw how to stop migration from your self-managed Redis cluster to your fully managed ElastiCache cluster, and how to delete your self-managed Redis cluster on Amazon EC2.

In this lesson, you migrated an existing, self-managed Redis cluster running on Amazon EC2 to a fully managed Redis cluster on ElastiCache. You used the ElastiCache online migration operation to safely and reliably replicate your data to your new cluster.