Use AWS Systems Manager Session Manager for port forwarding to Amazon ElastiCache for Redis inside a private subnet

With the increasing adoption of the public cloud, customers must minimize the attack surface of their infrastructure. When it comes to optimizing the response time of read-intensive applications, data caching is one of the first steps to consider. Amazon ElastiCache for Redis is versatile in-memory storage that offers highly available, highly scalable, and extremely fast retrieval time for frequently queried data. Malicious parties are always on the lookout for ways to exploit security flaws and obtain access to customer data. An ElastiCache cluster can quickly become a valuable target, so it’s important to keep every data storage medium as secure as possible.

For this reason, one of the core security and infrastructure design best practices is to encapsulate resources in private subnets of Amazon Virtual Private Cloud (Amazon VPC) and limit the inbound access of resources using mechanisms like security groups. However, placing resources in private subnets and restricting system access inevitably limits how developers can interact with the system and develop or test new features. In this blog post, we will solve a connectivity obstacle where developers have to query a remote Redis cluster because replicating the same development data locally is not feasible. This use case is not limited to ElastiCache for Redis. The same mechanisms apply for any resources inside a private subnet (for example, an Amazon Aurora DB cluster).

We’ll also show you how to use port forwarding through AWS Systems Manager Session Manager (SSM) in your development process. We’ll establish an SSH tunnel to an instance running HAProxy without having to manage any SSH bastion hosts or open inbound ports for external access. We use HAProxy because it offers us the option to balance requests between multiple ElastiCache nodes without explicitly using additional cloud services. As a result, we’ll create a secure access pattern from your local machine to the remote instance connecting to ElastiCache, without the security overhead or the burden of managing unnecessary infrastructure.

You’ll find the complete example project (CloudFormation template and a session-opening shell script) in this GitHub repository. The CloudFormation template creates all of the infrastructure components for you, including the VPC.

Figure 1: Architecture diagram

Project prerequisites

Assuming you have access to an AWS account, you will need the following tools to deploy described AWS resources, start port forwarding via SSM and test the forwarded connection. First of all, you will need to install the AWS Command Line Interface (CLI). Secondly, you need to install Session Manager plugin for the AWS CLI that enables you to start and close sessions with managed instances. Finally, in case you want to test the forwarded connection you need to install the Redis CLI.

Solution setup

Assuming you obtain the CloudFormation template from the provided GitHub repository, you need to configure several key properties before deployment. First, you need to specify the location where of the template file. Second, you need to provide a stack name that will be used by CloudFormation when deploying resources. Third, you need to provide parameter overrides for availability zones where the EC2 instance and ElastiCache nodes will be deployed. Finally, as we are creating new AWS Identity and Access Management resources, you need to provide the capabilities parameter along with the corresponding value.

AWS CLI sample command that provides all of described parameters and provisions template resources looks like as follows:

aws cloudformation deploy --template-file template.yml --stack-name SsmComputeStack --parameter-overrides AvailabilityZones="eu-west-1a,eu-west-1b,eu-west-1c" --capabilities CAPABILITY_IAM

In case your AWS CLI is configured for a region different than EU-Central-1 please adjust AvailabilityZones override values accordingly. Once all the resources are provisioned and ready, you can use the provided open-redis-tunnel.sh shell script to start the port forwarding and Redis CLI to test the connection.

Solution walkthrough

Our solution builds on top of infrastructure components distributed over three Availability Zones. We set up infrastructure that will enable us to place resources into private subnets where public access is not possible. We create two security groups. By default, security groups do not allow any inbound access. For this reason, we have to explicitly allow access to the desired port from either another IP source address or another security group. In our example, we allow inbound access to our ElastiCache security group from a bastion security group on the default Redis port. This ensures that HAProxy can distribute requests to the Redis nodes.

To be able to connect to the EC2 instance using Systems Manager, the Systems Manager Agent must be installed and running on the HAProxy bastion instance. Furthermore, the instance must have SSM permission policies included in the assumed AWS Identity and Access Management (IAM) service role that will allow it to use Systems Manager service core functionality. In our example, we use an Amazon Linux 2 AMI that already has the SSM Agent installed. The AWS CloudFormation template sets the permission policies. To create the SSH tunnel, the IAM user must have permissions to start and stop SSM sessions (SSM:StartSession, SSM:TerminateSession).

The architecture uses a NAT gateway to allow connections to the internet. As an alternative, you can provision VPC endpoints and remove the requirement for internet connectivity entirely. A VPC endpoint enables private connections between your VPC and supported AWS services and VPC endpoint services powered by AWS PrivateLink.

Amazon ElastiCache for Redis cluster

The Amazon ElastiCache for Redis cluster in our blog post solution is deployed in a private subnet. The nodes in the cluster will be private, which means they are not assigned a public IP address or a DNS name. You can use the ElastiCache console or the CloudFormation template to create an Amazon ElastiCache for Redis cluster. Specify a subnet group to which the clusters can be deployed.

  CacheSubnets:
    Type: 'AWS::ElastiCache::SubnetGroup'
    Properties:
      Description: Company Caching subnet group
      SubnetIds:
        - !GetAtt VPCStack.Outputs.PrivateSubnet1AID
        - !GetAtt VPCStack.Outputs.PrivateSubnet2AID
        - !GetAtt VPCStack.Outputs.PrivateSubnet3AID

If you use the provided CloudFormation template, the subnet ID values will be outputted by the CloudFormation VpcStack. It can take up to 10 minutes to create the ElastiCache cluster and subnet groups.

Primary endpoint of the Amazon ElastiCache for Redis cluster

The primary endpoint is a DNS name that always resolves to the primary node in the cluster. The endpoint address does not change. AWS recommends that applications always connect their write activity to this primary endpoint. For this reason, the HAProxy bastion instance also forwards all traffic to the primary endpoint. You can find the primary endpoint in the ElastiCache service console. It’s included in the cluster’s configuration details.

Figure 2: Cluster details

HAProxy to forward incoming traffic

HAProxy is free and open-source software that allows us to load balance traffic between the Redis node in the private subnet and the port-forwarded EC2 instance. In our example, HAProxy serves one function: It forwards incoming traffic on a specified port to the primary endpoint created by Amazon ElastiCache. We use the Listen functionality of the HAProxy, which combines the client and server configurations. In simple use cases like this one, HAProxy listens for and forwards traffic at the same time. We listen to any incoming requests on the default Redis port of the instance running HAProxy (port 6379) and forward it to the ${RedisEndpoint} variable.

```
                  listen redis
                      bind *:6379
                      server server1 ${RedisEndpoint} maxconn 32 check-ssl ssl verify none
```

Metadata property of HAProxy

Instance metadata is data that you can use to configure or manage the running instance (for example, configuration scripts and commands that should be executed when the instance is created). In our example, instance metadata can be used to configure HAProxy upon creation of the instance. In the following example, the CloudFormation configuration creates the haproxy.cfg file under the /etc/haproxy/ directory. It also executes the starthaproxy command.

```
    Metadata:
      AWS::CloudFormation::Init:
        configSets:
          haproxy:
            - haproxy_config
            - haproxy_start
        haproxy_config:
          files:
            /etc/haproxy/haproxy.cfg :
              content: !Sub
                - |
                  global
                      daemon
                      maxconn 256

                  defaults
                      mode tcp
                      timeout connect 5000ms
                      timeout client 5000ms
                      timeout server 5000ms

                  listen redis
                      bind *:6379
                      server server1 ${RedisEndpoint} maxconn 32 check-ssl ssl verify none 
                -   { RedisEndpoint: !Ref RedisElastiCachePrimaryEndpoint }
              mode: '0644'
              owner: haproxy
              group: haproxy
        haproxy_start:
          commands:
            starthaproxy:
              cwd: /tmp
              command: "systemctl enable haproxy; service haproxy restart; service rsyslog restart"
```

When you’re managing a large number of instances, it might not be practical to add the Metadata section. In this case, you can use Systems Manager State Manager and the CloudFormation Association resource to target multiple instances by properties like tags or an instance ID. For information about how to implement this in a robust and scalable way, see the Using State Manager over cfn-init in CloudFormation and its benefits blog post.

SSM port forwarding to connect locally to the ElastiCache machine

Port forwarding for AWS Systems Manager Session Manager allows the user to create secure tunnels between the local environment and instances in private subnets. As a result, this setup does not require you to allow SSH access on the instance itself. We use this technique to create one tunnel and connect the local development environment with the provisioned HAProxy bastion host. The bastion host can use routing over private IP addresses inside the VPC and securely connect to our private ElastiCache cluster. By implementing this access pattern, we combine the port-forwarding ability of Systems Manager with the HAProxy load balancing configuration to forward incoming traffic from our local port to the EC2 instance. The instance, in turn, sends the traffic to the ElastiCache cluster.

Figure 3: Connection is forwarded to ElastiCache

AWS CLI version 2 offers an interface to create port-forwarding functionality. To use the start-session, install the Session Manager plugin for the AWS CLI. When you use the following command, replace $BASTION_HOST_INSTANCE_ID with the instance ID that contains your HAProxy deployment.

aws ssm start-session --target $BASHION_HOST_INSTANCE_ID \
          --document-name AWS-StartPortForwardingSession \
          --parameters '{"portNumber":["6379"],"localPortNumber":["6379"]}' --region eu-central-1

Test the solution

We have set up the following:

• An ElastiCache for Redis cluster running inside a private subnet.
• An EC2 instance that is configured to start with HAProxy and forwards incoming traffic to the Redis primary endpoint.
• An AWS CLI command that starts a port forwarding session from our local port to a port on the EC2 instance.

To test the end-to-end solution, execute the CLI command above in a shell window. The command will start a port forwarding session that allows you to communicate with Redis as if it were a locally running instance. Start a second shell window and use the following command to check the connection.

redis-cli ping

If the solution has been successfully implemented, you will receive a PONG:

Figure 4: PONG

Cleanup

To avoid charges in your account for the ElastiCache cluster and EC2 instance, open the CloudFormation console and delete the stack. Within the CloudFormation console, find the provisioned stack under the provided name and select the radio button next to the stack name. Once the stack is selected you can click on the delete button within the same console page.

Alternatively, from any location on your local machine you can execute an AWS CLI command that will have the same effect. AWS CLI command that will delete all of the resources takes only stack name as a parameter and looks like as follows:

aws cloudformation delete-stack --stack-name SsmComputeStack

Conclusion

In this blog post, we showed you how to use Session Manager to create a secure access pattern for interaction with resources in private subnets. We shared an example of a secure infrastructure where resources are encapsulated in private subnets. We demonstrated how to automatically deploy and configure an open source HAProxy server on top of an EC2 instance. Finally, using the example of an ElastiCache for Redis cluster, we demonstrated how you can use Systems Manager to seamlessly forward ports from a local development machine to a provisioned bastion host. In this way, you obtain access to reach all other private resources in your VPC without exposing any surface in your cloud infrastructure to attack.

Although the access pattern we shared uses HAProxy and ElastiCache for Redis, you can use its logic to develop a similar solution with different resources. For more information, see the Port Forwarding Using AWS Systems Manager Session Manager and Replacing a Bastion Host with Amazon EC2 Systems Manager blog posts.

About the authors