Integration & Automation

Deploy bastion hosts into private subnets with AWS CDK

Deploying bastion hosts in private subnets is a way to provide temporary and limited access to non-production private resources in a virtual private cloud (VPC). Bastion hosts typically sit in public subnets. But in a non-production environment, if you want to allow a group of developers to access an private resource, you might not want to use a bastion host accessible from the internet.

In this post, we explain how to provision scalable and extendable secure bastion hosts in private subnets using AWS Cloud Development Kit (AWS CDK). First, I’ll show you how to configure AWS CDK and clone the GitHub repository I’ve prepared. Second, we demonstrate using AWS CDK to define the target environment and deploy the bastion host stack into a new or existing VPC.

Finally, we create an AWS Identity and Access Management (IAM) policy. We demonstrate how users with that policy can access the bastion host using three operations. We use the AWS Systems Manager Session Manager plugin for the AWS Command Line Interface (AWS CLI), SSH (Secure Shell), and the AWS Management Console. Our process follows AWS Security best practices of granting least privilege and using roles to delegate permissions.

About this blog post
Time to read ~8 min.
Time to complete ~30 min.
Cost to complete Costs depend on the VPC resources provisioned. For more information, refer to Amazon VPC pricing.
Learning level Intermediate (200)
AWS services AWS Systems Manager
AWS CDK
AWS CloudFormation
Amazon VPC
Amazon Elastic Compute Cloud (Amazon EC2)
IAM
AWS Secrets Manager

Overview

Figure 1 shows the high-level architecture of our process. Only IAM users can access the bastion host in the private subnet of the VPC. They can use the Session Manager plugin for AWS CLI, SSH, or the AWS Systems Manager console.

Figure 1. Accessing a bastion host in the private subnet

The GitHub repository we provide contains two stacks. AwsBastion-NetworkCdkStack deploys a new VPC, but you can also use an existing VPC. If you use an existing VPC, you won’t deploy the first stack. AwsBastion-Ec2CdkStack deploys into the private subnet the bastion host, EC2 key pair, security group, and IAM role and policy.

As we show in the walkthrough, we use the allowedSecurityGroups parameter in cdk.json to define the resources to which the bastion host connects. In the example shown in Figure 1, the bastion host connects to an Amazon RDS database instance in an isolated subnet.

Prerequisites

Before getting started, make sure that you have the following.

Walkthrough

Step 1: Configure deployment environment

  1. Create an IAM user in your AWS account. Make sure that you configure programmatic access with AWS access key ID and AWS secret access key and add IAM identity permissions to the AdministratorAccess managed policy.
  2. Run the following command to create a profile named bastion-cdk.

$ aws configure --profile bastion-cdk

  1. The AWS CLI prompts you for four configuration settings. The following example shows sample values. Replace AWS Access Key ID and AWS Secret Access Key with the credentials for the user you create in step 1.

AWS Access Key ID [None]: <AKIAIOSFODNN7EXAMPLE>
AWS Secret Access Key [None]: <wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY>
Default region name [us-east-1]: <us-east-1>
Default output format [json]: <json>

  1. (Optional) Provide a Default region name and Default output format.
  2. Clone the GitHub repository I’ve created for you.

$ git clone https://github.com/aws-samples/secure-bastion-cdk

  1. Navigate to the repository’s root directory.

$ cd secure-bastion-cdk

  1. Run the following cdk command to bootstrap your AWS environment. The cdk command is the primary tool for interacting with an AWS CDK application.

$ cdk bootstrap aws://{account_id}/{your_selected_region}

Note: Bootstrapping launches resources into your AWS environment that are required by AWS CDK. These include an Amazon Simple Storage Service (Amazon S3) bucket for storing files and IAM roles that grant the necessary permissions to run the deployment.

Step 2: Configure target environments

In this section, we explain how to edit cdk.json to define the environment to deploy.

In cdk.json, deploy the bastion host into the private subnet by entering a value for the existingVpcId parameter. To deploy a new VPC, keep the existingVpcId parameter blank and specify VPC settings in the vpcConfig section. In the allowedSecurityGroups section, enter the IDs of the security groups to which you want the bastion host to connect.

In the following cdk.json code example, dev is the name of the environment.

"dev": {
      "region": "eu-central-1",
      "prefix": "dev",
      "existingVpcId": "",
      "instances": [
        {
         "instanceId": "BastionHost",
         "instanceType": "t3.medium",
         "keyName": "BastionHostKey",
         "allowedSecurityGroups": "sg-ps-rds","sg-is-rds1","sg-is-rds2"
        }
       ],
       "vpcConfig":{
         "cidr": "10.100.0.0/17",
         "maxAZs": 3,
         "isolatedSubnetCidrMask": 23,
         "privateSubnetCidrMask": 20,
         "publicSubnetCidrMask": 23,
         "ssmPrefix": "/bastion/network"
       }
  },

The bastion host is configured to access security groups sg-ps-rds, sg-is-rdsdb1, and sg-isrds. Because existingVpcId is blank, a new VPC is configured using the vpcConfig parameters.

In the next section, you can define multiple environments in cdk.json and specify the one you want to deploy using AWS CDK.

Step 3: Stack deployment

Now let’s deploy the stacks.

Note: If you use an existing VPC in cdk.json, deploy only the second stack.

  1. Run the following Npm commands.

$ npm install
$ npm test

  1. If you specified an existing VPC in the cdk.json file, skip to step 4. To deploy a new VPC, run the following command. In the example shown, replace environment with the name of an environment from the cdk.json file. Replace the account_ID value with the ID of the target AWS account.

$ cdk deploy -c environment="<environment>" -c account="<account_ID>" AwsBastion-NetworkCdkStack --profile bastion-cdk

  1. When you’re prompted to deploy the changes, choose y. If it’s successful, AWS CDK returns the CIDR block and ID of the VPC (Outputs), and the Amazon Resource Name (ARN) of the stack (Stack ARN).
  2. Run the following command to deploy AwsBastion-Ec2CdkStack. In the example shown, replace environment with the name of an environment from the cdk.json file. Replace the account_ID value with the ID of the target AWS account.

$ cdk deploy -c environment="<environment>" -c account="<account_ID>" AwsBastion-Ec2CdkStack --profile bastion-cdk

  1. AWS CDK shows the IAM Statement Changes, IAM Policy Changes, and Security Group Changes that are deployed. When prompted, choose y to deploy the changes. If it’s successful, AWS CDK returns the stack’s ARN (Stack ARN).

Step 4: Create an IAM policy and user

In this section, we create an IAM policy that enables an IAM user to use Session Manager. We create a user and attach the policy to it. Then, in the next section, we demonstrate three operations an IAM user can use to connect to it.

  1. Create an IAM policy named bastion test using the following code. In the code provided, replace {account_id} with the ID of the target AWS account. To limit access to secret resources using tags, add a value for {tag_key} and {tag_value}.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:StartSession"
            ],
            "Resource": [
                "arn:aws:ec2:*:{account_id}:instance/*"
            ],
            "Condition": {
                "StringLike": {
                    "ssm:resourceTag/Name": [
                        "BastionHost"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeImages",
                "ec2:DescribeTags",
                "ec2:DescribeSnapshots"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:StartSession"
            ],
            "Resource": [
                "arn:aws:ssm:*::document/AWS-StartPortForwardingSession",
                "arn:aws:ssm:*::document/AWS-StartSSHSession"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:TerminateSession"
            ],
            "Resource": [
                "arn:aws:ssm:*:*:session/${aws:username}-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds",
                "secretsmanager:ListSecrets"
            ],
            "Resource": [
                "arn:aws:secretsmanager:*:{account_id}:secret:*"
            ],
            "Condition": {
                "StringEquals": {
                    "secretsmanager:ResourceTag/{tag_key}": "{tag_value}"
                }
            }
        }
    ]
}
  1. Create an IAM user in your AWS account and attach the bastion test policy to it.
  2. Configure the AWS profile.

aws configure --profile bastion-test

  1. The AWS CLI prompts you for four configuration settings. The following example shows sample values. Replace AWS Access Key ID and AWS Secret Access Key with the credentials for the user created in step 2.

AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

  1. (Optional) Provide a Default region name and Default output format.
  2. Run the following command.
$INSTANCE_ID=$(aws ec2 describe-instances \<br />
   --filter "Name=tag:Name,Values= BastionHost" \<br />
   --query "Reservations[].Instances[?State.Name =='running'].InstanceId[]" \<br />--output text 
   --profile bastion-test)

Step 5: Connect to the bastion host

In this section we demonstrate three ways an IAM user can connect to the bastion host.

Operation 1: Connect using AWS Systems Manager Session Manager

Use Session Manager to connect to the bastion host with the following command.

aws ssm start-session --target $INSTANCE_ID --profile bastion-test

Alternatively, use Session Manager to run the following command syntax to connect with port forwarding. Replace remote_port_number and your_local_port_number with the target remote and local port numbers.

aws ssm start-session --target $INSTANCE_ID \
--document-name AWS-StartPortForwardingSession \
--parameters '{"portNumber":["{remote_port_number}"],"localPortNumber":["{your_local_port_number}"]}' --profile bastion-test

Operation 2: Connect using SSH

Use this operation to connect to the bastion host using SSH.

  1. Update the SSH configuration file to allow a proxy command to run Session Manager. From a command prompt, open the SSH configuration file in the Vim editor.

vim ~/.ssh/config
Add the following to the configuration file.
#Add SSH over Session Manager
host i-* mi-*
ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

  1. Press Esc.
  2. Enter :wq and press Enter.
  3. Retrieve the host key pair for the bastion host from AWS Secrets Manager. This is created as part of the AwsBastion-Ec2CdkStack stack deployment.

aws secretsmanager get-secret-value \
--secret-id ec2-ssh-key/BastionHostKey/private \
--query SecretString \
--output text --profile bastion-test > bastion-key-pair.pem

  1. Run the following command.

chmod 400 bastion-key-pair.pem

  1. Run the following command to open an SSH tunnel. In the syntax shown, replace local_port with your local port. Replace host with the host IP address (such as an Amazon RDS endpoint). Replace remote_port with the target remote port (such as 3306 for a MySQL server).

ssh -f -N ec2-user@$INSTANCE_ID -L {local_port}:{host}:{remote_port} -i bastion-key-pair.pem
Enter yes when prompted to continue connecting. The $INSTANCE_ID will be added to the list of known hosts.

Operation 3: Connect using the AWS Management Console

A third operation is to connect to the bastion host using Systems Manager in the AWS Management Console.

  1. Sign into the AWS Systems Manager console as the IAM user that you created in “Step 4: Create an IAM policy and user”.
  2. In the navigation pane under Node Management, choose Session Manager.
  3. Choose Start session.
  4. (Optional) Enter a reason for connecting to the instance in the Reason for session – optional field.
  5. For Target instances, choose the BastionHost instance. This is the instance AwsBastion-Ec2CdkStack deploys.
  6. Choose Start session.
  7. In the Session Manager terminal, you can run bash or PowerShell commands to connect to the private resources configured in cdk.json. When you finish, close the terminal to end Session Manager.

Cleanup

To avoid unexpected charges to your account, delete the resources you deployed during the walkthrough.

  • If you deployed the VPC stack (AwsBastion-NetworkCdkStack), run the following command to delete it.

cdk destroy -c environment="<environment_name>" -c account="<ACCOUNT ID>" AwsBastion-NetworkCdkStack --profile bastion-cdk

Choosey when prompted to confirm stack deletion. If it’s successful, AWS CDK returns AwsBastion-Ec2CdkStack: destroyed.

  • Delete the bastion host stack (AwsBastion-NetworkCdkStack) by running the following command. Replace the environment and account values with the ones used to deploy the stack.

cdk destroy -c environment="<environment_name>" -c account="<account_ID>" AwsBastion-Ec2CdkStack --profile bastion-cdk

Choose y when prompted to confirm stack deletion. If successful, AWS CDK returns AwsBastion-NetworkCdkStack: destroyed.

Conclusion

In this blog post, we demonstrated using AWS CDK to deploy a bastion host in the private subnet of a new or existing VPC. By deploying a bastion host in the private subnet, you can provide secure access to private subnet resources in your VPC.

We invite you to adapt the code provided in our GitHub repository. For extra credit, we challenge you to integrate the process I’ve shown with a continuous integration and continuous delivery (CI/CD) pipeline. In this way, you can automatically deploy secure bastion hosts into private subnets across multiple environments. To get started, see Continuous integration and delivery (CI/CD) using CDK Pipelines, then aws-cdk/pipelines module for a construct library. For more about AWS CDK and Amazon RDS, refer to Use AWS CDK to initialize Amazon RDS instances.

Use the comments section to let me know if you have any questions.

About the authors

Ramy Nasreldin

Ramy Nasreldin is a DevOps architect at AWS, based in Sweden. Ramy helps customers design and implement their systems to run on the AWS Cloud. He also preaches best practices by automating everything, from infrastructures to application delivery, to achieve the most resilient and scalable solutions that best serve end users in a sustainable way. In his spare time, he enjoys swimming, playing football, and spending time with his family.

Rolando Santamaria Maso

Rolando is a senior cloud application development consultant at AWS Professional Services, based in Germany. He helps customers migrate and modernize workloads in the AWS Cloud, with a special focus on modern application architectures and development best practices, but he also creates IaC using AWS CDK. Outside work, he maintains open-source projects and enjoys spending time with family and friends.

Prasanna Tuladhar Gitelman

Prasanna is a cloud infrastructure architect at AWS Professional Services, based in Germany. He likes to explore new challenges, be it databases, containers, or cloud infrastructures. Outside work, he likes jogging, hiking, and spending time with his family.