How to setup and use AWS OpsWorks for Chef Automate or Puppet Enterprise in an isolated subnet

Introduction

For enhanced security, it’s often preferable to place resources in AWS within a subnet with no internet access. Recently, AWS expanded the number of services supported by interface VPC endpoints within the AWS ecosystem, so it’s now easier than ever to run AWS OpsWorks for Chef Automate or AWS OpsWorks for Puppet Enterprise within an isolated subnet. The solution outlined in this blog post doesn’t use a NAT instance or a NAT gateway, so it is a fully private solution, which many users require.

Overview

This blog post gives you a functional example of how to set up an AWS OpsWorks for Chef Automate or Puppet Enterprise Servers in a private subnet and connect nodes to those resources. In this example, AWS OpsWorks for Chef Automate is used. You can achieve the same result for AWS OpsWorks for Puppet Enterprise using the architecture from this blog post. More information on that can be found the Troubleshooting section.

Note that although OpsWorks for Chef Automate is supported in the Asia Pacific (Singapore) and the Asia Pacific (Tokyo) Regions, the solution in this blog post is not yet supported in those Regions. This is because this solution requires VPC endpoints that are not yet available in those Regions. A full copy of the solution can be found on Amazon S3.

Components

In this solution, all required resources will be created, starting with using the Amazon VPC service to create a new VPC. The main template creates a few key resources:

Amazon VPC: A new virtual private cloud will be created for the account. It will have both private and public subnets.
Within the Amazon VPC, there are 3 subnets with multiple interface VPC endpoints, as well as gateway VPC endpoints and other resources (such as route tables) required for making a properly configured VPC. VPC Endpoints allow for service specific traffic to go directly to the necessary endpoints, therefore eliminating the need for an IGW or NAT to facilitate communication. They also offer benefits for security and performance. For the solution in this blog post, VPC endpoints for AWS Systems Manager, Amazon S3, Amazon EC2 Messages, EC2, and AWS CloudFormation are leveraged. All VPC endpoints are used in both the private and public route tables. For this solution, there is a public subnet for the bastion host, and two private subnets: one for the Chef Automate server and one for nodes.

There are also two optional child stacks. One stack will create the bastion host, while the other will provision the AWS OpsWorks for Chef Automate server.

Bastion host: This Amazon Linux EC2 instance is optionally created based on the CreateBastion parameter. If created, it will be placed in the public subnet of the new VPC, and will have the SSH key pair given by the parameter “myKeyPair.” Although this is optional, it is strongly recommend to set this up as part of the CloudFormation stack launch. In the following examples, this blog will demonstrate how to log in to the EC2 instances in the private subnet through the bastion host. It’s also worth noting that the bastion host installs and starts a squid proxy in its user data. This proxy helps improve data flow and overall experience.
AWS Lambda-backed custom resource: To provision an OpsWorks for Chef Automate server, an AWS Lambda-backed custom resource will be used. This function takes the VPC resources created before, and uses them as parameters for the Boto3 CreateServer API call. Note that this example demonstrates how you can automate the server creation as well, instead of manually creating the server using the AWS OpsWorks console. The server created as a result of this custom resource will be named private-subnet-opsworks.

The solution

The CloudFormation template first requests some information from the user, via CloudFormation parameters. Two of these parameters allow CloudFormation to determine if it should provision the conditional resources mentioned previously. The bastion host is optional because not all OpsWorks CM customers need SSH access to their Chef Automate server. However, it’s important to note that the bastion host is also required to reach the UI, so creation of the bastion host is strongly advised. Creating the Chef Automate server is also optional, if you want to create the underlying VPC components and then create the server using the AWS Management Console. Both optional templates will be available in Amazon S3, should only those resources be required.

A key component of this solution is the ability to communicate across AWS resources independent of the external internet, which is done via VPC endpoints. There are two kinds of VPC endpoints: gateway and interface, and both are used in this solution. The Amazon S3 VPC endpoint, for example, is a gateway VPC endpoint. For EC2 Message, EC2, Systems Manager, and CloudFormation, interface VPC endpoints are used. An interface VPC endpoint enables connection to services powered by AWS PrivateLink.

In this solution, there two types of subnets: private and public. There are two different sets of requirements for security groups. The bastion host, if created, will exist in the public subnet, so SSH access is required. Egress rules also need to be open, so that traffic can leave the bastion host to go to its destination.

The private subnets are going to contain the OpsWorks CM resources, so port 443 is open for HTTPS communications. To connect using SSH to these resources via the bastion host, port 22 will be open to the public subnet security group. This security group will also have port 3128 open, for the squid proxy. If adjustments are made to this solution, only 443 would need to remain open because the proxy and SSH could be removed.

Moving forward

After launching the CloudFormation stack, you can manage your AWS OpsWorks for Chef Automate server from your local workstation. If you do not already have the ChefDK installed, please do so before moving forward. For demonstration purposes, the following steps are for a Mac workstation, although the equivalent steps should work in any other Linux distribution or Windows OS. Note that these steps will require the starter kit and sign-in credentials from the Chef Automate dashboard, which can be downloaded from the OpsWorks console while the server is being created. If needed, you can reset these items in the AWS Opsworks console. The console will also contain the Chef Automate server URL. This URL will be used in the examples moving forward, and should be substituted for private-subnet-opsworks-xxx.us-west-2.opsworks-cm.io.

Accessing the server

To view the Chef Automate dashboard using the local browser on the workstation, you first need a way to resolve the domain of your OpsWorks server locally. To do this, edit the /etc/hosts file of the local workstation to include a line that establishes this relationship from the local workstation to the OpsWorks server:

127.0.0.1 private-subnet-opsworks-xxx.us-west-2.opsworks-cm.io

This entry in the /etc/hosts file is necessary before setting up a tunnel:

sudo ssh -N ec2-user@bastion-host-ip -L443:private-subnet-opsworks-xxx.us-west-2.opsworks-cm.io:443 -i key-file-bastion-host

It is possible to set up the tunnel on a different port of the workstation. However, this example establishes the tunnel using port 443. This requires administrative privileges, which is why sudo was used in the previous command. After running this command, open a web browser and input the URL as https://private-subnet-opsworks-xxx.us-west-2.opsworks-cm.io. This way, you can check out the OpsWorks Chef Automate dashboard easily from your workstation, instead of using the bastion host directly. After following these steps, you should see the sign in console for Chef Automate:

To SSH to the server, the bastion host is used as an intermediary between the local workstation and the OpsWorks CM server:

ssh -i key-file-bastion-host -f -L 4567:private-ip-owcm-server:22 ec2-user@bastion-host-public-ip “sleep 10” && ssh -i key-file-owcm-server -p 4567 ec2-user@localhost

Note that in this case, the same key is used in both the bastion host and the OpsWorks for Chef Automate server. This setup saves an extra step in the process of getting logs from the OpsWorks CM server, because it’s possible to connect right from the local workstation to the server, instead of having to first use SSH to connect to the bastion.

Cookbooks

After the OpsWorks CM server is successfully launched, download and unzip the starter kit. Because the starter kit contains some essential configuration files, all of these commands are run from within the unzipped directory.

First, let’s grab cookbooks from Berkshelf, which is a common dependency manager for cookbooks.

berks install

Next, make sure that you can successfully connect to the server, using knife ssl check:

sudo ssh -N ec2-user@bastion-host-public-ip -L443:private-subnet-opsworks-xxx.us-west-2.opsworks-cm.io:443 -i key-file-bastion-host & SSL_CERT_FILE='.chef/ca_certs/opsworks-cm-ca-2016-root.pem' knife ssl check

Then, go ahead and upload the cookbooks:

ssh -N ec2-user@bastion-host-public-ip -L443:private-subnet-opsworks-xxx.us-west-2.opsworks-cm.io:443 -i key-file-bastion-host & SSL_CERT_FILE='.chef/ca_certs/opsworks-cm-ca-2016-root.pem' berks upload

After this, navigate to the dashboard in the browser as described earlier ( in the “Accessing the server” section), and observe that the cookbooks uploaded from the workstation are now visible on the dashboard.

Node association

To verify node association for the new server, launch a new EC2 instance with the prerequisites for launching a OpsWorks CM node in any of the two private subnets from the VPC template. In this example, a private subnet is used to launch a nodes in order to illustrate a fully private solution within the AWS infrastructure. However, if you prefer you can use the bastion host subnet as well to launch the nodes to make them accessible from the internet.

In this scenario, it’s not possible to use the OpsWorks user data script for unattended nodes because Chef relies on external internet access to download scripts directly from their website when registering the node. So instead, knife bootstrap will be used to associate the nodes.

While the new node EC2 instance is being launched, create a new SSH tunnel to the OpsWorks CM server as demonstrated earlier. Then, run:

eval ssh-agentssh-add <your-private-key-for-node>.pem

Those commands will add the SSH key to the agent so that it can be used on the following bootstrap command via the SSH gateway.

SSL_CERT_FILE='.chef/ca_certs/opsworks-cm-ca-2016-root.pem' knife bootstrap <node-private-ip> --forward-agent --ssh-gateway ec2-user@bastion-host-public-ip -x ec2-user --sudo --node-name <nodename> --bootstrap-proxy "http://bastion-host-private-ip:3128"

In this command, the bastion host’s squid proxy is used because Chef relies on downloading some scripts from their website while registering the node. After running this command, confirm in the Chef Automate dashboard that the node was successfully associated.

Troubleshooting

My CloudFormation stack fails due to VPC account limits.

- If this stack causes you to go over your account limits for VPCs or VPC resources, you can either remove any unneeded resources, or you can contact AWS Support for further assistance.

My CloudFormation stack encountered CREATE_FAILED on custom resource.

- Logs for the Custom Resource Lambda function will be in Amazon CloudWatch, and should contain more details. The most common issue seen is duplicate server names.

I lost or was unable to obtain my OpsWorks for Chef Automate credentials.

- If you forget to download the OpsWorks for Chef Automate credentials, they can still be reset. A walkthrough can be found in AWS documentation.

I tried to delete my OpsWorks for Chef Automate Server, and I am getting an error.

- If the roles which were used to create the OpsWorks for Chef Automate server have since been deleted, the server can’t be fully terminated. To work around this, create new roles with the same names and permissions as those used in the template. Then, delete the server.

I keep getting “Connection Refused” when I run the knife commands using the SSH tunnel.

- The pipe redirection sometimes might not work as expected because the https port is used with the administrative privileges. If the port is already in use, the easiest fix is to close all existing terminal sessions and then run sudo ssh -N ec2-user@bastion-host-public-ip -L443:private-subnet-opsworks-xxx.us-west-2.opsworks-cm.io:443 -i key-file-bastion-host in a new shell. Open another shell while this one is still running and run the knife commands.
- Verify if the tunnel is still up and running in the shell where you ran the tunneling command. If the tunnel is broken, you could see the “Connection Refused” as well.

I tried launching the solution in the Tokyo Region but the CloudFormation stack failed to create.

- Currently the solution provided works only in the following AWS Regions: us-east-1, us-east-2, us-west-1, us-west-2, eu-west-1, eu-central-1 and ap-southeast-2. Other Regions aren’t currently supported.

What changes do I need to make for this solution to work for OpsWorks for Puppet Enterprise?

- Although the examples and steps in this blog are specific to the OpsWorks for Chef Automate server, just a few changes would allow this to work for OpsWorks for Puppet Enterprise. Firstly, on the master template, open Puppet specific ports 8142, 8143, 4433, 8140 in the PrivateSubnetSecurityGroup. Leave ports 443, 3128, and 22 as is. Then, in the custom resource template, adjust the CreateServer call in the Lambda function such that the ‘Engine’ parameter is set to ‘puppet’, the ‘EngineModel‘ is set to ‘Monolithic’, the ‘EngineVersion’ is set to ‘2017’.

Conclusion

This blog post walked you through how to create a wholly private configuration management environment. To eliminate the need for external internet connection through a NAT or IGW, VPC endpoints are leveraged. A bastion host is used, which, when combined with SSH tunneling, allows connection to the Chef Automate dashboard directly from a workstation. Node association for this private setup will require running knife bootstrap in the bastion host from the workstation while using the bastion host as an SSH gateway. To see the full solution, check out the templates on Amazon S3.

About the Authors

	Ramesh Venkataraman is a Cloud Support Engineer who supports customers using AWS CloudFormation and AWS OpsWorks and AWS ECS among other DevOps AWS services. Outside of work, Ramesh enjoys following stack overflow questions and answers them in any way he can.
	Maggie O’Toole has been a Cloud Support Engineer at AWS since 2017. She focuses on supporting customers in using DevOps technologies, specializes in containers and configuration management, and enjoys building out infrastructure.

AWS Cloud Operations & Migrations Blog