Getting Started with Bottlerocket and Certified AWS Partners

By Curtis Rissi, Sr. Solutions Architect – Containers & App Modernization at AWS

Bottlerocket is an open source Linux-based operating system (OS) purpose-built for hosting containers. It focuses on security and maintainability, providing a reliable, consistent, and safe platform for container-based workloads.

To accomplish this, Bottlerocket simplifies the OS by only including the essential software needed to run containers. This is important for a couple of reasons. First, it reduces the footprint of the underlying OS, aiding overall performance. Second, it limits the exposure to potential attacks, as many of the tools leveraged by would-be attackers simply aren’t there.

Bottlerocket also implements Security-Enhanced Linux (SELinux) in enforcing mode, as well as Device Mapper’s “verity” target (dm-verity). These enforce strict access control policies, further confining the applications and daemons running on the system, and help prevent rootkit-based attacks.

Building on top of this foundation of security, the process for performing system updates to Bottlerocket is handled in an atomic manner. This means that version upgrades and rollbacks can be handled quickly and in a more reliable and fully automated manner.

Extending beyond the benefits of Bottlerocket itself, there are unique opportunities for AWS Partner Network (APN) Partners to leverage the OS and benefit their downstream customers.

In this post, we’ll go into these opportunities, but first I will show you how easy it is for APN Partners and AWS customers to get up and running with Bottlerocket. I’ll also discuss some important constructs that everyone should understand when building solutions on top of Bottlerocket.

Setting Up an Amazon EKS Cluster with Bottlerocket

Prerequisites

In order to complete this walk through, you will need to follow the instructions on Getting Started with eksctl to install and configure eksctl, kubectl, and the AWS Command Line Interface (CLI).

Creating an EKS Cluster Using eksctl

Let’s get started with a simple cluster, leveraging Bottlerocket Amazon Machine Images (AMI) for the underlying node groups. To do this, we’ll be using eksctl to provision our cluster using a simple YAML config file.

In the file, we’ll provide a name for the cluster as well as the region and version we want. We also need to specify the node group details, such as the type of Amazon Elastic Compute Cloud (Amazon EC2) instance, how many of them and the AMI family to use—in this case, Bottlerocket.

It’s also important to note the AWS Identity and Access Management (IAM) policies we’re going to attach to the nodes, as these are necessary for them to be configured properly and connect to the Amazon Elastic Kubernetes Service (Amazon EKS) control plane during provisioning.

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: bottlerocket
  region: us-west-2
  version: '1.17'

nodeGroups:
  - name: ng-bottlerocket
    instanceType: m5.large
    desiredCapacity: 3
    amiFamily: Bottlerocket
    iam:
       attachPolicyARNs:
          - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
          - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
          - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
          - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
    ssh:
        allow: true
        publicKeyName: <YOUR_EC2_KEYPAIR_NAME>
    bottlerocket:
      settings:
        motd: "Hello from eksctl!"

Note: Replace “<YOUR_EC2_KEYPAIR_NAME>” with a valid EC2 Keypair you would like to use with this environment.

Now, save the above YAML code to a file on your development machine and name it “bottlerocket-cluster.yaml.”

To create the cluster, enter the following command leveraging your config file:

eksctl create cluster --config-file ./bottlerocket-cluster.yaml

Once the command completes, your cluster will have been created and you should be able to see your nodes in your node group:

kubectl get nodes

This will return a list of your existing nodes:

Figure 1 – Listing nodes added to an Amazon EKS cluster.

To confirm these are Bottlerocket nodes, you can use the following command:

kubectl get nodes -o=wide

This will return a list of useful Nodes details including their underlying OS:

Figure 2 – Listing nodes in an Amazon EKS cluster along with the OS image.

At this point, these nodes are available in your cluster for pods to be scheduled on.

Let’s go ahead and test this with a simple application:

kubectl run hello-world --quiet --rm -i --image=busybox --restart=Never -- echo "Hello from Amazon EKS"

This should output a "Hello from Amazon EKS" message, denoting the nodes in your cluster were able to accept the scheduling of a pod. With that, you have a Bottlerocket-based EKS cluster up and running.

Considerations

Now that you see how easy it is to get up and running with Bottlerocket, let’s discuss some of the tools and considerations you should understand in order to effectively manage, update, and build workloads on top of it.

Bottlerocket includes only the essential software to run containers, which improves resource utilization and reduces the attack surface compared to general-purpose operating systems. However, this also means it doesn’t include certain libraries and binaries that are common to general purpose operating systems.

This should have no impact on the containerized workloads you run, but it does have implications for some scenarios. These include performing version upgrades, developing kernel modules, and even troubleshooting errors. For these scenarios, we provide a couple of resources and tools to help.

Control Container

The first tool at your disposal is the Control Container. Every Bottlerocket host comes with a Control Container that runs in a separate instance of containerd. This container can be connected to via AWS Systems Manager and allows you to leverage the Bottlerocket API to configure and manage the Bottlerocket host. This is useful for viewing or changing settings with regard to the underlying Kubernetes config, host containers, time, and even performing updates.

To test the Control Container, let’s use the API to list out the current settings for one of our Bottlerocket nodes.

Execute the the following AWS CLI command, making sure to replace “<INSTANCE ID>” with the one for the Bottlerocket node you want to run this on:

aws ssm send-command --instance-ids "<INSTANCE ID>" \
    --document-name "AWS-RunShellScript" \
    --comment "Bottlerocket API Settings" \
    --parameters commands="apiclient -u /settings" \
    --output text \
    --query "Command.CommandId"

This will run the “apiclient -u /settings” command in our Bottlerocket host’s Control Container using AWS Systems Manager Run Command and return a “CommandId” as a response.

Now, we will check the output of that command using Run Command’s “list-command-invocations”:

aws ssm list-command-invocations --command-id "<COMMAND ID>" \
    --details \
    --query 'CommandInvocations[*].CommandPlugins[*][Output]'

Note: Make sure to replace “<COMMAND ID>” with yours from the previous step.

This will show you a JSON output of the “send-command” command listing out the Bottlerocket host’s settings. One of the settings we can change is enabling the Admin Container.

For this one, lets create a simple JSON file that we’ll pass Run Command. Copy the following to a file called “enable-admin.json”:

{
    "Parameters": {
        "commands": [
            "#!/bin/bash",
            "apiclient -u /settings -m PATCH -d '{\"host-containers\": {\"admin\": {\"enabled\": true}}}'",
            "apiclient -u /tx/commit_and_apply -m POST"
        ]
    }
}

In this file, we’ll run two commands. One that patches the settings, and another that will commit and apply the patch.

To run this, use the following command:

aws ssm send-command --instance-ids "<INSTANCE ID>" \
--document-name "AWS-RunShellScript" \
--comment "Bottlerocket API Enable Admin" \
--cli-input-json file://enable-admin.json
--output text \
--query "Command.CommandId"

Note: Make sure to replace “<INSTANCE ID>” with your Bottlerocket nodes Instance ID.

Like before, this will return a “CommandId”. Using that CommandId, run the following command to validate the output:

aws ssm list-command-invocations --command-id "<COMMAND ID>" \
    --details \
    --query 'CommandInvocations[*].CommandPlugins[*][Output]'

Note: Make sure to replace “<COMMAND ID>” with yours from the previous step.

This will return a JSON object container “[“settings.host-containers.admin.enabled”], denoting the admin container has been enabled.

Admin Container

The administrative container has an SSH server that lets you log in as ec2-user using your EC2-registered SSH key. It runs outside of Bottlerocket’s container orchestrator in a separate instance of containerd and is disabled by default.

The admin container is useful during development and/or troubleshooting scenarios that require shell access to the underlying host. Rremember, Bottlerocket doesn’t come with a shell built into the OS.

The admin container is something that should typically be leveraged for development and troubleshooting processes only. Outside of that, we suggest you keep the container disabled.

Update Process

Unlike general-purpose Linux distributions that include a package manager allowing you to update and install individual pieces of software, Bottlerocket downloads a full filesystem image and reboots into it. It can automatically roll back if boot failures occur, and workload failures can trigger manual rollbacks. This simplifies the update processes and makes it easier, faster, and safer to perform updates through automation.

There are also multiple mechanisms you can leverage to perform updates—the Bottlerocket API, updog CLI tool, and the Bottlerocket Update Operator for Kubernetes.

For the sake of this walk through, I want to show how updates work rather than the nuances of each mechanism. For this, we’ll use the admin container and the updog CLI tool.

First, you will need to connect to the admin container of your Bottlerocket node via SSH by using the following command:

ssh -i <YOUR_EC2_KEYPAIR_FILE>.pem ec2-user@<INSTANCE IP>

Note: Make sure to replace “<YOUR_EC2_KEYPAIR_FILE>” with the name of your Keypair and the “<INSTANCE IP>” with the IP of your Bottlerocket Node that you can glean, for example, from kubectl get nodes -o wide.

Next, use sheltie to drop into a root shell on your bottle rocket node:

sudo sheltie

To check for updates:

updog check-update

If an update is available, you can initiate an update:

updog update

This will download the new update image and update the boot flags so that when you reboot it will attempt to boot to the new version.

When that’s complete, you can reboot:

reboot

And that’s it. The node is now running the latest version of Bottlerocket OS.

Now that you understand the high-level process for updates, please know our preferred mechanism is for you to use the Bottlerocket Update Operator for Kubernetes. It simplifies the process and automates the update steps, including draining, cordoning-off each host, and safely moving your workloads.

Getting Started with APN Partner Use Cases

Now that we have a high-level understanding of what Bottlerocket is and how you can use it, let’s talk about how APN Partners can leverage it for their offerings.

• APN Consulting Partners can leverage Bottlerocket within their existing offerings by simply making it their default OS for container implementations. By doing this, APN Partners can immediately provide the security, performance, and management benefits of Bottlerocket to their customers. AWS Managed Service Providers (MSPs) can also leverage Bottlerocket as a means to simplify management and enhance the security of these offerings.
  .
• APN Technology Partners have benefits in supporting and ensuring their applications run on top of Bottlerocket. As customers adopt Bottlerocket for their own workloads, it can provide them with confidence to leverage APN Partner applications. Partners can also use it to power the nodes for their container-based SaaS offerings, which helps simplify management and enhance the security of these offerings.

These are just a few of the scenarios in which APN Partners can leverage Bottlerocket.

The Bottlerocket Launch Partners featured below all went through security and quality assurance tests, and provide support for their products on Bottlerocket.

Certified Bottlerocket Partners

As of launch, many APN Partners provide support for their applications on Bottlerocket, giving customers confidence to run popular partner applications. We will continue to expand this list going forward.

Monitoring & Logging Partners

• Datadog
• Epsagon
• LogicMonitor
• New Relic
• Splunk

Security Partners

• Alcide
• Aqua Security
• NeuVector
• Sysdig
• Trend Micro
• Tigera

Management & DevOps Partners

• Armory
• GitLab
• Kasten
• Puppet
• Weaveworks

Clean Up Resources

To delete your cluster and ensure you do not continue to incur costs from this walk through, run the following command:

eksctl delete cluster bottlerocket

This will delete the AWS CloudFormation stacks for the node group, and the Amazon EKS cluster along with the resources associated with them.

Summary

Through very simple means, APN Partners and AWS customers can quickly get up and running with Bottlerocket. Our goal is to continue to enhance the capabilities of our APN Partners by working together to incorporate it within their offerings and services, so that our mutual customers benefit from the reduced footprint, improved security, performance, and automation capabilities of the OS.

To get started using Bottlerocket on Amazon EKS, Amazon Elastic Container Service (Amazon ECS), or Amazon EC2 in all commercial regions, you can fork the source on the GitHub repository and make your changes by following our building guide.

For more details, see the Bottlerocket FAQs. If you’re an APN Partner interested in becoming a Certified Bottlerocket Partner, please reach out to your APN representative to start the conversation.