Announcing the Firecracker Open Source Technology: Secure and Fast microVM for Serverless Computing

中文版

New Challenges for Virtualization

Today, our customers can use serverless computing to build applications without worrying about provisioning or managing infrastructure. Developers can package their code as serverless containers with AWS Fargate or serverless functions with AWS Lambda. Our customers tell us they love the low operational overhead of serverless, and we believe it will continue to play a pivotal role in the future of computing.

As our customers increasingly adopted serverless, we realized that existing virtualization technologies were not developed to optimize for the event-driven, sometimes short-lived nature of these kinds of workloads. We saw a need to build virtualization technology specifically designed for serverless computing. We needed something that could give us the hardware virtualization-based security boundaries of virtual machines, while maintaining the smaller package size and agility of containers and functions.

Firecracker Technology

Meet Firecracker, an open source virtual machine monitor (VMM) that uses the Linux Kernel-based Virtual Machine (KVM). Firecracker allows you to create micro Virtual Machines or microVMs. Firecracker is minimalist by design – it includes only what you need to run secure and lightweight VMs. At every step of the design process, we optimized Firecracker for security, speed, and efficiency. For example, we can only boot relatively recent Linux kernels, and only when they are compiled with a specific set of configuration options (there are 1000+ kernel compile config options). Also, there is no support for graphics or accelerators of any kind, no support for hardware passthrough, and no support for (most) legacy devices.

Firecracker boots a minimal kernel config without relying on an emulated bios and without a complete device model. The only devices are virtio net and virtio block, as well as a one-button keyboard (the reset pin helps when there’s no power management device). This minimal device model not only enables faster startup times (< 125 ms on an i3.metal with the default microVM size), but also reduces the attack surface, for increased security. Read more details about Firecracker’s promise to enable minimal-overhead execution of container and serverless workloads.

In the fall of 2017, we decided to write Firecracker in Rust, a modern programming language that guarantees thread and memory safety and prevents buffer overflows and many other types of memory safety errors that can lead to security vulnerabilities. Read more details about the features and architecture of the Firecracker VMM at Firecracker Design.

Firecracker microVMs improve efficiency and utilization with a low memory overhead of < 5 MiB per microVMs. This means that you can pack thousands of microVMs onto a single machine. You can use an in-process rate limiter to control, with fine granularity, how network and storage resources are shared, even across thousands of microVMs. All hardware compute resources can be safely oversubscribed, to maximize the number of workloads that can run on a host.

We developed Firecracker with the following guiding tenets (unless you know better ones) for the open source project:

• Built-In Security: We provide compute security barriers that enable multitenant workloads, and cannot be mistakenly disabled by customers. Customer workloads are simultaneously considered sacred (shall not be touched) and malicious (shall be defended against).
• Light-Weight Virtualization: We focus on transient or stateless workloads over long-running or persistent workloads. Firecracker’s hardware resources overhead is known and guaranteed.
• Minimalist in Features: If it’s not clearly required for our mission, we won’t build it. We maintain a single implementation per capability.
• Compute Oversubscription: All of the hardware compute resources exposed by Firecracker to guests can be securely oversubscribed.

We open sourced this foundational technology because we believe that our mission to build the next generation of virtualization for serverless computing has just begun.

Firecracker Usage

AWS Lambda uses Firecracker as the foundation for provisioning and running sandboxes upon which we execute customer code. Because Firecracker provides a secure microVM which can be rapidly provisioned with a minimal footprint, it enables performance without sacrificing security. This lets us drive high utilization on physical hardware, as we can now optimize how we distribute and run workloads for Lambda, mixing workloads based on factors like active/idle periods, and memory utilization.

Previously, Fargate Tasks consisted of one or more Docker containers running inside a dedicated EC2 VM to ensure isolation across Tasks. These Tasks now execute on Firecracker microVMs, which allows us to provision the Fargate runtime layer faster and more efficiently on EC2 bare metal instances, and improve density without compromising kernel-level isolation of Tasks. Over time, this will allow us to continue to innovate at the runtime layer, giving our customers even better performance while maintaining our high security bar, and lowering the overall cost of running serverless container architectures.

Firecracker runs on Intel processors today, with support for AMD and ARM coming in 2019.

You can run Firecracker on AWS .metal instances, as well as on any other bare-metal server, including on-premises environments and developer laptops.

Firecracker will also enable popular container runtimes such as containerd to manage containers as microVMs. This allows Docker and container orchestration frameworks such as Kubernetes to use Firecracker. We have built a prototype that enables containerd to manage containers as Firecracker microVMs and would like to with with community to take it further.

Getting Started with Firecracker

Getting Started with Firecracker provides detailed instructions on how to download the Firecracker binary, start Firecracker with different options, build from the source, and run integration tests. You can run Firecracker in production using the Firecracker Jailer.

Let’s take a look at how to get started with using Firecracker on AWS Cloud (these steps can be used on any bare metal machine):

Create an i3.metal instance using Ubuntu 18.04.1.

Firecracker is built on top of KVM and needs read/write access to /dev/kvm. Log in to the host in one terminal and set up that access:

sudo setfacl -m u:${USER}:rw /dev/kvm

Download and start the Firecracker binary:

curl -L https://github.com/firecracker-microvm/firecracker/releases/download/v0.11.0/firecracker-v0.11.0
./firecracker-v0.11.0 --api-sock /tmp/firecracker.sock

Each microVM can be accessed using a REST API. In another terminal, query the microVM:

curl --unix-socket /tmp/firecracker.sock "http://localhost/machine-config"

This returns a response:

{ "vcpu_count": 1, "mem_size_mib": 128,  "ht_enabled": false,  "cpu_template": "Uninitialized" }

This starts a VMM process and waits for the microVM configuration. By default, one vCPU and 128 MiB memory are assigned to each microVM. Now this microVM needs to be configured with an uncompressed Linux kernel binary and an ext4 file system image to be used as root filesystem.

Download a sample kernel and rootfs:

curl -fsSL -o hello-vmlinux.bin https://s3.amazonaws.com/spec.ccfc.min/img/hello/kernel/hello-vmlinux.bin
curl -fsSL -o hello-rootfs.ext4 https://s3.amazonaws.com/spec.ccfc.min/img/hello/fsfiles/hello-rootfs.ext4

Set up the guest kernel:

curl --unix-socket /tmp/firecracker.sock -i \
    -X PUT 'http://localhost/boot-source'   \
    -H 'Accept: application/json'           \
    -H 'Content-Type: application/json'     \
    -d '{        "kernel_image_path": "./hello-vmlinux.bin", "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"    }'

Set up the root filesystem:

curl --unix-socket /tmp/firecracker.sock -i \
    -X PUT 'http://localhost/drives/rootfs' \
    -H 'Accept: application/json'           \
    -H 'Content-Type: application/json'     \
    -d '{        "drive_id": "rootfs",        "path_on_host": "./hello-rootfs.ext4",        "is_root_device": true,        "is_read_only": false    }'

Once the kernel and root filesystem are configured, the guest machine can be started:

curl --unix-socket /tmp/firecracker.sock -i \
    -X PUT 'http://localhost/actions'       \
    -H  'Accept: application/json'          \
    -H  'Content-Type: application/json'    \
    -d '{        "action_type": "InstanceStart"     }'

The first terminal now shows a serial TTY prompting you to log in to the guest machine:

Welcome to Alpine Linux 3.8
Kernel 4.14.55-84.37.amzn2.x86_64 on an x86_64 (ttyS0)
localhost login:

Log in as root with password root to see the terminal of the guest machine:

localhost login: root
Password:
Welcome to Alpine! 

The Alpine Wiki contains a large amount of how-to guides and general information about administrating Alpine systems. 

See <http://wiki.alpinelinux.org>. 

You can setup the system with the command: setup-alpine 

You may change this message by editing /etc/motd.

login[979]: root login on 'ttyS0' 
localhost:~#

You can see the filesystem using ls /

localhost:~# ls /
bin         home        media       root        srv         usr
dev         lib         mnt         run         sys         var
etc         lost+found  proc        sbin        tmp

Terminate the microVM using the reboot command. Firecracker currently does not implement guest power management, as a tradeoff for efficiency. Instead, the reboot command issues a keyboard reset action which is then used as a shutdown switch.

Once the basic microVM is created, you can add network interfaces, add more drives, and continue to configure the microVM.

Want to create thousands of microVMs on your bare metal instance?

for ((i=0; i<1000; i++)); do
    ./firecracker-v0.10.1 --api-sock /tmp/firecracker-$i.sock &
done

Multiple microVMs may be configured with a single shared root file system, and each microVM can then be assigned its own read/write share.

Firecracker and Open Source

It is our mission to innovate on behalf of and for our customers, and we will continue to invest deeply in serverless computing at all three critical layers of the stack: the application, virtualization, and hardware layers. We want to offer our customers their choice of compute, whether instances or serverless, with no compromises on security, scalability, or performance. Firecracker is a fundamental building block for providing that experience.

Investing deeply in foundational technologies is one of the key ways that we at AWS approach innovation – not for tomorrow, but for the next decade and beyond. Sharing this technology with the community goes hand-in-hand with this innovation. Firecracker is licensed under Apache 2.0. Please visit the Firecracker GitHub repo to learn more and contribute to Firecracker.

By open sourcing Firecracker, we not only invite you to a deeper examination of the foundational technologies that we are building to underpin the future of serverless computing, but we also hope that you will join us in strengthening and improving Firecracker. See the Firecracker issues list and the Firecracker roadmap for more information.