Building Open Source Communities at AWS Serverless

In this post, I will introduce you to the amazing open source community around serverless developer tools that I’ve been a part of since 2016. I wasn’t aware of the scale and potential of the open source community until I embarked on my own journey of open sourcing two strategic products at AWS.

About open source at Amazon

At Amazon, we believe in and support open source software. We have contributed to hundreds of projects for over a decade, and continue to widen our collaboration, increase code contribution, and help sustain healthy open source communities. We have hundreds of orepos on GitHub where we have open sourced everything from SDKs, client side tools, open source JDK, Open Distro for Elasticsearch to core components of some of our services such as s2n and Firecracker.

Serverless created an opportunity for tool builders

In 2014, we launched AWS Lambda, our first serverless compute service, which lets you run code without worrying about servers. When you build applications on Lambda, you don’t have to manage servers, you pay only for what you use, your apps scale automatically with usage, and are highly-available and fault-tolerant by design. Your serverless application would typically use other AWS serverless services to provide API endpoints, databases, persistent stores, etc.

A lot of the heavy lifting is handled by AWS services for you. Using the Infrastructure-as-Code pattern, you write configuration files to provision services for the API endpoints, user auth, workflows etc. Once provisioned, AWS takes care of running the services for you. This means that you can focus primarily on the business logic, and just write a bit of plumbing here and there. This also means that all your code-first developer tools such as IDEs, test harnesses, build systems, deployment engines, etc. need to support authoring and managing the configuration.

Open source serverless developer tools

A slew of developer tools have been created to help developers feel comfortable with the dramatic simplification that serverless offers. Guess what? The majority of them are open source! Let’s look at some:

• Serverless Framework is the biggest and most popular tool by far, judging by open source community stats: 30k stars on Github, plenty of downloads on NPM, > 500 contributors, and a rich set of plugins built by the community.
• There are several programming language-specific libraries to create serverless apps: Claudia.JS for Nodejs, Zappa for Python, Sparta for Golang, Bref for PHP, etc.
• Apex provides a simple and opinionated tool to manage your Lambda functions. It can be combined with infrastructure provisioning tools such as Terraform to manage your entire app.

Open source serverless tools from AWS

Alongside the community, AWS has built and open sourced several tools for serverless developers:

• AWS SAM is a language-agnostic framework for creating, testing, deploying, and managing serverless apps. I was one of the engineers who built SAM, so I will go into SAM’s journey in more detail below.
• Chalice: A Python microservices framework to run Flask-like apps on Lambda.
• AWS Amplify: Build web and mobile apps with a serverless backend.
• Serverless Express: Run Express.js-based apps on Lambda.
• Serverless Java Container: Run Java Springs, Jersey, and Spark apps on Lambda.

Each of these projects has its own vibrant community of hundreds of developers working alongside AWS.

How did all this happen? Here are the highlights of how I created a community around SAM.

SAM’s open source journey

October, 2016: open specification

AWS SAM started its open source life as a specification on GitHub. The specification prescribes syntax for defining a serverless application as a YAML configuration file. Customers can write the configuration file (called a SAM template) and use the aws cloudformation deploy CLI command to deploy the application to the AWS cloud. We opened the specification and invited the community to work with us to build a model for defining serverless applications.

The community immediately began contributing everything from feature ideas and documentation improvements to bug reports. Within a couple of months, someone filed a GitHub Issue asking us to open source the implementation.

April, 2018: open source implementation

Fast-forward to 2018: we ended up open sourcing the SAM implementation. This was a proud moment for me personally, because it was the first time I had open sourced a business-critical component.

Before taking this decision, we had to answer a key question: SAM is a global service deployed to 15+ AWS regions and deeply integrated with other services. Is open sourcing even possible?

• What if someone steals our secret sauce?
• What if someone added malicious code?
• What if someone contributes poor-quality code?
• Can we run an open source product as a service?
• Can we keep some features secret?
• Do we have the right team to manage open source?

We wrote an internal document (one of Amazon’s famous “PRFAQs”) to answer these questions. The goal was to convince stakeholders that this was a good idea. But the document ended up mostly being about “how to run an open source project and build a successful community.” We knew that if we built a vibrant community we could combine the agility of open source development with the quality assurance practices of AWS, to produce high-quality software quickly.

Build communities first, then software

Open communication

To build a healthy community, we needed open communication among everyone, including engineers from AWS. We wanted to make everyone feel like part of the same team. So we created a Slack channel (#samdev) which now now has more than 1,000 members including contributors, customers, engineers, etc.

Open development

Within this one team, we wanted all software developers (again, including developers from AWS) to use the same tools, practices, and processes. We created detailed development guidelines, discussed designs publicly, reviewed and critiqued each others’ code changes publicly through pull requests, and communicated when releases happened.

When every aspect of software development is open, it builds trust, increases code quality, and makes the community feel more welcoming.

Open priorities

Everything around engineering was open. But the product priorities and decisions were still happening behind closed doors. We therefore started the Request for Comments process (example RFC) where the community would provide feedback and could up-vote Issues. We used labels to set community priorities for issues and PRs to ensure that the community agrees on what to work on next. This also provided customers and contributors clear visibility into how we think about a problem/task.

Deployment automation

This is my favorite part about open source projects at AWS. While the community contributes code and thought leadership, we take care of deployments and releases on behalf of the community. At AWS, we have deep knowledge of deploying and managing web services at scale. Internally, we deploy SAM to all AWS regions. We ensure safe deployments by applying best practices such as auto-rollback, alarms, canaries, etc. We perform security reviews on every code change, ensure backwards compatibility, and deal with the deployment failures and outages.

When the deployment is done, we communicate back to the community using GitHub Releases so customers can start using it.

Continuous improvement

Every week, we have internal meetings to track the health of the community, software, and service. We review a set of metrics, discuss top issues, and identify ways to improve the health of both the community and the software. Here is a peek into some of the metrics we review:

Information from each week’s operational review is visible to the broader AWS Serverless organization. We often trade notes with other popular open source projects (like AWS CLI and the SDKs) to ensure that we don’t repeat each others’ mistakes, and that we continue to raise the quality bar.

Incredible community

I am incredibly proud of the community we have built around the AWS SAM product. 100+ developers have contributed valuable features and continue to support this product as we grow. Join the #samdev Slack channel and check out the SAM Contributing Guide to become a part of this growing community!

Connect with me at OSCON 2019

I will be giving a talk on building open source communities at OSCON 2019. Come say hi, if you are around.