AWS Open Source Blog

The CDK Patterns open source journey

This post was contributed by Matthew Coulter, Technical Architect at Liberty Mutual.

In the summer of 2019, I successfully applied for a promotion to the position of account architect at Liberty IT Solutions, a part of Liberty Mutual Group. This changed everything, as I went from being the systems architect supporting one area to an architect responsible for enabling an organization of more than 1,000 engineers. My colleague, Mark McCann, and I were brought in to create an environment where our engineers felt motivated, empowered, and enabled to deliver business value rapidly, in a sustainable, well-architected, and serverless-first way.

So what had been holding them back? It turned out to be an industry problem. Read on to find out how CDK Patterns released the potential of our engineers, gained industry momentum, and caught the attention of Werner Vogels, CTO of Amazon.

Discovering a problem

I had significant experience developing serverless solutions using AWS CloudFormation templates. CloudFormation is an incredible service, but the feedback I heard from developers who moved to serverless on AWS was not always encouraging. People new to public cloud were intimidated by CloudFormation templates. It was so different that they didn’t know what “good” looked like. Experienced developers would say that the templates felt brittle and were more complicated than they appear.

Those coming from a SpringBoot background were used to running everything locally, while having extensive unit and integration test suites. They were now facing the prospect of testing CloudFormation templates changes during a deployment, which was against every development practice they had ever learned. Where were all the pre-deployment tests used to confirm that you hadn’t caused unintended consequences?

Couple this with Liberty Mutual’s public cloud posture, which places vital guardrails on our AWS accounts. These guardrails create a scenario in which a solution that may work on your personal AWS account, but it won’t deploy internally. This leads to a knowledge gap and developer frustration.

Because the goal was to make serverless first core for more than 1,000 developers, I needed to meet them where they were and create a frictionless developer experience.

Finding the AWS Cloud Development Kit (AWS CDK)

AWS CDK became generally available in July 2019, and I wondered what benefits this new technology could bring compared to the existing solutions. As an experiment, I took one of our most common scenarios—a private API gateway with a custom OAuth 2.0 authorizer Lambda function. This was more than 1,000 lines of CloudFormation templates YML, and it had some CloudFormation templates quirks. Either you had to rename parts manually to trigger a deployment, or you had to do a post-deploy CLI call.

With 14 lines of TypeScript, I was able to create a CDK construct that produced the same gateway without the CloudFormation templates deployment quirks. I was then able to layer on a set of infrastructure unit tests to provide developers a mechanism for test-driven development. Finally, to run the API gateway locally, my construct could pair easily with AWS Serverless Application Model (AWS SAM) Local. This felt like a game changer.

Discovering the real problem

I was certain that my job was done—serverless and CDK were a perfect match. Then I attended AWS re:Invent in 2019. As I listened to the other developer attendees and I tried drinking from the firehose of new features announcements, the real problem dawned on me. The issue when moving from traditional to serverless implementations was something I call “AWS Overload.” There are hundreds of serverless products, features, and configurations on AWS. How can a developer keep up with the right implementation for their current situation? The day after a new feature announcement, dozens of blog posts would appear. I would read them all and then try to implement something, only to meet an internal guardrail. This was exhausting, and this was the problem I needed to solve.

Building CDK Patterns MVP

The solution I came up with was to take the patterns that the AWS Heroes and Developer Advocates were discussing, and then implement them via open source on GitHub using AWS CDK. Developers could clone and deploy the the patterns into their accounts. I added links to all the external articles detailing the theory of the pattern in the readme so they could self-educate. I even recorded YouTube videos in which I shared my thoughts on the pattern.

To help Liberty Mutual developers, I created an internal fork tailored to our specific guardrails. That way, our developers could clone and deploy using the exact same patterns from GitHub, rather than creating something only seen internally.

Evolving from MVP to AWS Well-Architected

After a few patterns, I realized that unless I provided a mechanism for people to choose the right pattern for their situation, I wasn’t helping solve my original problem. I refocused and collaborated with Heitor Lessa (Principal Solutions Architect, Developer Acceleration at AWS) to create the Well-Architected Pattern Matcher. Now developers could use the AWS Well-Architected Framework to guide them to the right pattern.

What is the reception so far?

As of October 2020, there are 23 patterns using 25 serverless components from 17 contributors. The patterns all launch with CloudFormation, TypeScript, and Python support. The community has also been adding Java and C# support. The GitHub repo gets an average of 1,000 views a day, and it has nearly 700 stargazers. On social media, there are now more than 2,100 followers of the @cdkpatterns Twitter account.

At Liberty Mutual, we have deployed more than 3,000 serverless CDK patterns between January 2020 and today. In a recent informal poll, more than 66% of developers chose AWS CDK as their technology of choice for building serverless applications on AWS.

AWS has also embraced CDK Patterns, with Werner Vogels referring to two patterns and my work during the AWS Americas Online Summit in September 2020.

Be a part of CDK Patterns

Do you have a serverless architecture pattern that you would like to contribute? Visit our GitHub for more details, or open an issue on the repo stating the pattern you want to add.

Additional resources

Matt Coulter

Matt Coulter

Matt is an AWS DevTools Hero and a Technical Architect for Liberty IT in Belfast, Northern Ireland. He is focused on creating the right environment for empowered teams to rapidly deliver business value in a well-architected, sustainable, and serverless-first way. To create this environment he has been building the open source project CDK Patterns then sharing the content through multiple channels. To help accelerate the community he also founded and organised the first ever CDK Day.

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

Ricardo Sueiras

Ricardo Sueiras

Cloud Evangelist at AWS. Enjoy most things where technology, innovation and culture collide into sometimes brilliant outcomes. Passionate about diversity and education and helping to inspire the next generation of builders and inventors with Open Source.