Keeping the Security and Scalability of Serverless Apps Problem-Free with AWS Secrets Manager

By Chase Douglas, CTO at Stackery

At Stackery, an AWS Partner Network (APN) Advanced Technology Partner, we help teams build and manage serverless infrastructure faster than ever so they can make production-grade serverless applications following Amazon Web Services (AWS) best practices.

In this post, I will focus on an important category of best practices: secrets management.

A modern cloud application built from multiple AWS resources is filled with confidential data multiplied by dev, test, and production environments. These applications require many credentials that need to be separated from the code and managed.

AWS Secrets Manager helps you protect secrets needed to access your applications, services, and IT resources. The service enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle.

If you’re researching how to improve secrets management for your customers, it’s critical to curate the advice you find. Our engineers at Stackery have spent lots of time doing just that.

Here are several benefits we’ve found for using AWS Secrets Manager to keep security and scalability problem-free down the line.

Fetch Secrets from AWS Secrets Manager at Runtime

Using environment variables to pass environment configuration information to your serverless functions is a common best practice for separating config from your source code. However, using environment variables to pass secrets (passwords, API keys) may result in unexpected issues.

For example, if your app framework prints environment variables for debugging or error reporting, it will also print your secrets accidentally. Similarly, environment variables are passed down to child processes and can be used in unintended ways.

At Stackery, we never put secrets in environment variables. Instead, we fetch secrets from AWS Secrets Manager at runtime and store them in local variables while they’re in use. This makes it more difficult for secrets to be logged or otherwise exfiltrated from the runtime environment.

Store Secrets in the Right Place

If you’re dealing with secrets, they should always be encrypted at rest and encrypted in transmission. Keeping secrets in source code is inadvisable, but where should you keep them?

At Stackery, we use AWS Secrets Manager to store secrets securely. The service also has fine-grained access policies, auto-scales to handle traffic spikes, and is straightforward to query at runtime.

Figure 1 – Use AWS Secrets Manager to manage the database credentials.

Scope IAM Permissions Tightly

Each function in your application should only have access to the secrets it needs to do its work. However, it’s very common for teams to run configurations (often unintentionally) where every function by default is granted access to all secrets from all environments.

These “/*” permissions mean a compromised function in a test environment can be used to fetch all production secrets from the secrets store. Instead, permission-access should be tightly scoped by environment and usage, with functions defaulting to no secrets access.

We automatically scope an IAM role per function and AWS Fargate Docker tasks, limiting AWS Secrets Manager access to the current environment the function is running in and the set of secrets required by that specific function.

Rotate Secrets Automatically on a Schedule

As much as we may try to prevent secrets leakage, you should also guard against the possibility that a secret has been leaked in the past. One mechanism for dealing with this problem is to rotate secrets frequently. A leaked secret is useless once its use has been rotated out.

It can be tricky figuring out how to rotate secrets. Thankfully, AWS Secrets Manager has built-in mechanisms for rotating secrets. Its capabilities go beyond simple single-string key rotation to complex multi-user credentials rotation for services like SQL databases.

Audit Secrets Access

Finally, be sure to audit access and modifications to secrets. If you’re following the AWS Well-Architected Framework security principles, you should already have AWS CloudTrail set up to log all activity.

Beyond simply having CloudTrail logs, though, you should set up automated checks for inappropriate usage. You can find an example that checks for accesses to secrets scheduled for deletion in the AWS Secrets Manager documentation. Automated patterns like these are what enable AWS itself to have only one security engineer on-call, monitoring security events across their entire cloud.

Managing Serverless Environment Secrets with Stackery

By keeping the above concepts in mind, our team has helped our customers establish a solid foundation of security and scalability. We’ve learned to manage serverless secrets, running production serverless applications, and working with many serverless teams and pioneers.

We’ve integrated these best practices back into Stackery, so serverless teams can easily layer secure secrets management onto their existing projects. To read more about how Stackery handles secrets check out the Environment Secrets in the Stackery Docs.

Don’t leave secrets and environment management to chance!

Next Steps

Managing secrets and other aspects of dev/test/production environments when building serverless apps can be done manually by engineers well versed in YAML, but that’s hard to manage when multiple people are working on multiple services.

Fortunately, our team at Stackery has put lots of engineering effort into environment and secrets management, along with virtual private cloud (VPC) configuration. We welcome you to experience professional tooling with our free tier.

We host a weekly livestream with our serverless engineers and occasional guest hosts, every Wednesday at 10 a.m. PDT. It’s a great way for you to see Stackery in action and ask questions about all things serverless, including AWS Secrets Manager. You can also get in touch with our serverless experts.

The content and opinions in this blog are those of the third party author and AWS is not responsible for the content or accuracy of this post.

.

Stackery – APN Partner Spotlight

Stackery is an APN Advanced Technology Partner. Its operations dashboards and command line (CLI) tools provide sophisticated runtime support to the most complex serverless systems.

Contact Stackery | Solution Overview

*Already worked with Stackery? Rate this Partner

*To review an APN Partner, you must be an AWS customer that has worked with them directly on a project.