AWS Compute Blog
ICYMI: Serverless Q4 2020
Welcome to the 12th edition of the AWS Serverless ICYMI (in case you missed it) quarterly recap. Every quarter, we share all of the most recent product launches, feature enhancements, blog posts, webinars, Twitch live streams, and other interesting things that you might have missed!
In case you missed our last ICYMI, check out what happened last quarter here.
re:Invent was entirely virtual in 2020 and free to all attendees. The conference had a record number of registrants and featured over 700 sessions. The serverless developer advocacy team presented a number of talks to help developers build their skills. These are now available on-demand:
- Decoupling serverless workloads with Amazon EventBridge – James Beswick
- AWS Lambda – Part 1: Optimizing your serverless applications – Chris Munns
- Best practices for securing your serverless applications – Rob Sutter
- API Gateway – Beyond the proxy – Eric Johnson
- The serverless LAMP stack – Ben Smith
- Becoming proficient with serverless application observability – Julian Wood
There were three major Lambda announcements at re:Invent. Lambda duration billing changed granularity from 100 ms to 1 ms, which is shown in the December billing statement. All functions benefit from this change automatically, and it’s especially beneficial for sub-100ms Lambda functions.
Lambda has also increased the maximum memory available to 10 GB. Since memory also controls CPU allocation in Lambda, this means that functions now have up to 6 vCPU cores available for processing. Finally, Lambda now supports container images as a packaging format, enabling teams to use familiar container tooling, such as Docker CLI. Container images are stored in Amazon ECR.
There were three feature releases that make it easier for developers working on data processing workloads. Lambda now supports self-hosted Kafka as an event source, allowing you to source events from on-premises or instance-based Kafka clusters. You can also process streaming analytics with tumbling windows and use custom checkpoints for processing batches with failed messages.
We launched Lambda Extensions in preview, enabling you to more easily integrate monitoring, security, and governance tools into Lambda functions. You can also build your own extensions that run code during Lambda lifecycle events. See this example extensions repo for starting development.
You can now send logs from Lambda functions to custom destinations by using Lambda Extensions and the new Lambda Logs API. Previously, you could only forward logs after they were written to Amazon CloudWatch Logs. Now, logging tools can receive log streams directly from the Lambda execution environment. This makes it easier to use your preferred tools for log management and analysis, including Datadog, Lumigo, New Relic, Coralogix, Honeycomb, or Sumo Logic.
Lambda launched support for Amazon MQ as an event source. Amazon MQ is a managed broker service for Apache ActiveMQ that simplifies deploying and scaling queues. The event source operates in a similar way to using Amazon SQS or Amazon Kinesis. In all cases, the Lambda service manages an internal poller to invoke the target Lambda function.
Lambda announced support for AWS PrivateLink. This allows you to invoke Lambda functions from a VPC without traversing the public internet. It provides private connectivity between your VPCs and AWS services. By using VPC endpoints to access the Lambda API from your VPC, this can replace the need for an Internet Gateway or NAT Gateway.
For developers building machine learning inferencing, media processing, high performance computing (HPC), scientific simulations, and financial modeling in Lambda, you can now use AVX2 support to help reduce duration and lower cost. In this blog post’s example, enabling AVX2 for an image-processing function increased performance by 32-43%.
Lambda now supports batch windows of up to 5 minutes when using SQS as an event source. This is useful for workloads that are not time-sensitive, allowing developers to reduce the number of Lambda invocations from queues. Additionally, the batch size has been increased from 10 to 10,000. This is now the same batch size as Kinesis as an event source, helping Lambda-based applications process more data per invocation.
Code signing is now available for Lambda, using AWS Signer. This allows account administrators to ensure that Lambda functions only accept signed code for deployment. You can learn more about using this new feature in the developer documentation.
AWS Step Functions
Synchronous Express Workflows have been launched for AWS Step Functions, providing a new way to run high-throughput Express Workflows. This feature allows developers to receive workflow responses without needing to poll services or build custom solutions. This is useful for high-volume microservice orchestration and fast compute tasks communicating via HTTPS.
The Step Functions service recently added support for other AWS services in workflows. You can now integrate API Gateway REST and HTTP APIs. This enables you to call API Gateway directly from a state machine as an asynchronous service integration.
Step Functions now also supports Amazon EKS service integration. This allows you to build workflows with steps that synchronously launch tasks in EKS and wait for a response. The service also announced support for Amazon Athena, so workflows can now query data in your S3 data lakes.
Amazon API Gateway
API Gateway now supports mutual TLS authentication, which is commonly used for business-to-business applications and standards such as Open Banking. This is provided at no additional cost. You can now also disable the default REST API endpoint when deploying APIs using custom domain names.
HTTP APIs now supports service integrations with Step Functions Synchronous Express Workflows. This is a result of the service team’s work to add the most popular features of REST APIs to HTTP APIs.
X-Ray now integrates with Amazon S3 to trace upstream requests. If a Lambda function uses the X-Ray SDK, S3 sends tracing headers to downstream event subscribers. This allows you to use the X-Ray service map to view connections between S3 and other services used to process an application request.
X-Ray announced support for end-to-end tracing in Step Functions to make it easier to trace requests across multiple AWS services. It also launched X-Ray Insights in preview, which generates actionable insights based on anomalies detected in an application. For Java developers, the services released an auto-instrumentation agent, for collecting instrumentation without modifying existing code.
Additionally, the AWS Distro for Open Telemetry is now in preview. OpenTelemetry is a collaborative effort by tracing solution providers to create common approaches to instrumentation.
You can now use event replay to archive and replay events with Amazon EventBridge. After configuring an archive, EventBridge automatically stores all events or filtered events, based upon event pattern matching logic. Event replay can help with testing new features or changes in your code, or hydrating development or test environments.
EventBridge also launched resource policies that simplify managing access to events across multiple AWS accounts. Resource policies provide a powerful mechanism for modeling event buses across multiple account and providing fine-grained access control to EventBridge API actions.
EventBridge announced support for Server-Side Encryption (SSE). Events are encrypted using AES-256 at no additional cost for customers. EventBridge also increased PutEvent quotas to 10,000 transactions per second in US East (N. Virginia), US West (Oregon), and Europe (Ireland). This helps support workloads with high throughput.
The AWS Serverless Application Model (AWS SAM) is an AWS CloudFormation extension that makes it easier to build, manage, and maintain serverless applications. The latest versions include support for cached and parallel builds, together with container image support for Lambda functions.
You can use AWS SAM in the new AWS CloudShell, which provides a browser-based shell in the AWS Management Console. This can help run a subset of AWS SAM CLI commands as an alternative to using a dedicated instance or AWS Cloud9 terminal.
Amazon SNS announced support for First-In-First-Out (FIFO) topics. These are used with SQS FIFO queues for applications that require strict message ordering with exactly once processing and message deduplication.
Developers can now use PartiQL, an SQL-compatible query language, with DynamoDB tables, bringing familiar SQL syntax to NoSQL data. You can also choose to use Kinesis Data Streams to capture changes to tables.
For customers using DynamoDB global tables, you can now use your own encryption keys. While all data in DynamoDB is encrypted by default, this feature enables you to use customer managed keys (CMKs). DynamoDB also announced the ability to export table data to data lakes in Amazon S3. This enables you to use services like Amazon Athena and AWS Lake Formation to analyze DynamoDB data with no custom code required.
AWS Amplify and AWS AppSync
You can now use existing Amazon Cognito user pools and identity pools for Amplify projects, making it easier to build new applications for an existing user base. With the new AWS Amplify Admin UI, you can configure application backends without using the AWS Management Console.
AWS AppSync enabled AWS WAF integration, making it easier to protect GraphQL APIs against common web exploits. You can also implement rate-based rules to help slow down brute force attacks. Using AWS Managed Rules for AWS WAF provides a faster way to configure application protection without creating the rules directly.
- Oct 5 – Building resilient serverless patterns by combining messaging services
- Oct 6 – The serverless LAMP stack part 6: From MVC to serverless microservices
- Oct 8 – Building Extensions for AWS Lambda – In preview
- Oct 8 – Introducing AWS Lambda Extensions – In preview
- Oct 8 – Improved failure recovery for Amazon EventBridge
- Oct 12 – ICYMI: Serverless Q3 2020
- Oct 19 – Optimizing the cost of serverless web applications
- Oct 22 – Building event-driven architectures with Amazon SNS FIFO
- Oct 28 – Choosing between AWS Lambda data storage options in web apps
- Nov 3 – Building Serverless Land: Part 1 – Automating content aggregation
- Nov 5 – Using Amazon MQ as an event source for AWS Lambda
- Nov 9 – Archiving and replaying events with Amazon EventBridge
- Nov 10 – Building Serverless Land: Part 2 – An auto-building static site
- Nov 12 – Using AWS Lambda extensions to send logs to custom destinations
- Nov 18 – Performing canary deployments for service integrations with Amazon API Gateway
- Nov 19 – Simplifying cross-account access with Amazon EventBridge resource policies
- Nov 24 – New Synchronous Express Workflows for AWS Step Functions
- Nov 24 – Creating faster AWS Lambda functions with AVX2
- Nov 24 – Introducing Amazon API Gateway service integration for AWS Step Functions
- Nov 27 – ICYMI: Serverless pre:Invent 2020
- Dec 1 – Building more cost-effective Lambda functions with 1 ms billing
- Dec 3 – Working with Lambda layers and extensions in container images
- Dec 7 – Packaging AWS Lambda functions as container images
- Dec 15 – Optimizing batch processing with custom checkpoints in AWS Lambda
- Dec 15 – Using AWS Lambda for streaming analytics
- Dec 15 – Using self-hosted Apache Kafka as an event source for AWS Lambda
- Dec 16 – Using container image support for AWS Lambda with AWS SAM
Tech Talks & Events
We hold AWS Online Tech Talks covering serverless topics throughout the year. These are listed in the Serverless section of the AWS Online Tech Talks page. We also regularly deliver talks at conferences and events around the world, speak on podcasts, and record videos you can find to learn in bite-sized chunks.
Here are some from Q4:
- Testing and Monitoring APIs on AWS with Chris Munns.
- Building CI/CD Workflows for Serverless Applications with Rob Sutter.
- AWS SAM for local development
- Lambda Extensions: easily integrate Lambda with your favorite observability and security tools
- Lambda Layers: Using layers to manage code dependencies in serverless applications
- AWS SAM custom builds: How to use Makefiles to customize builds in SAM
- Learning AWS SAM: sam build
- AWS SAM build: how it provides artifacts for deployment
- ServerlessMN: managing dependencies for AWS Lambda functions
- AWS SAM utilities to help simplify local development
- Using OpenAPI in AWS SAM to manage API Gateway
- Serverless @ re:Invent #1
- Serverless @ re:Invent #2
- Serverless @ re:Invent #3
- Serverless @ re:Invent #4
- Serverless @ re:Invent #5
- Serverless @ re:Invent #6
There are also other helpful videos covering Serverless available on the Serverless Land YouTube channel.
The Serverless Land website
To help developers find serverless learning resources, we have curated a list of serverless blogs, videos, events, and training programs at a new site, Serverless Land. This is regularly updated with new information – you can subscribe to the RSS feed for automatic updates or follow the LinkedIn page.
Still looking for more?
The Serverless landing page has lots of information. The Lambda resources page contains case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials.
You can also follow all of us on Twitter to see latest news, follow conversations, and interact with the team.
- Chris Munns: @chrismunns
- Eric Johnson: @edjgeek
- James Beswick: @jbesw
- Ben Smith: @benjamin_l_s
- Rob Sutter: @rts_rob
- Julian Wood: @julian_wood