Serverless Data Processing

Find resources to build and run data processing applications without thinking about servers

Learn how to build a serverless data processing application

In this tutorial, you will learn the basics of stream data processing using AWS Lambda and Amazon Kinesis. You will use Amazon Kinesis to create a data stream and perform AWS Lambda operations such as creating a Lambda function to add records to the stream.
 
You will learn how to set up a Lambda function using the AWS Command Line Interface and how to test it in the AWS console. In addition, you will learn how to easily deploy your data processing application with the AWS Serverless Application Model (SAM) and AWS CloudFormation.
Follow the tutorial

Dive Deeper into Serverless Stream Processing Patterns

In this whitepaper, called Serverless Stream Architectures and Best Practices, we will explore three Internet of Things (IoT) stream processing patterns using a serverless approach. For each pattern, we’ll describe how it applies to a real-world IoT use-case, the best practices and considerations for implementation, and cost estimates. You will get familiarity with services like  AWS Lambda and Amazon Kinesis and AWS IoT Core. Each pattern also includes a template which enables you to easily and quickly deploy these patterns in your AWS accounts.
 
You'll also hear from Thomson Reuters, iRobot the maker of Roomba, and Nextdoor on how they have benefited from using a serverless approach for data processing.
Go to the whitepaper

What is serverless computing?

Serverless computing allows you to build and run applications and services without thinking about servers. Serverless applications don't require you to provision, scale, and manage any servers. You can build them for nearly any type of application or backend service, and everything required to run and scale your application with high availability is handled for you.

Building serverless applications means that you can focus on your core product instead of worrying about managing and operating servers or runtimes, either in the cloud or on-premises. This reduced overhead lets you reclaim time and energy that you can spent on developing great products which scale and that are reliable.

Find additional resources

GETTING STARTED GUIDE

A good way to get started with AWS Lambda is to work through the Getting Started Guide, part of our technical documentation. Within a few minutes, you will be able to deploy and use an AWS Lambda function. The documentation provides a conceptual overview of AWS Lambda, includes detailed instructions for using the various features, and provides a complete API reference for developers.

BATCH PROCESSING GUIDE

In this tutorial, you will learn how to build a simple image processing application and develop a Lambda function to automatically convert an image into a thumbnail. You will learn how to use AWS Lambda in conjunction with Amazon Simple Storage Service (S3), the AWS Serverless Application Model, and AWS CloudFormation.

REFERENCE ARCHITECTURE

This sample application demonstrates a Markdown conversion application where Lambda is used to convert Markdown files to HTML and plain text. It uses an even-driven, parallel data processing architecture which is ideal for workloads that need more than one data derivative of an object.

Customer case studies

THOMSON REUTERS

Thomson Reuters uses a serverless architecture to process up to 4,000 events per second for its usage analytics service. The service reliably handles spikes of twice its normal traffic and has high durability. The company deployed the service into production in only five months using AWS.

SQUARE ENIX

Square Enix uses AWS Lambda to run image processing for its Massively Multiplayer Online Role-Playing Game (MMORPG). With AWS Lambda, Square Enix was able to reliably handle spikes of up to 30 times normal traffic. Lambda also lowered the time required for image processing from several hours to just over 10 seconds, and reduced infrastructure and operational costs.

NEXTDOOR

In this video, Nextdoor, a social networking service for neighbourhoods, will talk about how they replaced their home-grown data pipeline based on a topology of Flume nodes with a completely serverless architecture based on Kinesis and Lambda. By making these changes, they improved both the reliability of their data and the delivery times of billions of records of data to their Amazon S3–based data lake and Amazon Redshift cluster.

Learn more about AWS Lambda

Visit the product page
Ready to get started?
Sign up
Have more questions?
Contact us