AWS Compute Blog

Category: Analytics

Kinesis producers and consumers

Understanding data streaming concepts for serverless applications

In this post, I introduce some of the core streaming concepts for serverless applications. I explain some of the benefits of streaming architectures and how Kinesis works with producers and consumers. I compare different ways to ingest data, how streams are composed of shards, and how partition keys determine which shard is used. Finally, I explain the payload formats at the different stages of a streaming workload, how message ordering works with shards, and why idempotency is important to handle.

Read More
Monitoring the Kinesis stream

Monitoring and troubleshooting serverless data analytics applications

In this post, I show how the existing settings in the Alleycat application are not sufficient for handling the expected amount of traffic. I walk through the metrics visualizations for Kinesis Data Streams, Lambda, and DynamoDB to find which quotas should be increased.

Read More
Solution architecture

Building leaderboard functionality with serverless data analytics

In this post, I explain the all-time leaderboard logic in the Alleycat application. This is an asynchronous, eventually consistent process that checks batching of incoming records for new personal records. This uses Kinesis Data Firehose to provide a zero-administration way to deliver and process large batches of records continuously.

Read More
Solution architecture

Building serverless applications with streaming data: Part 3

In this post, I explain the all-time leaderboard logic in the Alleycat application. This is an asynchronous, eventually consistent process that checks batching of incoming records for new personal records. This uses Kinesis Data Firehose to provide a zero-administration way to deliver and process large batches of records continuously.

Read More
Start race UI

Building serverless applications with streaming data: Part 2

This post focuses on ingesting data into Kinesis Data Streams. I explain the two approaches used by the Alleycat frontend and the simulator application and highlight other approaches that you can use. I show how messages are routed to shards using partition keys. Finally, I explore additional factors to consider when ingesting data, to improve efficiency and reduce cost.

Read More
Setting up NAT Gateway

Setting up AWS Lambda with an Apache Kafka cluster within a VPC

Using resources such as NAT Gateways and VPC endpoints with PrivateLink, you can ensure that your data remains secure while also granting access to resources such as Lambda to help you create a Kafka consumer application. This post provides some tips to help you set up a Lambda function using Kafka as a trigger. It also explains various options available to send data securely.

Read More

Building serverless applications with streaming data: Part 1

In this post, I introduce the Alleycat racing application for processing streaming data. I explain the virtual racing logic and provide an overview of the application architecture. I summarize the deployment process for the different parts of the solution and show how to test the frontend once the deployment is complete.

Read More

Analyzing Freshdesk data using Amazon EventBridge and Amazon Athena

This post is written by Shashi Shankar, Application Architect, Shared Delivery Teams Freshdesk is an omnichannel customer service platform by Freshworks. It provides automation services to help speed up customer support processes. The Freshworks connector to Amazon EventBridge allows real time streaming of Freshdesk events with minimal configuration and setup. This integration provides real-time insights […]

Read More