How Koch Agriculture and Energy Solutions (KAES) worked with AWS to modernize their application architecture using Event Driven Design

Introduction

KAES began its cloud transformation journey in 2017, when their on-premises environment led to point-to-point design of Middleware technologies.  Their journey to AWS was focused on building an Event Driven Architecture using AWS following the serverless best practices. In this blog we look at KAES’s AWS cloud journey and the lesson’s learned along the way.

About KAES

Koch Ag & Energy Solutions, is a global provider of value-added solutions for the agriculture, turf and ornamental, energy, and chemical markets. KAES is a wholly owned subsidiary of Koch Industries, Inc.

Initial Event Architecture Design: Event Bus 0.0

In 2017, KAES’s first approach to Event Driven Architecture was to use serverless services such as Amazon API Gateway, AWS Lambda, and Amazon DynamoDB to publish events to entity-based microservices. After the microservices received new data, their initial design allowed for publication to Amazon Simple Queue Service (Amazon SQS) and Amazon Simple Notification Service (Amazon SNS) for other applications to receive or write data to Amazon Simple Storage Service (Amazon S3). They could then load that data into Amazon Redshift for analytics applications. The above Event Driven Architecture introduced unacceptable delay for end users who were trying to load data for analytics, specifically when there was a large update to a source system.

It also gave KAES insight into DevOps, particularly into the way business events were handled.  For example, one event may use Amazon SNS/SQS and another may just push the data to Amazon S3.

KAES started looking for a better approach. The goals were:

• Require all messages to go through the same bus
• Make publishing new messages trivial
• Maintain a consistent message format
• Increase agility
• Keep management minimal

Self-Managed Event Bus Design – Event Bus 1.0

KAES’s first self-managed Event Bus largely solved the issues with the initial architecture.  Prior to using self-managed event bus, it could take hours or even days of developer hours to setup and handle a new business event. The new Self-managed Event Bus took this down to minutes. This allowed to quickly move business transactions between systems, and reduced point-to-point integrations, tight coupling, and risk.  By moving to an Event Driven Architecture, KAES saved anywhere from 4 to 40 hours per message transformation.

Figure 1: Self-managed Event Bus

1. Source system sends message to Amazon API Gateway. For large messages, a second API is used to post the object to Amazon S3.
2. The API passes the message to AWS Lambda function.
3. The AWS Lambda function posts a message to SQS queue.
4. A routing AWS Lambda function picks up the message and determines the routing from a consumer Amazon DynamoDB table.
5. Consumer registrations are handled through the consumer service (Amazon API Gateway, AWS Lambda, and Amazon DynamoDB).
6. Some simple message transformations are handled through an AWS Lambda function.
7. Messages are written to the consumer queue setup through the consumer registration process. Each consumer has a lambda function handler that writes the data to Amazon S3, Amazon Kinesis Data Firehose, Amazon API Gateway, or to an Amazon SNS topic.
8. Each queue has a dead letter queue, which writes data to Amazon DynamoDB table.
9. Messages in the dead letter queue can be resubmitted with an Amazon API call.

Enhanced Event Bus Design using Amazon EventBridge – Event Bus 2.0

While KAES saw significant performance benefits to their custom approach, maintaining code and continuous improvement was still a challenge. In simplifying their architecture, they believed Amazon EventBridge would reduce code maintenance and enable them to benefit from future features.

Figure 2: Introducing Amazon EventBridge

1. Instead of having the proxy AWS Lambda post to Amazon SQS, messages are posted to the Amazon EventBridge.
2. Instead of storing consumer information in Amazon DynamoDB, KAES used Amazon EventBridge to route messages.

Introducing Event Bus simplified the architecture and code maintenance overhead; however, the bus was hosted on a single AWS account and the recipient applications were distributed across multiple AWS account. This was causing more work for downstream applications. By using the Cross Account messaging feature of Amazon EventBridge, KAES improved the robustness of the architecture and security.

Cross Account Messaging using Amazon EventBridge

1. Setup Amazon EventBridge in a new core services account.
2. Setup separate Amazon EventBridges in each consumer account that needs to receive events.
3. In the core account, setup a rule for each downstream account to receive events.

Results

The solution described in this post was low maintenance, scalable, reliable, and significantly reduced the effort required to distribute events to distributed business environments. This architecture positioned AWS to assist and mentor other Koch companies to fail fast and to set-up Event Driven Architectures in their environments. The successes and lessons learned with this implementation enables AWS architects to accelerate and build scalable solutions for their customers and organizations. Amazon EventBridge simplifies the process of building event-driven architectures at scale to process events generated from your applications, integrated Software-as-a-Service (SaaS) applications, and AWS services.