Moving to event-driven architectures with serverless event aggregators

Event-driven architectures are gaining attention because they help organizations achieve benefits through the decoupling of services, increasing scalability, adding flexibility, and increasing developers’ agility. As companies grow over time, they need to find ways to integrate (couple) multiple processes and applications (which are usually owned by different teams.)

Coupled applications work in certain scenarios where synchronous communication is required (e.g. API calls.) These may serve critical business functionality, but can be cumbersome over time when cross-application dependencies grow beyond the original context. For instance, with the growing complexity, changes to one application may require changes to the tightly coupled applications as well. It is these tightly coupled applications that typically start to introduce bottlenecks and reduce agility when looking at overall modernization efforts. Simple changes become complex, and in some cases block the modernization efforts fully or delay them extensively. In these cases, it may be worthwhile to review the benefits of event driven architectures.

In event driven architectures, each application raises an event whenever anything changes. It is then the dependent applications accountability to listen and decide what actions to take. In this approach, you are able to decouple your events sources and targets. Reducing dependency across applications also enables teams to act more independently and gives them required flexibility for achieving further modernization efforts.

Today, you can use Amazon EventBridge to choreograph your events. In this choreography, Amazon EventBridge rules match incoming events from event sources. It routes them to targets based on defined triggering patterns. However, if you have complex processes where multiple conditions are analyzed before triggering an event, you will require additional logic to orchestrate events at scale. This is what we call event aggregators.

In this blog post, we will cover how you can transform your existing environment with event driven architectures. This will include how you can build “serverless event aggregators” when building complex triggering mechanisms.

Scenario Overview

In order to explain the value of the approach using serverless event aggregators, we will use a simplified version of a common ecommerce scenario. This example scenario is for an online business where your customers are using your website to buy your products. In this case, you have multiple teams (and backend systems/applications) looking after different parts of the process. They all need to complete tasks based on the information received.

When your customers create an order:

You check and validate their account for sufficient credits to cover the cost of the product, AND
You check for sufficient inventory for the requested product.

And, if BOTH processes are successful:

You send a message to your customer that “Order Submitted” or
If there is any issue in the process (invalid-order), you take further actions.

Figure 1. Contextual diagram showing the order flow described earlier with two paths followed after order creation.

In the past, with similar scenarios we saw that two primary patterns were followed:

Polling mechanism: A polling mechanism is built for each subsequent application to continuously check if the previous processes are complete.
Fixed scheduling: A fixed schedule is set for each process, this then facilitates the sequence is followed.

Even though these integration approaches worked for some use-cases, they inadvertently introduced tight-coupling between applications. This resulted in static architectures that became bottlenecks for modernization efforts. This also had a direct customer impact, where end-to-end synchronous architectures (like API operations) required all steps to complete before a response was shared.

Building Serverless Event Aggregators

In order to remediate those challenges described in the preceding scenario, we can implement asynchronous checks with an event aggregator. This decision also decouples applications, potentially shielding the customer experience while all steps are processed.

This serverless event aggregator solution can be built with DynamoDB, AWS Lambda, and Amazon EventBridge. In this solution, we use DynamoDB streams functionality and AWS Lambda to aggregate events, and used event based routing with Amazon EventBridge. This solution has four main steps:

Figure 2. Architecture diagram showing the event aggregator solution consists of four steps starting from 1. Update DynamoDB from event sources, 2. Enable DyanamoDB stream to trigger event, 3. Aggregate your events with event aggregator and 4. Filter and route your events with Amazon EventBridge.

Step 1: Update DynamoDB from your event sources

Everything starts with event sources (in our example, “account check” and “in-stock” applications). Once your source applications are run, they update DynamoDB table via UpdateItem API.

The following screenshot shows the “Events” table created in DynamoDB and updated events from the source applications (“account check” and “in-stock”) with their OrderID.

Figure 3. Amazon DynamoDB table showing event records generated by event sources like “account check” and “in-stock” events.

By using DynamoDB as the event store, you can add more event sources or update them as your business needs change. This will give you further flexibility and scalability to include more teams and processes into your event driven architecture.

Step 2: Using the DynamoDB stream to trigger an event

Once your applications (event sources) update the DynamoDB table, your DynamoDB table will trigger a Lambda function, using the Lambda and DynamoDB integrated trigger mechanism.

You can implement this integration using DynamoDB streams. Once you enabled DynamoDB stream in your DynamoDB table, it will be seen as a “trigger” in your AWS Lambda function. You can follow this tutorial to create the DynamoDB stream and AWS Lambda trigger. The following image shows the Lambda console once the setup is completed.

Figure 4. Lambda console showing DynamoDB events are enabled as “triggers.”

Step 3: Aggregate your events

Now you will build your event aggregator using a Lambda function in order to aggregate all your events. This will include all your aggregation business logic. In our scenario, this function receives the OrderID that was created as the input. It then performs a scan of the DynamoDB table for all the events related to the same order. It then generates an EventBridge custom event with a list of all the events for the order using a PutEvents API.

Following is an example of this event aggregator Lambda function written in Python:

import json
import boto3
from boto3.dynamodb.conditions import Attr

dynamodb = boto3.resource('dynamodb')
client = boto3.client('events')

def lambda_handler(event, context):
  print(json.dumps(event))
  order = int(event['Records'][0]['dynamodb']['NewImage']['OrderID']['N'])
  print(order)
  table = dynamodb.Table('Events')
  table_scan = table.scan(FilterExpression=Attr('OrderID').eq(order))
  event_status = {}
  for item in table_scan['Items']:
    event_status[item['EventName']] = order
  print (json.dumps(event_status))
  response = client.put_events(
  Entries=[
  {
  'Source': 'my-event',
  'DetailType': 'events list from dynamodb',
  'Detail': json.dumps(event_status),
  'EventBusName': 'default',
  },
  ])

Step 4: Filter and route your events with Amazon EventBridge

Once your aggregator function is run, it will create an event that includes an aggregated view of the multiple events status. For-example, in the generated event in the following table, you can see both “in-stock” and “account check” events have been updated with the OrderID: 7 value. This means that both checks are completed for Order 7 and we can now proceed with further events.

{
   "version":"0",
   "id":"711ef30b-582a-cda5-c132-bceb46bc5c6a",
   "detail-type":"events list from dynamodb",
   "source":"my-event",
   "account":"123456789098",
   "time":"2022-10-13T13:03:02Z",
   "region":"eu-west-1",
   "resources":[
      
   ],
   "detail":{
      "In Stock":7,
      "Account Check":7
   }
}

Once EventBridge receives this event, it will be processed in EventBridge rules to capture the specific condition of both events (“account check” and “in-stock”) exists. And if both pass the checks, it will trigger the “InvoiceProcessing” Lambda function.

Following, you can see the configuration of the EventBridge rule defined for our scenario.

{
  "source": ["my-event"],
  "detail-type": ["events list from dynamodb"],
  "detail": {
    "Account Check": [{
      "exists": true
    }],
    "In Stock": [{
      "exists": true
    }]
  }
}

EventBridge rules use event patterns to select and match incoming events and trigger their specified targets for processing.

Amazon EventBridge supports several comparison operators. You can refer to Amazon EventBridge Event Patterns for the full supported list.

In our example pattern, we used “And” operator to select both “account check” and “in-stock” events “exists: true”. Once our pattern conditions are met, EventBridge will trigger the target “InvoiceProcessing”.

The following screenshot shows how the “InvoiceProcessing” Lambda function is configured to be triggered with the matched event.

Figure 5. Amazon EventBridge console showing the defined target event “InvoiceProcessing” Lambda function is configured to be triggered in EventBridge rules.

With this step, we now concluded our scenario. We collected events from multiple Event Sources, and aggregated them with discretionary logic through the “Event Aggregator.” After all defined conditions were met, we orchestrated them through Amazon EventBridge to trigger our defined “InvoiceProcessing” target application. Due to flexible nature of this solution, you can add/remove and change the components without the need of re-architecting.

Conclusion

With serverless event aggregators, you can achieve scalability, flexibility and control by decoupling your applications that require complex triggering mechanisms. As in any modernization project, your journey starts from understanding your business needs and your current capabilities. Bringing your teams together and figuring out how your applications interact will be a valuable step to moving to event driven architectures.

In this blog, we explored how you can approach transforming your applications with Event Driven architectures using serverless event aggregators. We used a simplistic scenario for an ecommerce application that in the past was tightly coupled to backend processing systems. The transformation broke apart the tight coupling of dependent processes, and allowed for independent and asynchronous processing of events to complete. This allows for tight coupling bottlenecks to be removed, and modernization efforts to scale more effectively.

With the DynamoDB integration, you can have as many as event sources as you need with your growing solution and having the full control of your Aggregator Lambda function. These two integrations will give you flexibility to address changing business demands. Once your events are aggregated, AWS EventBridge can help you to integrate your solution with over 35 targets. These targets include many SaaS applications like Datadog, Onelogin, and PagerDuty. With ready-made integrations, you don’t need to manage any integration set-up. This will boost your developer agility by reducing the need to coordinate across different teams and SaaS applications.

To learn more about event Driven Architectures, try “Building event-driven architectures on AWS workshop” ,check Serverless Land for latest blogs, videos, and training materials and check AWS Serverless Application repository to discover the pre-built applications for your modernization journey.

About the author:

AWS Cloud Operations & Migrations Blog