Responsive Event-Driven Architectures on AWS for Reduced Costs and Improved Agility

By Charles Christopher, Cloud Solution Architect – DXC Technology
By Rakesh Kootheri, Cloud Solution Lead – DXC Technology
By Dhiraj Thakur, Solutions Architect – AWS

DXC Technology

Event-driven architecture is becoming more prominent in modern application development and makes building cloud applications easier, especially applications required to create, detect, consume, and react to multiple events in real time.

Common use cases for event-driven applications include the Internet of Things (IoT), fraud detection, payment processing, website monitoring, and real-time marketing.

In this post, you will learn how DXC Technology helped a customer in the energy industry collect and push events from electricity meters using event-driven architecture.

DXC Technology is an AWS Premier Tier Services Partner and Managed Service Provider (MSP) with over 4,000 AWS certified resources to help customers harness the power of innovation and drive their business transformations.

Meter-Usage Distribution Using Event-Driven Architecture

A DXC Technology customer needed to collect electricity meter usage from different market participants and send it to multiple electricity vendors. The data transfer happens asynchronously, and it must be archived for future reference.

Figure 1 – Meter usage distribution using AWS event-driven architecture.

More specifically, electricity market participants retrieve electricity usage data from smart meters via a message broker system. Messages are in JSON format and may contain payloads of large data.

Event-driven applications treat data as immutable, which makes it easier to look up values of data at previous points in time. This guarantees that multiple event consumers receive the data as it was when the event was generated. If information changes, a new data point is created.

As shown in Figure 1, the message broker system receives messages from different market participants and routes them to registered receiver electricity vendor applications or market participants’ applications based on the message header.

This scalable event-driven architecture can support a large number of events in real time and provides improved responsiveness and agility.

DXC Technology’s Solution Architecture

This section explains the solution architecture DXC Technology proposed for the customer, as shown in Figure 2.

First, the sender emits the events to the receiver, which is acknowledged by the message broker system. There is no direct communication between sender and receiver applications. That makes the system loosely coupled.

Configured rules for each market participant determine where to post the message based on database lookup and the meter ID found in the message header.

The message broker system ingests the electricity usage messages emitted from the sender through message broker APIs. The message broker system uses the following AWS products for further message acknowledgment, archiving, and transfer to respective receivers:

• Amazon API Gateway
• AWS Shield
• AWS WAF
• AWS Lambda
• Amazon EventBridge
• Amazon Simple Notification Service (SNS)
• Amazon Simple Queue Service (SQS)
• Amazon Simple Storage Services (Amazon S3)
• Amazon DynamoDB

The architecture also covers disaster recovery (DR) and exception handling for all stages of message delivery.

DXC Technology chose AWS Lambda because the serverless, scalable solution can spin up instances based on the number of transactions or messages and manage large volumes of requests.

Figure 2 – Solution architecture for meter-usage distribution.

The different message flows are explained below. Note the numbered lists correspond to the numbers in Figure 2.

Positive Message Flow

Below is the step-by-step message flow from sender to receiver:

1. Sender uses Amazon Route 53 to determine current API endpoints to post the message.

2. The message from the sender reaches Amazon API Gateway via AWS Shield and AWS WAF.

3. API Gateway uses API key-based basic authentication and invokes the Lambda function to preprocess the message. The Lambda function uses the request header for authorization, and adds a unique sequence ID maintained in Amazon DynamoDB (replicated to DR by global tables), stores the message body in production S3 (cross-region replicated to Ireland), and sends it to Amazon EventBridge.

4. Amazon EventBridge processes the message and routes it to the SNS.

5. SNS fans out the message to production and DR SQS.

6. The Lambda function picks up the message from SQS, combines it with content from S3, and attempts to deliver it to the receiver with retries. If the message delivery is successful, the message cleared from SQS and Lambda function writes success to DynamoDB, replicated to DR by global tables. In the event of message delivery failure, the message is cleared from SQS and moved to SNS dead-letter queue (DLQ). The Lambda function writes failure to DynamoDB (replicated to DR by global tables).

7. SNS DLQ fans out the message to production and disaster recovery SQS DLQ.

Disaster Recovery Message Flow

This flow represents the process for failover. The London region fails Route 53 and sends all traffic to the Ireland region. Again, note that numbers correspond to those shown in Figure 2.

2a. Message reaches disaster recovery infrastructure.

3a. Message body is stored in DR S3.

4a. Amazon EventBridge processes the message and routes it to SNS.

5a. SNS sends the message to DR SQS.

6a. The DR Lambda function receives a message header from DR SQS. It discards messages that DynamoDB indicates have already been processed, otherwise, the DR system routes the messages to the receiver based on the message header. Since the DynamoDB data replication is on, data will be synched with the London region.

7a. SNS DLQ only sends the message to DR SQS DLQ.

Message Exception Flow

If message delivery fails after configured retries, the following exception flow will be triggered.

1. Lambda sends the message to SNS DLQ.

2. SNS DLQ fans out the message to production and DR SQS DLQ.

3. These messages will be processed as a batch or manually.

Message Archival

Messages stored in Amazon S3 can be used for archival purposes and resending messages based on the request. If any receiver application requests an archived message, that can be resent using the Lambda function by reinjecting the messages from the S3 bucket to the message-sending flow.

Benefits

This event-driven architecture designed by DXC Technology improved the responsiveness of the customer application and helped coordinate microservices, exception handling, and retries.

The customer’s tightly coupled, monolithic applications were negatively impacting innovation and scalability. In this architecture, components or services are loosely coupled so events broadcast regardless of who is consuming them.

This saves time because events can be queued and forwarded whenever the receiver is ready to process them. Here, the receiver systems can be spun up and polls the message on a scheduled interval so that cost and energy can be saved. This helps to build a scalable, highly modifiable system.

The decoupled nature of the architecture is also more independent, which speeds development and enables teams to build new features much faster.

Latency is minimized because communications are asynchronous and calls are made in parallel. Since the services are decoupled and interoperable, failure is minimized. If one service fails, others can keep working.

This event-driven architecture is push-based, meaning everything happens on demand as events occur and the customer doesn’t have to pay for continuous polling to check for an event. That helps reduce central processing unit (CPU) utilization and lowers bandwidth consumption.

Conclusion

This post introduced you to one of DXC Technology’s solutions for event-driven architecture on AWS that can be used for any solution or application irrespective of technology.

The different message flows outlined above show service components that are loosely coupled, leading to greater agility.

When the application complexity increases, this event-driven approach provides better scalability, fault tolerance, and faster development. This is one of the major factors in building modern, distributed applications.

.

.

DXC Technology – AWS Partner Spotlight

DXC Technology is an AWS Premier Tier Services Partner that understands the complexities of migrating workloads to AWS in large-scale environments, and the skills needed for success.

Contact DXC Technology | Partner Overview | AWS Marketplace | Case Studies