Integrating Amazon EventBridge and Amazon ECS

This post is courtesy of Jakub Narloch, Senior Software Development Engineer.

Today, AWS announced the support for Amazon API Gateway as an event target in Amazon EventBridge. This feature enables new integration scenarios for web applications and services. It allows customers to seamlessly connect their infrastructure, SaaS services, and APIs hosted in AWS.

With API Gateway as a target for EventBridge, this creates new integration capabilities for new or existing web applications. This post explains how developers can now deliver events directly to their applications hosted on Amazon ECS, Amazon Elastic Kubernetes Service (EKS), or Amazon EC2 using EventBridge and API Gateway. In this post, I show how to build an event driven application running on ECS Fargate that process events from EventBridge using API Gateway integration.

EventBridge is a serverless event bus that makes it easy to connect applications together. It uses data from your own applications, integrated software as a service (SaaS) applications, and AWS services. This simplifies the process of building event-driven architectures by decoupling event producers from event consumers. This allows producers and consumers to be scaled, updated, and deployed independently. Loose coupling improves developer agility in addition to application resiliency.

API Gateway helps developers to create, publish, and maintain secure APIs at any scale. When used with EventBridge, API Gateway authenticates and authorizes API calls. It also acts as an HTTP proxy layer for integrating other AWS services or third-party web applications.

Previously, EventBridge customers could consume events from EventBridge in ECS via Amazon SNS or Amazon SQS, or by triggering an ECS task directly. API Gateway as a target replaces this approach and brings additional API Gateway features like authentication and rate limiting. This can help you build more resilient and feature-rich integrations. API Gateway throttling limits the maximum number events delivered at a same time, while EventBridge retries events delivery for up to 24 hours.

This blog post uses an ecommerce application as an example of a custom integration. The application is responsible for processing customer orders. The following diagram illustrates the interaction of the components of the system. The application itself is hosted as ECS service on top of AWS Fargate.

To achieve high availability, the application cluster is distributed across subnets in different Availability Zones. The Application Load Balancer ensures that the incoming traffic is distributed across the nodes in the cluster. API Gateway is responsible for authenticating requests and routing to the backend. The application logic is responsible for receiving the event and persisting it in Amazon DocumentDB.

The order event is modeled as follows:

{
  "version": "0",
  "region": "us-east-1",
  "account": "123456789012",
  "id": "4236e18e-f858-4e2b-a8e8-9b772525e0b2",
  "source": "ecommerce",
  "detail-type": "CreateOrder",
  "resources": [],
  "detail": {
    "order_id": "ce4fe8b7-9911-4377-a795-29ecca9d7a3d",
    "create_date": "2020-06-02T13:01:00Z",
    "items": [
      {
        "product_id": "b8575571-5e91-4521-8a29-4af4a8aaa6f6",
        "quantity": 1,
        "price": "9.99",
        "currency": "CAD"
      }
    ],
    "customer": {
      "customer_id": "5d22899e-3ff5-4ce0-a2a3-480cfce39a56"
    },
    "payment": {
      "payment_id": "fb563473-bef4-4965-ad78-e37c6c9c0c2a",
    },
    "delivery_address": {
      "street": "510 W Georgia St",
      "city": "Vancouver",
      "state": "BC",
      "country": "Canada",
      "zip_code": "V6B0M7"
    }
  }
}

Application layer
The application that processes the orders is implemented using a reactive stack through Spring Boot. A reactive application design can help build a scalable application capable of handling thousands of transactions per second from a single instance. This is important for applications with high throughput and can help in achieving economies of scale.

The resource handler
The application itself defines a OrderResource, which acts as entry handler for receiving the events from EventBridge and processing them. The handler logic is responsible for unmarshalling the event and retrieving the order details out of the event detail. The order is then persisted in DocumentDB using a dedicated DAO instance.

@Slf4j
@RequestMapping("/orders")
@RestController
public class EventResource {
 
    private final OrderRepository orderRepository;
 
    public EventResource(OrderRepository orderRepository) {
        this.orderRepository = Objects.requireNonNull(orderRepository);
    }
 
    @RequestMapping(method = RequestMethod.PUT)
    public Mono<ResponseEntity<Object>> onEvent(@Valid @RequestBody Event<Order> event) {
 
        log.info("Processing order {}", event.getDetail().getOrderId());
 
        return orderRepository.save(event.getDetail())
                .map(order -> ResponseEntity.created(UriComponentsBuilder.fromPath("/events/{id}")
                        .build(order.getOrderId())).build())
                .onErrorResume(error -> {
                    log.error("Unhandled error occurred", error);
                    return Mono.just(ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).build());
                });
    }
}

The handler is a mapped to process requests at ‘/orders’ path. The implementation unmarshals an event payload and stores it into DocumentDB. Upon successful execution, the service responds with a 201 Created HTTP status code.

You can store EventBridge events in a document database like Amazon DocumentDB. This is a non-relational database that allows you to store JSON content directly. This example uses DocumentDB for storing the documents, making it easy for writing the event payload directly. It also supports general querying of event content.

Prerequisites
To build and deploy the application, you must have AWS CDK and JDK 11 installed. Start by cloning the GitHub repository. The repository contains the example code and supporting infrastructure for deploying to AWS.

Step 1: Create Amazon ECR repository.
Start by creating a dedicated Amazon ECR repository, where Docker images are uploaded. There is an AWS CDK template in the application code repo for this purpose.

First, install Node.js dependencies needed to execute the CDK command:

cd ../eventbridge-integration-solution-aws-api-cdk
npm install

Next, compile the CDK TypeScript template.

npm run build

Finally, synthesize the CloudFormation stack.

cdk synth "*"

Now bootstrap CloudFormation resources needed to deploy the remaining templates.

cdk bootstrap

Finally, deploy the stack that creates the Amazon ECR registry.

cdk deploy EventsRegistry

Step 2: Build the application

Before the application is deployed, it must be built and uploaded to Amazon ECR.
To get started, compile the source code and build the application distribution.

cd ../eventbridge-integration-solution-aws-api
./gradlew clean build

Step 3: Containerizing the application
The build system is configured to include the task for containerizing the artifacts and creating the Docker image. To create a new Docker image from the build artifact, run the following command:

./gradlew dockerBuildImage

The build task generates the Dockerfile using the provided settings. It then executes the docker build command to create a new Docker image named eventbridge-integration-solution-aws-api.

Step 4: Upload the image to Amazon ECR
You can now upload the image directly to Amazon ECR. First, login into the Amazon ECR registry through Docker. Replace AWS_ACCOUNT_ID with your actual account.

aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin "${AWS_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com"

Before uploading the image to ECR, tag it with the expected remote repository name. To do that, first list all of the Docker images.

docker images

Copy the image id attribute of eventbridge-integration-solution-aws-api image and use it in the tag command, also replacing AWS_ACCOUNT_ID.

docker tag $DOCKER_IMAGE "${AWS_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com/eventbridge-integration-solution-aws-api"

Finally, push the Docker image to ECR, replacing AWS_ACCOUNT_ID with your AWS account ID.

docker push "${AWS_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com/eventbridge-integration-solution-aws-api"

Step 5: Deploying the application stack
Once the application image is uploaded to Amazon ECR, you can deploy the entire application stack using CDK. The stack creates multiple resources including a VPC, DocumentDB cluster, ECS TaskDefinition and Service, Application Load Balancer, API Gateway and EventBridge rule. You can inspect the resources created in the CDK definition by opening the TypeScript files in the eventbridge-integration-solution-aws-api-cdk/lib directory.

At this point, you can proceed with deploying the CloudFormation stack.

cd ../eventbridge-integration-solution-aws-api-cdkcdk deploy "*"

Step 6: Testing running application
Now, test the end to end event delivery by publishing the sample events to the EventBridge PutEvents API. Create a file named event.json and paste the following code:

[
  {
    "Source": "ecommerce",
    "DetailType": "CreateOrder",
    "Detail": "{\"order_id\": \"ce4fe8b7-9911-4377-a795-29ecca9d7a3d\",\"create_date\": \"2020-06-02T13:01:00Z\",\"items\": [{\"product_id\": \"b8575571-5e91-4521-8a29-4af4a8aaa6f6\",\"quantity\": 1,\"price\": \"9.99\",\"currency\": \"CAD\"}],\"customer\": {\"customer_id\": \"5d22899e-3ff5-4ce0-a2a3-480cfce39a56\"},\"payment\": {\"payment_id\": \"fb563473-bef4-4965-ad78-e37c6c9c0c2a\"},\"delivery_address\": {\"street\": \"510 W Georgia St\",\"city\": \"Vancouver\",\"state\": \"BC\",\"country\": \"Canada\",\"zip_code\": \"V6B0M7\"}}"
  }
]

Publish this event with the following AWS CLI command.

aws events put-events --entries file://event.json

EventBridge delivers the event to API Gateway and the application persists it in DocumentDB.

Step 7: Cleanup
Delete all the resources created in this tutorial by running this CDK command:

cdk destroy "*"

Additional considerations
The demo application is simplified for the purpose of showcasing the EventBridge integration with API Gateway. In production, it’s recommended that you isolate the DocumentDB cluster in a private subnet. Additionally, the Application Load Balancer can be hidden from public access and connected to API Gateway through VPC Link.

Conclusion

This post demonstrates how to set up a sample application for consuming events directly from EventBridge into a custom application hosted in ECS. This integration uses EventBridge’s native support for API Gateway as a target that allows to integrate any HTTP base web applications.

Learn more from the EventBridge documentation.

AWS Compute Blog

Integrating Amazon EventBridge and Amazon ECS

Conclusion

Resources

Follow