Introducing Amazon EventBridge schema registry and discovery – In preview

Today, AWS announces the preview of Amazon EventBridge schema registry and discovery. These are new developer tool features, which allow you to automatically find events and their structure, or schema, and store them in a shared central location. This makes it faster and easier to build event-driven applications. You can access the registry and generate code bindings for schemas directly in popular Integrated Development Environments (IDEs), including JetBrains IntelliJ and PyCharm, Microsoft Visual Studio Code, as well as through the Amazon EventBridge console and APIs.

What is a schema?

A schema represents the structure of an event, and commonly includes information such as the title and type of each piece of data. For example, in a review on a product website, a schema might include fields for the reviewer’s name, user id, and review description, and that the name is a text string, and the user id is an integer. The event schema is important for developers as it shows what data is contained in the event, and allows them to write code based on that data.

Event-driven architectures

Developers embracing event-driven architectures may use an event bus such as Amazon EventBridge. EventBridge allows application decoupling without needing to write point-to-point integrations between services. This decoupling increases developer independence, as they simply subscribe to the events they’re interested in, reducing dependencies on other teams to write integrations.

However, decoupling introduces a new set of challenges. Finding events and their schema is a manual process. Developers must coordinate with the team responsible for publishing an event, or look through documentation to find its schema, and then manually create an object for the event in order to use it in their code.

EventBridge schema registry solves these problems by introducing two capabilities, a schema registry and schema discovery.

Schema registry

A schema registry stores a collection of schemas. You can use schema registry to search for, find, and track different schemas used and generated by your applications. Schemas for all AWS sources supported in EventBridge are automatically visible in your schema registry. SaaS partner and custom schemas can be generated and added to the registry using the schema discovery feature.

Schema discovery

Schema discovery automates the process of finding schemas and adding them to your registry. When schema discovery is enabled for an EventBridge event bus, the schema of each event sent to the bus is automatically added to the registry. If the schema of an event changes, schema discovery automatically creates a new version in the registry. Once a schema is added to the registry, you can generate a code binding for the schema, either in the EventBridge console or directly in your IDE.

Generally, you only enable schema discovery in your development environments (AWS Free Tier includes 5 million ingested events). Schemas of any new events you create are automatically added to your registry to use when developing your application. If you need to audit all of the events going through your event bus, you can enable discovery on your production event bus, and pay $0.10 per million events ingested for any usage outside of the Free Tier.

Code bindings

Once a schema is added to the registry, you can download a code binding. This allows you to represent the event as a strongly typed object in your code, and take advantage of IDE features such as validation and auto-complete. Code bindings are available for Java, Python, or TypeScript programming languages. You can download bindings from the AWS Management Console, or directly from your IDE with the AWS Toolkit plugin for IntelliJ and VS Code.

If you use the AWS Serverless Application Model (SAM), you can now use the interactive sam init command to generate a serverless application with a schema as a trigger. This automatically adds a class file for the schema to your code, and generates a handler function that serializes the trigger event into an object. This makes it quicker to build serverless event-driven applications.

Viewing the schema registry

You can view the schema registry in the Amazon EventBridge console and download code bindings.

Choose Go to schema registry and then choose Schemas in the left-side navigation bar. You can view and search for built-in AWS schemas, as well as your discovered and custom schemas within the registry.
For example, searching for a schema for AWS Step Functions, there is an existing schema for: aws.states.StepFunctionsExecutionStatusChange. The schema details as well as the JSON representation are visible.

To Download code bindings to use in your IDE, select a language option, download the .zip file, and manually import the schema into your IDE.

Automatically discovering schemas

Schema registry’s discovery capabilities allow you to generate schemas for your own events. EventBridge can ingest events from a number of SaaS vendors including, for example MongoDB.

In this example, I use an online shop, which stores product reviews in a MongoDB database. A MongoDB trigger is configured to send all new database entries to the EventBridge MongoDB partner event bus. A Lambda function is then triggered for all new events and calls Amazon Comprehend to do sentiment analysis on the new reviews. Any negative reviews generate a service desk ticket for further investigation.

I have previously set up MongoDB Atlas to connect as a SaaS partner to EventBridge. I have configured a stitch trigger to send events to EventBridge for any updates to the database.
To discover the schema, in the EventBridge console, I navigate to Events and Partner event sources.
I select the MongoDB event source and choose Associate with event bus.

I choose Associate.
I Navigate to Event buses and choose the MongoDB Custom event bus and choose Start discovery. An event bus-managed rule is created automatically.

I write a new product review, creating a new database record, which triggers a new event on the event bus. If you don’t have a partner event bus setup, you can create a custom event bus, and use the following CLI command to send an example event to that event bus. Replace the EventBusName with your partner event bus.

aws events put-events --entries '[{"Source": "mystore","DetailType": "Review Created","EventBusName":"my-custom-event-bus","Detail": "{\"star_rating\": 5,  \"description\": \"The size and length fit me well and the design is fun. I felt very secure wearing this tshirt. \",  \"helpful_count\": 34,  \"unhelpful_count\": 1,  \"pros\": [\"lightweight\",\"fits well\"  ],  \"cons\": [],  \"customer\": {\"name\": \"Julian Wood\",\"email\": \"julianreview@amazon.com\",\"phone\": \"+1 604 123 1234\"  },  \"product\": {\"product_id\": 788032119674292922,\"title\": \"Encrypt Everything Tshirt\",\"sku\": \"encrypt-everything-tshirt\",\"inventory_id\": 23190823132,\"size\": \"medium\",\"taxable\": true,\"image_url\": \"https://img.mystore.test/encrypt-tshirt.jpg\",\"weight\": 200.0}}"}]'

I navigate to Schema registry and schemas, choose Discovered schema registry, and can see the discovered schema from the new review event.
Choosing the schema name, I can view the generated schema. Here is an excerpt

Schema discovery has automatically discovered the MongoDB schema from events passing through the event bus and added it to the registry.

Downloading code bindings directly into an IDE

Schema code bindings can be downloaded directly from the AWS Management Console as well as within an IDE, for example JetBrains IntelliJ.

I have IntelliJ installed.
I launch IntelliJ, navigate to File | Settings, and choose Plugins.
On the Marketplace tab, in Search plugins, I enter AWS. When AWS Toolkit by Amazon Web Services is displayed, I select it and choose Install. Minimum version 1.9 is required.

I accept the third-party Plugins Privacy Note and choose Restart IDE.

Once the IDE has restarted, I navigate to AWS Explorer at the bottom-left of the IDE to view available schemas.
I choose Configure AWS connection.

I choose Credentials, my local AWS account profiles are automatically loaded from the AWS credentials file.
I choose Schemas\discovered-schemas, right-click and then choose View Schema to see the schema.

I can then use the code binding in a project, I navigate to File | New Project.
A number of languages and frameworks are available to build an application. I want to build a serverless application and so choose AWS and AWS Serverless Application, which builds an AWS Serverless Application Model application project. I choose Next.

I enter a Project name and file location, choosing java8 as the Runtime.
I choose AWS SAM EventBridge App from Scratch (for any Event trigger from a Schema Registry) for Gradle.
I select Credentials and Region
Under Event Schema, I browse the available schema. At the end of the list, I find the discovered MongoDB schema. I select it and choose Finish.

Once the project is created, I can see the event schema imported into the SAM project.

I navigate to and open mongodb-app\HelloWorldFunction\src\main\java\helloworld\App.java to see the Lambda handler created with the event schema details. I can also use full IDE auto-complete with the schema.

I can then add the Amazon Comprehend sentiment analysis code using the schema into the Lambda function.
The SAM template.yaml in the project root directory specifies the Lambda function triggered by an EventBridge event (previously called CloudWatchEvent) using the discovered MongoDB schema.
I change the EventBusName to the correct partner event bus.

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: HelloWorldFunction
      Handler: helloworld.App::handleRequest
      Runtime: java8
      Environment: 
        Variables:
          PARAM1: VALUE
      Events:
        HelloWorld:
          Type: CloudWatchEvent 
          Properties:
            EventBusName:
                - aws.partner/mongodb.com/stitch.trigger/5ddf5c9476ff0ff8b0916763
            Pattern:
              source:
                - aws.partner/mongodb.com.test/stitch.trigger/5ddf5c9476ff0ff8b0916
              detail-type:
                - MongoDB Database Trigger for my_store.reviews

I can then deploy the SAM application.
I navigate to File | Deploy SAM Application .I choose Create Stack and enter a name. Select an S3 bucket and choose Deploy.

The SAM application is deployed. I can also automate the process using SAM CLI. Version 0.35.0 includes the schema features in the new interactive sam init command. This uses an AWS Quick Start Template to generate a handler function and add a class file for the schema to my code.

Pricing

Usage of the schema registry is free.

Schema discovery includes a free tier of 5M ingested events per month. In use cases where discovery is used in your development environment, your usage should stay within the free tier.

For additional usage outside of the free tier:

$0.10 per million events ingested for discovery.

All ingested events are measured in 8 KB chunks.

Availability

The EventBridge schema registry preview is available in the US East (Ohio), US West (Oregon), US East (Northern Virginia) Asia Pacific (Tokyo) Region, and Europe (Ireland) Regions. For details on EventBridge availability, please see the AWS Region table (https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/).

Conclusion

Amazon EventBridge schema registry and discovery helps developers take advantage of new schema capabilities to build exciting event-driven applications. All AWS sources supported in EventBridge are visible in your schema registry. SaaS partner and custom schemas can be added automatically using schema discovery. Code bindings, which allow you to represent the event as an object in your code, can be downloaded from the console or directly within IDEs.

Happy coding with event schemas!

AWS Compute Blog