Open Sourcing Event Ruler

In 2019, we launched Amazon EventBridge, a serverless event bus service that makes it easy for developers and architects to connect applications with data from a variety of sources. At the core of this service are routing rules that determine where to send your data in real-time based on the event payload. Our customers tell us that they love this event-based pattern because it’s fast, clean, and easy to use.

Today, Amazon is excited to open source a key technology that powers EventBridge as Event Ruler under the Apache 2.0 license. It is a Java library that allows you to build applications that can match any number of rules against events at several hundred thousand events per second. Both events and rules are JSON objects, but rules additionally can be expressed through an inbuilt query language that helps describe custom matching patterns.

This offers a novel solution for anyone in need to match lots of patterns, policies, or expressions against any amount of events without compromising on speed. Whether the events are in single digits or several hundred thousand, you can route, filter, or compare them in near real-time against any traffic. This speed is mostly independent of the number of rules or the patterns you define within these rules.

It’s easiest to explain how to use Event Ruler by example. Let’s take the example event from RFC8259 (which specifies the JSON data format).

{
  "Image" : {
    "Width" :  800,
    "Height" : 600,
    "Title" :  "View from 15th Floor",
    "Thumbnail" : {
      "Url" :    "http://www.example.com/image/481989943",
      "Height" : 125,
      "Width" :  100
    },
    "Animated" : false,
    "IDs" : [116, 943, 234, 38793]
  }
}

Now if you take this sample rule, also a JSON object, where we are looking for all static images; it will match against the event.

{
  "Image": {
    "Animated" : [ false ]
  }
}

You can also combine multiple rules into a single expressions for brevity. For example, you can create a rule that looks for any image that has either 800 width and 600 height, or has 600 width and 800 height.

{
    "Image" : {
        "$or" : [
            { "Width" : [ 800 ],  "Height" : [ 600 ] },
            { "Width" : [ 600 ],  "Height" : [ 800 ] }
        ]
    }
}

In practice, most rules are simple, but Event Ruler can handle arbitrary rule complexity. For example, you can match the same event against this convoluted sample rule where we look for all images that are a specific size, have a title, contain a thumbnail URL from “http://www.example.com”, and do not reference ID 100 or 200.

{
  "Image": {
    "Width" : [ { "numeric" : [ "<", 1000 ] } ],
    "Height" : [ { "numeric" : [ "<=", 600 ] } ],
    "Title" : [ { "exists": true } ],
    "Thumbnail" : {
      "Url" : [ { "prefix" : "http://www.example.com" } ]
    },
    "IDs" : [ { "anything-but": [ 100, 200 ] } ]
  }
}

For more examples and explanation, see the README.

Features And Performance

The library supports matching against numbers, numeric ranges, IP addresses, booleans, arrays, prefixes, and suffixes. You can also do complex matches against nested JSON objects and check for non-existence of values. You can find details of these and more matchers on GitHub. Despite these features, the library will continue to perform at speed even if you add millions of rules. On my 2019 Intel MacBook Pro, the benchmark tests run at 180K evaluations per second or better, with performance that depends very little on the number or complexity of rules.

Internals

Event Ruler is built on top of finite state machines where we look at each JSON key-value pair one at a time, stepping through nested JSON objects as necessary, and then comparing each against pre-compiled rules. There’s also the ability to add and remove rules at runtime, which will dynamically update a compiled machine. As rules are pre-compiled and optimized for matching, we can perform checks fast. You can visit our GitHub repo to learn more about the internals.

History and Open Sourcing

Event Ruler first shipped in January 2016, as part of Amazon CloudWatch Events which was a precursor to Amazon EventBridge. At that point rules were much less sophisticated than they are now, offering only exact string matching against Event values. Rules grew over the years with contributions from both the CloudWatch Events team and other engineers across AWS and Amazon.

This work was driven, first, by the huge growth in the number of customers using CloudWatch Events, who naturally wanted richer features. Another influence was the adoption of Event Ruler by other groups across AWS and Amazon. This broader adoption brought requirements, which often came with a proposed implementation and improvements.

We know from experience that the features offered by Event Ruler are useful in a wide range of cloud-native application scenarios, which is why we are releasing it as an open source project. By sharing this technology, we hope to drive innovation within the community surrounding event routers, and enable richer experiences in the Event-Driven Architecture space. We hope to invite a deeper examination of the technology that powers EventBridge and many more AWS services, and look forward to working with you in strengthening and enriching it.

Please visit our GitHub repo for more information and to collaborate.