AWS Compute Blog

Simplify Your Pub/Sub Messaging with Amazon SNS Message Filtering

Contributed by: Stephen Liedig, Senior Solutions Architect, ANZ Public Sector, and Otavio Ferreira, Manager, Amazon Simple Notification Service

Want to make your cloud-native applications scalable, fault-tolerant, and highly available? Recently, we wrote a couple of posts about using AWS messaging services Amazon SQS and Amazon SNS to address messaging patterns for loosely coupled communication between highly cohesive components. For more information, see:

Today, AWS is releasing a new message filtering functionality for SNS. This new feature simplifies the pub/sub messaging architecture by offloading the filtering logic from subscribers, as well as the routing logic from publishers, to SNS.

In this post, we walk you through the new message filtering feature, and how to use it to clean up unnecessary logic in your components, and reduce the number of topics in your architecture.

Topic-based filtering

SNS is a fully managed pub/sub messaging service that lets you fan out messages to large numbers of recipients at one time, using topics. SNS topics support a variety of subscription types, allowing you to push messages to SQS queues, AWS Lambda functions, HTTP endpoints, email addresses, and mobile devices (SMS, push).

In the above scenario, every subscriber receives the same message published to the topic, allowing them to process the message independently. For many use cases, this is sufficient.

However, in more complex scenarios, the subscriber may only be interested in a subset of the messages being published. The onus, in that case, is on each subscriber to ensure that they are filtering and only processing those messages in which they are actually interested.

To avoid this additional filtering logic on each subscriber, many organizations have adopted a practice in which the publisher is now responsible for routing different types of messages to different topics. However, as depicted in the following diagram, this topic-based filtering practice can lead to overly complicated publishers, topic proliferation, and additional overhead in provisioning and managing your SNS topics.

Attribute-based filtering

To leverage the new message filtering capability, SNS requires the publisher to set message attributes and each subscriber to set a subscription attribute (a subscription filter policy). When the publisher posts a new message to the topic, SNS attempts to match the incoming message attributes to the filter policy set on each subscription, to determine whether a particular subscriber is interested in that incoming event. If there is a match, SNS then pushes the message to the subscriber in question. The new attribute-based message filtering approach is depicted in the following diagram.

Message filtering in action

Look at how message filtering works. The following example is based on a sports merchandise ecommerce website, which publishes a variety of events to an SNS topic. The events range from checkout events (triggered when orders are placed or canceled) to buyers’ navigation events (triggered when product pages are visited). The code below is based on the existing AWS SDK for Python.

First, create the single SNS topic to which all shopping events are published.

topic_arn = sns.create_topic(
    Name='ShoppingEvents'
)['TopicArn']

Next, subscribe the endpoints that will be listening to those shopping events. The first subscriber is an SQS queue that is processed by a payment gateway, while the second subscriber is a Lambda function that indexes the buyer’s shopping interests against a search engine.

A subscription filter policy is set as a subscription attribute, by the subscription owner, as a simple JSON object, containing a set of key-value pairs. This object defines the kind of event in which the subscriber is interested.

payment_gateway_subscription_arn = sns.subscribe(
    TopicArn = topic_arn,
    Protocol = 'sqs',
    Endpoint = 'arn:aws:sqs:ap-southeast-2:123456789012:PaymentQueue'
)['SubscriptionArn']

sns.set_subscription_attributes(
    SubscriptionArn = payment_gateway_subscription_arn, 
    AttributeName = 'FilterPolicy', 
    AttributeValue = '{"event_type": ["order_placed", "order_cancelled"]}'
)
search_engine_subscription_arn = sns.subscribe(
    TopicArn = topic_arn,
    Protocol = 'lambda',
    Endpoint = 'arn:aws:lambda:ap-southeast-2:123456789012:function:SearchIndex'
)['SubscriptionArn']

sns.set_subscription_attributes(
    SubscriptionArn = search_engine_subscription_arn,
    AttributeName ='FilterPolicy', 
    AttributeValue ='{"event_type": ["product_page_visited"]}'
)

You’re now ready to start publishing events with attributes!

Message attributes allow you to provide structured metadata items (such as time stamps, geospatial data, event type, signatures, and identifiers) about the message. Message attributes are optional and separate from, but sent along with, the message body. You can include up to 10 message attributes with your message.

The first message published in this example is related to an order that has been placed on the ecommerce website. The message attribute “event_type” with the value “order_placed” matches only the filter policy associated with the payment gateway subscription. Therefore, only the SQS queue subscribed to the SNS topic is notified about this checkout event.

message = '{"order": {"id": 5678, "status": "confirmed", "items": [' \
          '{"code": "P-9012", "product": "Santos FC Jersey", "units": 1},' \
          '{"code": "P-3156", "product": "Soccer Ball", "units": 2}]},' \
          ' "buyer": {"id": 4454}}'

sns.publish(
    TopicArn = topic_arn,
    Subject = 'Order Placed #5678',
    Message = message,
    MessageAttributes = {
        'event_type': {
            'DataType': 'String',
            'StringValue': 'order_placed'
        }
    }
)

The second message published is related to a buyer’s navigation activity on the ecommerce website. The message attribute “event_type” with the value “product_page_visited” matches only the filter policy associated with the search engine subscription. Therefore, only the Lambda function subscribed to the SNS topic is notified about this navigation event.

message = '{"product": {"id": 1251, "status": "in_stock"},' \
          ' "buyer": {"id": 4454}}'

sns.publish(
    TopicArn = topic_arn,
    Subject = 'Product Visited #1251',
    Message = message,
    MessageAttributes = {
        'event_type': {
            'DataType': 'String',
            'StringValue': 'product_page_visited'
        }
    }
)

The following diagram represents the architecture for this ecommerce website, with the message filtering mechanism in action. As described earlier, checkout events are pushed only to the SQS queue, whereas navigation events are pushed to the Lambda function only.

Message filtering criteria

It is important to remember the following things about subscription filter policy matching:

  • A subscription filter policy either matches an incoming message, or it doesn’t. It’s Boolean logic.
  • For a filter policy to match a message, the message must contain all the attribute keys listed in the policy.
  • Attributes of the message not mentioned in the filtering policy are ignored.
  • The value of each key in the filter policy is an array containing one or more values. The policy matches if any of the values in the array match the value in the corresponding message attribute.
  • If the value in the message attribute is an array, then the filter policy matches if the intersection of the policy array and the message array is non-empty.
  • The matching is exact (character-by-character), without case-folding or any other string normalization.
  • The values being matched follow JSON rules: Strings enclosed in quotes, numbers, and the unquoted keywords true, false, and null.
  • Number matching is at the string representation level. Example: 300, 300.0, and 3.0e2 aren’t considered equal.

When should I use message filtering?

We recommend using message filtering and grouping subscribers into a single topic only when all of the following is true:

  • Subscribers are semantically related to each other
  • Subscribers consume similar types of events
  • Subscribers are supposed to share the same access permissions on the topic

Technically, you could get away with creating a single topic for your entire domain to handle all event processing, even unrelated use cases, but this wouldn’t be recommended. This option could result in an unnecessarily large topic, which could potentially impact your message delivery latency. Also, you would lose the ability to implement fine-grained access control on your topics.

Finally, if you already use SNS, but had to add filtering logic in your subscribers or routing logic in your publishers (topic-based filtering), you can now immediately benefit from message filtering. This new approach lets you clean up any unnecessary logic in your components, and reduce the number of topics in your architecture.

Summary

As we’ve shown in this post, the new message filtering capability in Amazon SNS gives you a great amount of flexibility in your messaging pattern. It allows you to really simplify your pub/sub infrastructure requirements.

Message filtering can be implemented easily with existing AWS SDKs by applying message and subscription attributes across all SNS supported protocols (Amazon SQS, AWS Lambda, HTTP, SMS, email, and mobile push). It’s now available in all AWS commercial regions, at no extra charge.

Here’s a few ideas for next steps to get you started: