In this tutorial, you will learn how to leverage the message filtering feature of Amazon Simple Notification Service (SNS). The message filter feature enables endpoints subscribed to an SNS topic to receive only the subset of topic messages it is interested in.

In our example, users visit a website to place insurance quote requests. These quote requests can be for car insurance, boat insurance, or life insurance. When an insurance quote request is placed, the request is forwarded to a series of backend systems. There’s a sales system that handles vehicle insurance for cars and boats, another sales system that handles life insurance, and an analytics system that needs to process all quote requests.

As we learned in the Send Fanout Event Notifications Tutorial, we will use a fanout messaging pattern using SNS and SQS to decouple the website from the backend systems. To get the event notifications to the right backend system, you could create a separate topic for each type of quote request, then add message routing logic to your publisher. However, this options can result in overly complicated publishers, topic proliferation, and additional overhead in provisioning and managing your SNS topics. SNS message filtering is much simpler!

You'll use Amazon SNS and Amazon Simple Queue Service (SQS) in this tutorial. Both services are within the AWS Free Tier.

 

Filtering messages on AWS requires an account

Create a Free Account

AWS Free Tier includes 1,000,000 publishes for Amazon Simple Notification Service and 1,000,000 requests of Amazon Simple Queue Service.

View AWS Free Tier Details »

 

Your first step will be to create an Amazon SNS topic to publish messages whenever quote requests are placed on an insurance website. These leads can be related to car insurance, boat insurance, or life insurance.  


a. When you click here, the AWS Management Console will open in a new browser window, so you can keep this step-by-step guide open. When the screen loads, enter your user name and password to get started. Then type notification in the search bar and select Simple Notification Service to open the service console.

tmt_filter-messages-sent-to-topics_1a

(click to zoom)

tmt_filter-messages-sent-to-topics_1a

b.  If the SNS console landing page appears, click Get Started Now. If you don't see this page, skip to the next step.

tmt_filter-messages-sent-to-topics_1b

(click to zoom)

tmt_filter-messages-sent-to-topics_1b

c.  Under Common actions, click Create topic.

 

tmt_filter-messages-sent-to-topics_1c

(click to zoom)

tmt_filter-messages-sent-to-topics_1c

d.  The Create new topic dialog box appears.

In the Topic name field, type Insurance-Quote-Requests, then click Create topic.

 

tmt_filter-messages-sent-to-topics_1d

(click to zoom)

tmt_filter-messages-sent-to-topics_1d

e.  The new topic appears on the Topic details page.

 

tmt_filter-messages-sent-to-topics_1e

(click to zoom)

tmt_filter-messages-sent-to-topics_1e

Each insurance sales system polls quote requests from its corresponding SQS message queue that is subscribed to the SNS topic.

The first queue will handle both car and boat insurance quote requests; the second queue will handle life insurance ones; and the third queue will catch all. The third queue will catch all quote requests, regardless of the insurance type, in order to feed an analytics system. These preferences will be set as filter policies on the SNS subscriptions.

In this step, you will create the three queues and subscribe each of them to the SNS topic.


a. Click here to open the Amazon SQS console in a new browser window. If the SQS landing page appears, click Get Started Now. Otherwise, proceed to the next step.

 

tmt_filter-messages-sent-to-topics_2a

(click to zoom)

tmt_filter-messages-sent-to-topics_2a

b.  On the Create New Queue page, enter Vehicle-Insurance-Quotes in the Queue Name field.

Leave Standard Queue selected and click on Quick-Create Queue.

 

tmt_filter-messages-sent-to-topics_2b

(click to zoom)

tmt_filter-messages-sent-to-topics_2b

c.  Click Create New Queue and repeat Step 2b to create two additional queues:
Life-Insurance-Quotes
  and All-Quotes.

 

tmt_filter-messages-sent-to-topics_2c

(click to zoom)

tmt_filter-messages-sent-to-topics_2c

d.  All three queues are now listed in the SQS Console.

 

tmt_filter-messages-sent-to-topics_2d

(click to zoom)

tmt_filter-messages-sent-to-topics_2d

Now that you have created your three Amazon SQS queues, you need to subscribe them to the Amazon SNS topic that broadcasts new insurance quote requests.


a.  From the list of queues, select the Vehicle-Insurance-Quotes, Life-Insurance-Quotes and All-Quotes queues.

Click Queue Actions and select Subscribe Queues to SNS Topic.

 

 

tmt_filter-messages-sent-to-topics_3a

(click to zoom)

tmt_filter-messages-sent-to-topics_3a

b.  The Subscribe to a Topic dialog box is displayed. From the Choose a Topic drop-down list, select your Insurance-Quote-Requests topic, which you want to subscribe your three queues to. Because you created the SNS topic from the same account, it shows up in the list.

If the SNS topic was made by another account, you could subscribe to it by using the Topic ARN field. For more details, see the Amazon SNS documentation or our blog, Cross-Account Integration with SNS.

 

 

tmt_filter-messages-sent-to-topics_3b

(click to zoom)

tmt_filter-messages-sent-to-topics_3b

c.  Leave the Topic Region unchanged, and click Subscribe.  

 

 

tmt_filter-messages-sent-to-topics_3c

(click to zoom)

tmt_filter-messages-sent-to-topics_3c

d.  The Topic Subscription Result dialog box is displayed. Click OK.

tmt_filter-messages-sent-to-topics_3d

(click to zoom)

tmt_filter-messages-sent-to-topics_3d

Now that all SNS and SQS resources have been created, you are ready to set filter policies to your SNS subscriptions. A filter policy is simple JSON document, set as an attribute of the SNS subscription, which defines the type of notification the subscriber is interested in.

Subscriptions without filter policies will receive all messages published to SNS topic, so you don't need to set a filtering policy for the All-Quotes queue for this scenario.


a.  Return to the Amazon SNS console in your other browser window, and click on Topics to find a list of all your topics in the region.

Click on the topic ARN for Insurance-Quote-Requests in the list.  

tmt_filter-messages-sent-to-topics_4a

(click to zoom)

tmt_filter-messages-sent-to-topics_4a

b.  This will take you to the Topic details page where you will be able to see all SQS queues subscribed to this SNS topic.

tmt_filter-messages-sent-to-topics_4b

(click to zoom)

tmt_filter-messages-sent-to-topics_4b

c.  Select the subscription related to the Vehicle-Insurance-Quotes queue.

From Other subscription actions, click on Edit subscription filter policy.  

tmt_filter-messages-sent-to-topics_4c

(click to zoom)

tmt_filter-messages-sent-to-topics_4c

d.  On the dialog box displayed, enter the following JSON object:
{"insurance_type": ["car", "boat"]}

Click Set subscription filter policy.

 

tmt_filter-messages-sent-to-topics_4d

(click to zoom)

tmt_filter-messages-sent-to-topics_4d

e.  A confirmation dialog appears that confirms the subscription filter policy was successfully created.

 

tmt_filter-messages-sent-to-topics_4e

(click to zoom)

tmt_filter-messages-sent-to-topics_4e

f.  Now you will add the filter policy to the second queue. Select the Life-Insurance-Quotes queue, click Other subscription actions, and choose Edit subscription filter policy.

On the dialog box displayed, enter the following JSON object:
{"insurance_type": ["life"]}

Click Set subscription filter policy.

You don’t need to set any filter policy for the subscription related to the All-Quotes queue. This lack of policy will make this queue catch all events published to the topic.

tmt_filter-messages-sent-to-topics_4f

(click to zoom)

tmt_filter-messages-sent-to-topics_4f

Let’s test it out! In this step, you will be able to simulate new insurance quote requests coming in from your lead generation website by publishing messages to the topic.

You will publish messages with different attributes, so you can view the message filtering mechanism of Amazon SNS in action.


a.  While still on the Topic details page, click on Publish to topic.

tmt_filter-messages-sent-to-topics_5a

(click to zoom)

tmt_filter-messages-sent-to-topics_5a

b.  The Publish a message page is displayed. Create the first message by entering the following values in the form:

In Subject, enter: Insurance Quote Request #1

In Message format, leave Raw selected

In Message, enter the following text to represent a car insurance quote request:
2017 Volvo S60, Montreal

In Message attributes:

  • Enter insurance_type in the key field
  • Select String in the Attribute type field
  • Enter car in the value field

Click Publish message.

tmt_filter-messages-sent-to-topics_5b

(click to zoom)

tmt_filter-messages-sent-to-topics_5b

c.  You are returned to the Topic details page. Click on Publish to topic again, and create a second message using the following values:

In Subject, enter: Insurance Quote Request #2

In Message format, leave Raw selected

In Message, enter the following text to represent a life insurance quote request:
Male, 33 years old, Vancouver

In Message attributes:

  • Enter insurance_type in the key field
  • Select String in the Attribute type field
  • Enter life in the value field

Click Publish message.

tmt_filter-messages-sent-to-topics_5c

(click to zoom)

tmt_filter-messages-sent-to-topics_5c

d.  You are returned to the Topic details page once more. To create your third and final message, click on Publish to topic and enter the following values:

In Subject, enter: Insurance Quote Request #3

In Message format, leave Raw selected

In Message, enter the following text to represent a home insurance quote request:
Townhouse, 1500 sq ft, Toronto

In Message attributes:

  • Enter insurance_type in the key field
  • Select String in the Attribute type field
  • Enter home in the value field

Click Publish message.

tmt_filter-messages-sent-to-topics_5d

(click to zoom)

tmt_filter-messages-sent-to-topics_5d

It’s time for you to verify that each type of insurance quote request was routed to its corresponding SQS message queue, by inspecting the message deliveries that took place.

You will see that a subscribing Amazon SQS queue has gotten a message only if its corresponding SNS subscription had a filter policy that matched the attributes of this incoming message.

You will also notice that one queue in particular actually received all messages published to the topic. This happened because this subscription had no filter policy set, which resulted in a catch-all behavior.


a.  Return to the Amazon SQS console. Select the Vehicle-Insurance-Quotes queue, and click on Queue Actions.

Choose View/Delete Messages.  

tmt_filter-messages-sent-to-topics_6a

(click to zoom)

tmt_filter-messages-sent-to-topics_6a

b.  Click Start Polling for Messages.

Note that only one message was pushed into this queue, related to that car insurance quote (Insurance Quote Request #1). Close the dialog box.

tmt_filter-messages-sent-to-topics_6b

(click to zoom)

tmt_filter-messages-sent-to-topics_6b

c.  Repeat the process for the second queue. Check the box for the Life-Insurance-Quotes queue. From the Queue Actions drop-down list, select View/Delete Messages.

Click Start Polling for Messages.

Note that another message was pushed into this queue, related to that life insurance quote (Insurance Quote Request #2).

Click Close.

tmt_filter-messages-sent-to-topics_6c

(click to zoom)

tmt_filter-messages-sent-to-topics_6c

d.  Finally, check the box for the All-Quotes queue. Click Queue Actions, select View/Delete Messages, and click Start Polling for Messages.

Note that all three messages were pushed into this queue (Insurance Quote Requests #1, #2 and #3).

Click Close.

tmt_filter-messages-sent-to-topics_6d

(click to zoom)

tmt_filter-messages-sent-to-topics_6d

In this step, you will delete the resources you have created for this tutorial, which include a topic, subscriptions and queues. It is a best practice to delete resources you are no longer using, so you don’t incur charges.


a.  In the Amazon SNS console, click Topics, then click on the ARN link related to topic Insurance-Quote-Requests.

tmt_filter-messages-sent-to-topics_7a

(click to zoom)

tmt_filter-messages-sent-to-topics_7a

b.  On the Topic details page, select all three subscriptions.

Click Other subscription actions, then click Delete subscriptions.

Click Delete in the dialog box to confirm the deletion.

tmt_filter-messages-sent-to-topics_7b

(click to zoom)

tmt_filter-messages-sent-to-topics_7b

c.  On the Topic details page, click on Other topic actions, then choose Delete topic.

Confirm the deletion on the dialog box.


tmt_filter-messages-sent-to-topics_7c

(click to zoom)

tmt_filter-messages-sent-to-topics_7c

d.  Return to the Amazon SQS console. Select all three queues used in this tutorial, click Queue Actions, then click Delete Queues.

Confirm the deletion on the dialog box.

tmt_filter-messages-sent-to-topics_7d

(click to zoom)

tmt_filter-messages-sent-to-topics_7d

e.  You can now sign out of the AWS Management Console.  

tmt_filter-messages-sent-to-topics_7e

(click to zoom)

tmt_filter-messages-sent-to-topics_7e

You can now use message filtering on Amazon Simple Notification Service (SNS) to build simpler and more streamlined pub/sub architectures.

Message filtering enables Amazon SNS topic subscribers to selectively receive only a subset of the messages they are interested in, as opposed to receiving all messages published to a topic. This removes the design complexity of implementing message filtering logic in your subscribers, or message routing logic in your publisher systems.


Now that you have learned how to implement message filtering using Amazon Simple Notification Service, pick one of the following options to dive deeper into cloud messaging on AWS.

Learn more

Get an overview of the capabilities, components, and use cases for message topics on Amazon Web Services (AWS).

Visit the Amazon SNS webpage »

Dive deeper

Explore the Developer Guide to see how to implement message filtering for your SNS topics.

Read the documentation »

See it in action

Explore an example that shows you how to implement message filtering using the AWS SDK for Python.

Read the blog »

Yes
No