Filter Messages Published to Topics

with Amazon Simple Notification Service (SNS) and Amazon Simple Queue Service (SQS)

Introduction

In this tutorial, you will learn how to leverage the message filtering feature of Amazon Simple Notification Service (SNS). The message filter feature enables endpoints subscribed to an SNS topic to receive only the subset of topic messages it is interested in.

In our example, users visit a website to place insurance quote requests. These quote requests can be for car insurance, boat insurance, or life insurance. When an insurance quote request is placed, the request is forwarded to a series of backend systems. There’s a sales system that handles vehicle insurance for cars and boats, another sales system that handles life insurance, and an analytics system that needs to process all quote requests.

As we learned in the Send Fanout Event Notifications Tutorial, we will use a fanout messaging pattern using SNS and SQS to decouple the website from the backend systems. To get the event notifications to the right backend system, you could create a separate topic for each type of quote request, then add message routing logic to your publisher. However, this option can result in overly complicated publishers, topic proliferation, and additional overhead in provisioning and managing your SNS topics. SNS message filtering is much simpler!

You'll use Amazon SNS and Amazon Simple Queue Service (SQS) in this tutorial. Both services are within the AWS Free Tier.

 

 Cost to Complete

Free Tier

 Services Used

Filtering Messages on AWS
Requires an Account

AWS Free Tier includes 1,000,000 publishes for Amazon Simple Notification Service and 1,000,000 requests of Amazon Simple Queue Service.

View AWS Free Tier Details »

Implementation

  • Create an Amazon SNS Topic

    Your first step will be to create an Amazon SNS topic to publish messages whenever quote requests are placed on an insurance website. These leads can be related to car insurance, boat insurance, or life insurance.  

    a. When you click here, the AWS Management Console will open in a new browser window, so you can keep this step-by-step guide open. When the screen loads, enter your user name and password to get started. Then type notification in the search bar and select Simple Notification Service to open the service console.

    AWS Management Console

    b.  If the SNS console landing page appears, click Start with an overview. If you don't see this page, skip to the next step.

    If the SNS console landing page appears, click Start with an overview

    c.  Click Topics on the left-hand side menu.

    Click topics on the left-hand side menu

    d.  Click Create topic.

    Click Create topic

    e.  In the Topic name field, type Insurance-Quote-Requests, then scroll down to click Create topic.

    Enter name in topic name field and click create topic

    f.  The new topic appears on Topic details page.

    The new topic appears on topic details page
  • Create the Amazon SQS Queues

    Each insurance sales system polls quote requests from its corresponding SQS message queue that is subscribed to the SNS topic.

    The first queue will handle both car and boat insurance quote requests; the second queue will handle life insurance ones; and the third queue will catch all. The third queue will catch all quote requests, regardless of the insurance type, in order to feed an analytics system. These preferences will be set as filter policies on the SNS subscriptions.
    In this step, you will create the three queues and subscribe each of them to the SNS topic.

    a. Click here to open the Amazon SQS console in a new browser window. If the SQS landing page appears, click Get Started Now. Otherwise, proceed to the next step.

    Click Get Started Now

    b.  On the Create New Queue page, enter Vehicle-Insurance-Quotes in the Queue Name field.

    Leave Standard Queue selected and click on Quick-Create Queue.

    Leave Standard Queue selected

    c.  Click Create New Queue and repeat Step 2b to create two additional queues: Life-Insurance-Quotes and All-Quotes.

    Click create new queue

    d.  All three queues are now listed in the SQS Console.

    All three queues are now listed in the SQS Console
  • Subscribe the Queues to the Topic

    Now that you have created your three Amazon SQS queues, you need to subscribe them to the Amazon SNS topic that broadcasts new insurance quote requests.

    a.  From the list of queues, select the Vehicle-Insurance-Quotes, Life-Insurance-Quotes and All-Quotes queues.

    Click Queue Actions and select Subscribe Queues to SNS Topic.

    Click Queue Actions and select Subscribe Queues to SNS Topic

    b.  The Subscribe to a Topic dialog box is displayed. From the Choose a Topic drop-down list, select your Insurance-Quote-Requests topic, which you want to subscribe your three queues to. Because you created the SNS topic from the same account, it shows up in the list.

    If the SNS topic was made by another account, you could subscribe to it by using the Topic ARN field. For more details, see the Amazon SNS documentation or our blog, Cross-Account Integration with SNS.

    The Subscribe to a Topic dialog box is displayed

    c.  Leave the Topic Region unchanged, and click Subscribe.  

    Leave the Topic Region unchanged, and click Subscribe

    d.  The Topic Subscription Result dialog box is displayed. Click OK.

    The Topic Subscription Result dialog box is displayed. Click OK
  • Set Filter Policies to the SNS Subscriptions

    Now that all SNS and SQS resources have been created, you are ready to set filter policies to your SNS subscriptions. A filter policy is simple JSON document, set as an attribute of the SNS subscription, which defines the type of notification the subscriber is interested in.

    Subscriptions without filter policies will receive all messages published to SNS topic, so you don't need to set a filtering policy for the All-Quotes queue for this scenario.

    a.  Return to the Amazon SNS console in your other browser window, and click on Topics to find a list of all your topics in the region.

    Click on the topic name Insurance-Quote-Requests in the list.  

    Click on the topic name Insurance-Quote-Requests in the list

    b.  This will take you to the Insurance-Quote-Requests Topic details page where you will be able to see all SQS queues subscribed to this SNS topic. Select Vehicle-Insurance-Quotes queue then click Edit.

    Topic details page

    c.  This will take you to the Edit subscription page. Click on the arrow to edit subscription filter policy. 

    Click on the arrow to edit subscription filter policy

    d.  On the dialog box displayed, enter the following JSON object:
    {"insurance_type": ["car", "boat"]}

    Click Save changes.

    Enter JSON object in the dialog box displayed

    e.  A confirmation dialog appears that confirms the subscription filter policy was successfully created.

    A confirmation dialog appears that confirms the subscription filter policy was successfully created

    f.  Now you will repeat the process add the filter policy to the second queue. Select the Life-Insurance-Quotes queue, click Edit.

    Click on the arrow to edit subscription filter policy.

    On the dialog box displayed, enter the following JSON object:
    {"insurance_type": ["life"]}

    Click Save changes.

    You don’t need to set any filter policy for the subscription related to the All-Quotes queue. This lack of policy will make this queue catch all events published to the topic.

    Click on the arrow to edit subscription filter policy
  • Publish Messages to the Topic

    Let’s test it out! In this step, you will be able to simulate new insurance quote requests coming in from your lead generation website by publishing messages to the topic.

    You will publish messages with different attributes, so you can view the message filtering mechanism of Amazon SNS in action.

    a.  On the Insurance-Quote-Requests Topic page, click on Publish message.

    Click publish message

    b.  Create the first message by entering the following values in the form:

    In Subject, enter: Insurance Quote Request #1

    In Message structure, leave Identical payload for all delivery protocols selected

    In Message body to send to the endpoint, enter the following text to represent a car insurance quote request:
    2017 Volvo S60, Montreal

    In Message attributes:

    • Select String in the Type field
    • Enter insurance_type in the Name field
    • Enter car in the Value field

    Click Publish message.

    Click Publish Message

    c. You are returned to the Insurance-Quote-Requests Topic page. Click on Publish message again, and create a second message using the following values:

    In Subject, enter: Insurance Quote Request #2

    In Message structure, leave Identical payload for all delivery protocols selected

    In Message body to send to the endpoint, enter the following text to represent a car insurance quote request:
    Male, 33 years old, Vancouver

    In Message attributes:

    • Select String in the Type field
    • Enter insurance_type in the Name field
    • Enter life in the Value field

    Click Publish message.

    Click Publish Message

    d. You are returned to the Insurance-Quote-Requests Topic page. Click on Publish message again, and create a second message using the following values:

    In Subject, enter: Insurance Quote Request #3

    In Message structure, leave Identical payload for all delivery protocols selected

    In Message body to send to the endpoint, enter the following text to represent a car insurance quote request:
    Townhouse, 1500 sq ft, Toronto

    In Message attributes:

    • Select String in the Type field
    • Enter insurance_type in the Name field
    • Enter home in the Value field

    Click Publish message.

    Click Publish Message
  • Verify Your Message Deliveries

    It’s time for you to verify that each type of insurance quote request was routed to its corresponding SQS message queue, by inspecting the message deliveries that took place.

    You will see that a subscribing Amazon SQS queue has gotten a message only if its corresponding SNS subscription had a filter policy that matched the attributes of this incoming message.

    You will also notice that one queue in particular actually received all messages published to the topic. This happened because this subscription had no filter policy set, which resulted in a catch-all behavior.

    a.  Return to the Amazon SQS console. Select the Vehicle-Insurance-Quotes queue, and click on Queue Actions.

    Choose View/Delete Messages.  

    From the Queue Action drop-down, select View/Delete Messages

    b.  Click Start Polling for Messages.

    Note that only one message was pushed into this queue, related to that car insurance quote (Insurance Quote Request #1). Close the dialog box.

    Click Start Polling for Messages

    c.  Repeat the process for the second queue. Check the box for the Life-Insurance-Quotes queue. From the Queue Actions drop-down list, select View/Delete Messages

    Click Start Polling for Messages.

    Note that another message was pushed into this queue, related to that life insurance quote (Insurance Quote Request #2).

    Click Close.

    The View/Delete Messages in Life-Insurance-Quotes dialog box appears

    d.  Finally, check the box for the All-Quotes queue. Click Queue Actions, select View/Delete Messages, and click Start Polling for Messages.

    Note that all three messages were pushed into this queue (Insurance Quote Requests #1, #2 and #3).

    Click Close.

    Click close
  • Delete Your Resources

    In this step, you will delete the resources you have created for this tutorial, which include a topic, subscriptions and queues. It is a best practice to delete resources you are no longer using, so you don’t incur charges.

    a.  In the Amazon SNS console, click Topics, then click on Insurance-Quote-Requests topic link.

    Click on Insurance-Quote-Requests topic link

    b.  On the Insurance-Quote-Requests Topic details page, select each of the subscriptions and click Delete.

    Click Delete in the dialog box to confirm the deletion. 

    Click Delete in the dialog box to confirm the deletion

    c.  On the Insurance-Quote-Requests Topic details page, click on Delete.

    Confirm the deletion by typing delete me on the dialog box and click on Delete

    Select Delete

    d.  Return to the Amazon SQS console. Select all three queues used in this tutorial, click Queue Actions, then click Delete Queues.

    Confirm the deletion on the dialog box.

     Click Yes, Delete 2 Queues

    e.  You can now sign out of the AWS Management Console.  

    Sign Out of the AWS Management Console

Was this tutorial helpful?

Thank you
Please let us know what you liked.
Close
Sorry to disappoint you
Is something out-of-date, confusing or inaccurate? Please help us improve this tutorial by providing feedback.
Close

Congratulations!

You can now use message filtering on Amazon Simple Notification Service (SNS) to build simpler and more streamlined pub/sub architectures.

Message filtering enables Amazon SNS topic subscribers to selectively receive only a subset of the messages they are interested in, as opposed to receiving all messages published to a topic. This removes the design complexity of implementing message filtering logic in your subscribers, or message routing logic in your publisher systems.