Filter Messages Published to Topics with Amazon SNS and Amazon SQS

TUTORIAL

Overview

In this tutorial, you will learn how to use the message filtering feature of Amazon Simple Notification Service (Amazon SNS). The message filter feature enables subscribers to an SNS topic to receive only the subset of topic messages they are interested in.

In our example, users visit a website to request insurance quotes. These quote requests can be for vehicle insurance (such as for cars and boats), life insurance, or other insurance types (such as for homes). When a user requests an insurance quote, the request is forwarded to a series of backend systems: a sales system handles vehicle insurance for cars and boats, another sales system handles life insurance, and an analytics system processes all quote requests.

As with the Send Fanout Event Notifications tutorial, we will use a fanout messaging pattern using Amazon SNS and Amazon Simple Queue Service (Amazon SQS) to decouple the website from the backend systems. To get the event notifications to the right backend system, you could create a separate topic for each type of quote request, then add message routing logic to your publisher. However, this option can result in overly complicated publishers, topic proliferation, and additional overhead in provisioning and managing your SNS topics. Amazon SNS message filtering is much simpler.

You will use Amazon SNS and Amazon SQS in this tutorial. Both services are within the AWS Free Tier.

 AWS experience

Intermediate

 Time to complete

30 minutes

 Cost to complete (with free metrics)

Free

 Cost to complete (with advanced metrics)

Prices may vary: $0.20 per million objects monitored per month ( Amazon S3 pricing page)

 Requires

AWS account and a populated Amazon S3 bucket

 Services used

 Last updated

August 23, 2022

Implementation

Step 1: Create an SNS topic

Your first step will be to create an Amazon SNS topic to publish messages whenever quote requests are placed on an insurance website. These leads can be related to car insurance, boat insurance, or life insurance.  

a. Open the AWS Management Console in a new browser window, so you can keep this step-by-step guide open. When the screen loads, enter your user name and password to get started. Then enter notification in the search bar and select Simple Notification Service to open the service console.


b. If the SNS console landing page appears, choose Start with an overview. If you don't see this page, skip to the next step.

c. Choose Topics in the navigation pane.

d. Choose the Create topic button.

e. For Type, select Standard. In the Topic name field, enter Insurance-Quote-Requests, then scroll down and choose the Create topic button.

f.  The new topic appears on the Topic details page.

Step 2: Create the Amazon SQS queues

Each insurance sales system polls quote requests from the corresponding SQS message queue that is subscribed to the SNS topic.
 
The first queue will handle both car and boat insurance quote requests; the second will handle life insurance quote requests; and the third will catch all. Regardless of the insurance type, the third queue will catch all quote requests to feed the analytics system. These preferences will be set as filter policies on the SNS subscriptions.
 
In this step, you will create the three queues and subscribe each of them to the SNS topic.

a. Open the Amazon SQS console in a new browser window. If the Amazon SQS landing page appears, choose Create queue. Otherwise, proceed to the next step.

b. On the Create queue page, leave Standard selected for Type. Enter Vehicle-Insurance-Quotes in the Name field and then scroll down and choose the Create queue button.

c. The new queue appears on the Queue details page. Select Queues to return to the Queues landing page.

d. Choose the Create queue button and repeat Step 2b to create two additional queues using the following names: Life-Insurance-Quotes and All-Quotes.

e.  All three queues are now listed in the Amazon SQS console.

Step 3: Subscribe the queues to the topic

Now that you have created the three SQS queues, you need to subscribe them to the SNS topic that broadcasts new insurance quote requests.

a. From the list of queues, select the All-Quotes queue.

Choose Actions and select Subscribe to Amazon SNS topic.

b. The Subscribe to Amazon SNS topic dialog box is displayed. From the Specify an Amazon SNS topic available for this queue dropdown list, select the Insurance-Quote-Requests topic, which you will subscribe your three queues to. Because you created the SNS topic from the same account, it shows up in the list.

If the SNS topic was made by another account, you could subscribe to it by using the Topic ARN field. For more details, see the Amazon SNS documentation or our blog post Cross-Account Integration with Amazon SNS.

c. Choose Save.

d. The query is now subscribed to the SNS topic. Choose Queues to return to the Queues page and repeat Steps a-c for Life-Insurance-Quotes and Vehicle-Insurance Quotes.

Step 4: Assign filter policies to the SNS subscriptions

Now that all Amazon SNS and Amazon SQS resources have been created, you are ready to assign filter policies to your SNS subscriptions. A filter policy is a simple JSON document, set as an attribute of the SNS subscription, which defines the type of notification the subscriber is interested in.

Subscriptions without filter policies will receive all messages published to an SNS topic, so you don't need to set a filtering policy for the All-Quotes queue for this scenario.

a. Return to the Amazon SNS console in your other browser window, and select Topics in the navigation pane to list all of your Topics in the Region.

Select the topic name Insurance-Quote-Requests in the list. 

b. This will take you to the Insurance-Quote-Requests Topic details page, where you will be able to see all SQS queues subscribed to this SNS topic. Select the Vehicle-Insurance-Quotes queue then choose Edit.

c. This will take you to the Edit subscription page. Select the arrow to the left of Subscription filter policy to edit the policy.

d. In the dialog box displayed, enter the following JSON object in the JSON editor:
{"insurance_type": ["car", "boat"]}

Choose Save changes.

e. A confirmation message appears that confirms the subscription filter policy was successfully saved.

f. Now you will repeat the process to add the filter policy to the second queue. Select the Life-Insurance-Quotes queue and choose Edit.

g. Select the arrow to the left of Subscription filter policy to edit the policy.

h. In the dialog box displayed, enter the following JSON object in the JSON editor:
{"insurance_type": ["life"]}

 
Choose Save changes.

 

You don’t need to set any filter policy for the subscription related to the All-Quotes queue. This lack of policy will make this queue catch all events published to the topic.

Step 5: Publish messages to the topic

Let’s test it out. In this step, you will be able to simulate new insurance quote requests coming in from your lead-generation website by publishing messages to the topic.

You will publish messages with different attributes, so you can view the message-filtering mechanism of Amazon SNS in action.

a. On the Insurance-Quote-Requests topic page, choose Publish message.

b. Create the first message by entering the following values in the form:
 
In Subject, enter Insurance Quote Request #1.
 
In Message structure, leave Identical payload for all delivery protocols selected.
 
In Message body to send to the endpoint, enter the following text to represent a car insurance quote request:

2017 Volvo S60, Montreal

In Message attributes:

  • Select String in the Type field
  • Enter insurance_type in the Name field
  • Enter car in the Value field

Choose Publish message.

c. You are returned to the Insurance-Quote-Requests topic page. Choose Publish message again.

d. Create a second message using the following values:

In Subject, enter Insurance Quote Request #2
 
In Message structure, leave Identical payload for all delivery protocols selected.
 
In Message body to send to the endpoint, enter the following text to represent a car insurance quote request:
 
Male, 33 years old, Vancouver

In Message attributes:

  • Select String in the Type field
  • Enter insurance_type in the Name field
  • Enter life in the Value field

Choose Publish message.

e. You are returned to the Insurance-Quote-Requests Topic page. Choose Publish message again.

f. Create a third message using the following values:
 
In Subject, enter Insurance Quote Request #3.
 
In Message structure, leave Identical payload for all delivery protocols selected
 
In Message body to send to the endpoint, enter the following text to represent a home insurance quote request:

Townhouse, 1500 sq ft, Toronto

In Message attributes:

  • Select String in the Type field
  • Enter insurance_type in the Name field
  • Enter home in the Value field

Choose Publish message.

Step 6: Verify your message deliveries

It’s time for you to verify that each type of insurance quote request was routed to its corresponding Amazon SQS message queue, by inspecting the message deliveries that took place.

You will see that a subscribing Amazon SQS queue has gotten a message only if its corresponding Amazon SNS subscription had a filter policy that matched the attributes of this incoming message.
 
You will also notice that one queue in particular actually received all messages published to the topic. This happened because this subscription had no filter policy set, which resulted in a catch-all behavior.

a. Return to the Amazon SQS console. Select the Vehicle-Insurance-Quotes queue, and choose Send and receive messages.

b. Choose Poll for messages.

c. Note that only one message was pushed into this queue, related to the car insurance quote (Insurance Quote Request #1). Once the polling process is complete, view the message pushed into the queue by selecting the message under Messages.

d. View the message, then choose Done to close the dialog box.

e. Repeat the process for the second queue. Select the Life-Insurance-Quotes queue, and choose Send and receive messages.

f. Choose Poll for messages.

g. Note that only one message was pushed into this queue, related to the life insurance quote (Insurance Quote Request #2). Once the polling process is complete, view the message pushed into the queue by selecting the message under Messages.

h. View the message, then choose Done to close the dialog box.

i. Finally, select the All-Quotes queue, and choose Send and receive messages.

j. Choose Poll for messages.

k. Note that all three messages were pushed into this queue (Insurance Quote Requests #1, #2, and #3).

Step 7: Clean up resources

In this step, you will delete the resources you have created for this tutorial, which include a topic, subscriptions, and queues. It is a best practice to delete resources you are no longer using so you don’t incur charges.

a. In the navigation pane of the Amazon SNS console, select Topics, then choose the Insurance-Quote-Requests topic link.

b. On the Insurance-Quote-Requests topic details page, select each of the subscriptions and choose Delete.

c. Choose Delete in the dialog box to confirm the deletion.

d. On the Insurance-Quote-Requests topic details page, choose Delete.

e. Confirm the deletion by entering delete me in the dialog box and then choose Delete.

f. Return to the Amazon SQS console. Select each of the queues used in this tutorial and choose Delete.

g. Confirm the deletion in the dialog box.

h.  You can now sign out of the AWS Management Console.

Conclusion

You can now use message filtering in Amazon SNS to build simpler and more streamlined pub/sub architectures.

Message filtering enables Amazon SNS topic subscribers to selectively receive only a subset of the messages they are interested in, as opposed to receiving all messages published to a topic. This allows you to offload message-filtering logic from your subscribers and message-routing logic from your publisher systems.

Was this page helpful?

Next steps