Filter Messages Published to Topics with Amazon SNS and Amazon SQS
TUTORIAL
Overview
In this tutorial, you will learn how to use the message filtering feature of Amazon Simple Notification Service (Amazon SNS). The message filter feature enables subscribers to an SNS topic to receive only the subset of topic messages they are interested in.
In our example, users visit a website to request insurance quotes. These quote requests can be for vehicle insurance (such as for cars and boats), life insurance, or other insurance types (such as for homes). When a user requests an insurance quote, the request is forwarded to a series of backend systems: a sales system handles vehicle insurance for cars and boats, another sales system handles life insurance, and an analytics system processes all quote requests.
As with the Send Fanout Event Notifications tutorial, we will use a fanout messaging pattern using Amazon SNS and Amazon Simple Queue Service (Amazon SQS) to decouple the website from the backend systems. To get the event notifications to the right backend system, you could create a separate topic for each type of quote request, then add message routing logic to your publisher. However, this option can result in overly complicated publishers, topic proliferation, and additional overhead in provisioning and managing your SNS topics. Amazon SNS message filtering is much simpler.
You will use Amazon SNS and Amazon SQS in this tutorial. Both services are within the AWS Free Tier.
AWS experience
Time to complete
30 minutes
Cost to complete (with free metrics)
Free
Cost to complete (with advanced metrics)
Requires
AWS account and a populated Amazon S3 bucket
Services used
Last updated
August 23, 2022
Implementation
Step 1: Create an SNS topic
Your first step will be to create an Amazon SNS topic to publish messages whenever quote requests are placed on an insurance website. These leads can be related to car insurance, boat insurance, or life insurance.
a. Open the AWS Management Console in a new browser window, so you can keep this step-by-step guide open. When the screen loads, enter your user name and password to get started. Then enter notification in the search bar and select Simple Notification Service to open the service console.
b. If the SNS console landing page appears, choose Start with an overview. If you don't see this page, skip to the next step.
c. Choose Topics in the navigation pane.
d. Choose the Create topic button.
e. For Type, select Standard. In the Name field, enter Insurance-Quote-Requests, then scroll down and choose the Create topic button.
f. The new topic appears on the Topic details page.
Step 2: Create the Amazon SQS queues
a. Open the Amazon SQS console in a new browser window. If the Amazon SQS landing page appears, choose Create queue. Otherwise, proceed to the next step.
b. On the Create queue page, leave Standard selected for Type. Enter Vehicle-Insurance-Quotes in the Name field and then scroll down and choose the Create queue button.
c. The new queue appears on the Queue details page. Select Queues to return to the Queues landing page.
d. Choose the Create queue button and repeat Steps 2b and 2c to create two additional queues using the following names: Life-Insurance-Quotes and All-Quotes.
e. All three queues are now listed in the Amazon SQS console.
Step 3: Subscribe the queues to the topic
Now that you have created the three SQS queues, you need to subscribe them to the SNS topic that broadcasts new insurance quote requests.
a. From the list of queues, select the All-Quotes queue.
Choose Actions and select Subscribe to Amazon SNS topic.
If the SNS topic was made by another account, you could subscribe to it by using the Topic ARN field. For more details, see the Amazon SNS documentation or our blog post Cross-Account Integration with Amazon SNS.
c. Choose Save.
d. The queue is now subscribed to the SNS topic. Choose Queues to return to the Queues page and repeat Steps a-c for the Life-Insurance-Quotes and Vehicle-Insurance-Quotes queues.
Step 4: Assign filter policies to the SNS subscriptions
Subscriptions without filter policies will receive all messages published to an SNS topic, so you don't need to set a filtering policy for the All-Quotes queue for this scenario.
Select the topic name Insurance-Quote-Requests in the list.
b. This will take you to the Insurance-Quote-Requests Topic details page, where you will be able to see all SQS queues subscribed to this SNS topic. Select the Vehicle-Insurance-Quotes queue then choose Edit.
c. This will take you to the Edit subscription page. Select the arrow to the left of Subscription filter policy, and then select Enabled to edit the policy.
d. In the dialog box displayed, enter the following JSON object in the JSON editor:
{"insurance_type": ["car", "boat"]}
Choose Save changes.
e. A confirmation message appears that confirms the subscription filter policy was successfully saved.Click on Insurance-Quote-Requests to go back to the Topic details page,
f. Now you will repeat the process to add the filter policy to the second queue. Select the Life-Insurance-Quotes queue and choose Edit.
g. Select the arrow to the left of Subscription filter policy, and then select Enabled to edit the policy.
h. In the dialog box displayed, enter the following JSON object in the JSON editor:
{"insurance_type": ["life"]}
You don’t need to set any filter policy for the subscription related to the All-Quotes queue. This lack of policy will make this queue catch all events published to the topic.
Step 5: Publish messages to the topic
Let’s test it out. In this step, you will be able to simulate new insurance quote requests coming in from your lead-generation website by publishing messages to the topic.
a. On the Insurance-Quote-Requests topic page, choose Publish message.
2017 Volvo S60, Montreal
In Message attributes:
- Select String in the Type field
- Enter insurance_type in the Name field
- Enter car in the Value field
Choose Publish message.
c. You are returned to the Insurance-Quote-Requests topic page. Choose Publish message again.
d. Create a second message using the following values:
In Message attributes:
- Select String in the Type field
- Enter insurance_type in the Name field
- Enter life in the Value field
Choose Publish message.
e. You are returned to the Insurance-Quote-Requests Topic page. Choose Publish message again.
Townhouse, 1500 sq ft, Toronto
In Message attributes:
- Select String in the Type field
- Enter insurance_type in the Name field
- Enter home in the Value field
Choose Publish message.
Step 6: Verify your message deliveries
It’s time for you to verify that each type of insurance quote request was routed to its corresponding Amazon SQS message queue, by inspecting the message deliveries that took place.
a. Return to the Amazon SQS console. Select the Vehicle-Insurance-Quotes queue, and choose Send and receive messages.
b. Choose Poll for messages.
c. Note that only one message was pushed into this queue, related to the car insurance quote (Insurance Quote Request #1). Once the polling process is complete, view the message pushed into the queue by selecting the message under Messages.
d. View the message, then choose Done to close the dialog box.
e. Repeat the process for the second queue. Select the Life-Insurance-Quotes queue, and choose Send and receive messages.
f. Choose Poll for messages.
g. Note that only one message was pushed into this queue, related to the life insurance quote (Insurance Quote Request #2). Once the polling process is complete, view the message pushed into the queue by selecting the message under Messages.
h. View the message, then choose Done to close the dialog box.
i. Finally, select the All-Quotes queue, and choose Send and receive messages.
j. Choose Poll for messages.
k. Note that all three messages were pushed into this queue (Insurance Quote Requests #1, #2, and #3).
Step 7: Clean up resources
In this step, you will delete the resources you have created for this tutorial, which include a topic, subscriptions, and queues. It is a best practice to delete resources you are no longer using so you don’t incur charges.
a. In the navigation pane of the Amazon SNS console, select Topics, then choose the Insurance-Quote-Requests topic link.
b. On the Insurance-Quote-Requests topic details page, select each of the subscriptions and choose Delete.
c. Choose Delete in the dialog box to confirm the deletion.
d. On the Insurance-Quote-Requests topic details page, choose Delete.
e. Confirm the deletion by entering delete me in the dialog box and then choose Delete.
f. Return to the Amazon SQS console. Select each of the queues used in this tutorial and choose Delete.
g. Confirm the deletion by entering delete in the dialog box and then choose Delete.
Conclusion
You can now use message filtering in Amazon SNS to build simpler and more streamlined pub/sub architectures.
Message filtering enables Amazon SNS topic subscribers to selectively receive only a subset of the messages they are interested in, as opposed to receiving all messages published to a topic. This allows you to offload message-filtering logic from your subscribers and message-routing logic from your publisher systems.