Q: What is Amazon SQS?
Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable hosted queue for storing messages as they travel between computers. By using Amazon SQS, developers can simply move data between distributed application components performing different tasks, without losing messages or requiring each component to be always available. Amazon SQS makes it easy to build a distributed, decoupled application, working in close conjunction with the Amazon Elastic Compute Cloud (Amazon EC2) and the other AWS infrastructure web services.
Q: What can I do with Amazon SQS?
Amazon SQS is a web service that gives you access to a message queue that can be used to store messages while waiting for a computer to process them. This allows you to quickly build message queuing applications that can be run on any computer on the internet. Since Amazon SQS is highly scalable and you only pay for what you use, you can start small and grow your application as you wish, with no compromise on performance or reliability. This lets you focus on building sophisticated message-based applications, without worrying about how the messages are stored and managed. You can use Amazon SQS with software applications in various ways. For example, you can:
- Integrate Amazon SQS with other AWS infrastructure web services to make applications more reliable and flexible.
- Use Amazon SQS to create a queue of work where each message is a task that needs to be completed by a process. One or many computers can read tasks from the queue and perform them.
- Build a microservices architecture, using queues to connect your microservices.
- Keep notifications of significant events in a business process in an Amazon SQS queue. Each event can have a corresponding message in a queue, and applications that need to be aware of the event can read and process the messages.
Q: How can I get started using Amazon SQS?
You must have an Amazon Web Services account to access this service. After signing up, please visit the AWS Management Console or refer to the Amazon SQS documentation and sample code in the Resource Center to begin using Amazon SQS.
Q: What are the benefits of Amazon SQS vs. homegrown or packaged message queuing systems?
Using Amazon SQS provides several advantages over building your own software for managing message queues or using a commercial or open-source message queuing system. The alternatives require significant upfront time to develop and configure compared to integrating with an existing web service. Also, these alternatives require ongoing hardware and system administrative resources to operate. The complexity of configuring and managing these systems is compounded if they need to be configured to provide redundant storage of messages so messages are not lost if hardware fails. By contrast, Amazon SQS requires no administrative overhead and little configuration. Additionally, Amazon SQS provides massive scale, processing billions of messages per day. You can scale up or down the amount of traffic you send to SQS without any configuration or management. Finally, Amazon SQS provides extremely high message durability - giving you and your stakeholders added confidence.
Q: When should I use Amazon Simple Workflow (SWF) vs Amazon SQS?
Both Amazon SWF and Amazon SQS can be used to develop distributed, decoupled applications. Amazon SWF provides an infrastructure that is designed for coordinating tasks when building highly scalable and auditable applications. Amazon SQS, on the other hand, provides a reliable, highly scalable, hosted queue for storing messages. While you may use Amazon SQS to build the messaging support needed to implement your distributed application, you get this facility out-of-the-box with Amazon SWF together with other application-level capabilities. We recommend looking at both Amazon SQS and Amazon SWF to determine which solution best fits your needs.
Q: Does Amazon use Amazon SQS for its own applications?
Yes. Developers within Amazon use Amazon SQS for a wide variety of projects and represent a large number of Amazon SQS messages per day. Applications using Amazon SQS include key business processes for the Amazon.com retail web site and Amazon Web Services.
Q: What can I do with the Amazon SQS free tier?
The Amazon SQS free tier provides 1 million requests per month at no charge. Many small scale applications may be able to operate entirely within this free tier limit. Data transfer charges may still apply (see Pricing). The free tier is a monthly offer. Free usage does not accumulate across months.
Q: How much does Amazon SQS cost?
You pay only for what you consume, and there is no minimum fee. You pay $0.50 for every 1 million requests, plus data transfer charges for data transferred out of Amazon SQS. Please see the Amazon SQS Pricing page for more detail.
Q: Do Amazon SQS batch operations cost more than other requests?
Batch operations which include; SendMessageBatch, DeleteMessageBatch and ChangeMessageVisbilityBatch all cost the same as other Amazon SQS requests. By grouping messages into batches, developers can reduce their cost.
Q: How will I be charged and billed for my use of Amazon SQS?
There are no set-up fees to begin using the service. At the end of the month, your credit card will automatically be charged for that month’s usage. You can view your charges for the current billing period at any time on the Amazon Web Services web site by logging into your Amazon Web Services account and clicking “Account Activity” under “Your Web Services Account”.
Q: Do your prices include taxes?
Except as otherwise noted, our prices are exclusive of applicable taxes and duties, including VAT and applicable sales tax. For customers with a Japanese billing address, use of AWS (in any region) is subject to Japanese Consumption Tax. Learn more.
Q: Can Amazon SQS be used with other AWS services?
Amazon SQS can be used with compute services such as Amazon EC2, Amazon EC2 Container Service and AWS Lambda, as well as storage and database services such as Amazon S3 and Amazon DynamoDB, to make applications more flexible and scalable. A common use case is to create a distributed, decoupled application, where multiple components or modules need to communicate with each other, but can’t all process the same amount of work simultaneously. In this case, Amazon SQS queues carry messages to be processed in an orderly fashion by the user’s application running on Amazon EC2 instances. The Amazon EC2 instances can read the queue, process the job, and then post the results as messages to another Amazon SQS queue (possibly for further processing by another application). Because Amazon EC2 allows applications to scale up and down dynamically, application developers can easily vary the number of compute instances based on the amount of work in the SQS queues, to ensure that jobs are executed in a timely manner.
For example, here is how a video transcoding website uses Amazon EC2, Amazon SQS, Amazon S3, and Amazon DynamoDB together. End users submit videos to be transcoded to the website. The videos are stored in Amazon S3, and a message (“the request message”) is placed in an Amazon SQS queue (“the incoming queue”) with a pointer to the video and to the target video format in the message. The transcoding engine, running on a set of Amazon EC2 instances, reads the request message from the incoming queue, retrieves the video from Amazon S3 using the pointer, and transcodes the video into the target format. The converted video is put back into Amazon S3 and another message (“the response message”) is placed in another Amazon SQS queue (“the outgoing queue”) with a pointer to the converted video. At the same time, metadata about the video (e.g. format, date created and length) can be indexed into Amazon DynamoDB for easy querying. During this whole workflow, a dedicated Amazon EC2 instance can constantly monitor the incoming queue and, based on the number of messages in the incoming queue, is able to dynamically adjust the number of transcoding Amazon EC2 instances to meet customers’ response time requirements.
Q: How do I interface with Amazon SQS?
Amazon SQS provides simple APIs that are designed to work with any Internet-development toolkit. The operations are intentionally made simple to work with messages and queues.
Q: What are the available operations for queues?
The operations available are listed on the Product Details page.
Q: Who can perform operations on a queue?
Operations can be performed only by the AWS account owner or an AWS account that the account owner has delegated to.
Q: Can I use Java Message Service (JMS) with Amazon SQS?
Yes. Amazon provides a client library that implements the JMS 1.1 specification and uses Amazon SQS as the JMS provider. You can count on the scale, low cost, and high availability of SQS without the worry and overhead of running your own JMS cluster.
Q: How are messages identified in the system?
All messages have a globally unique ID that Amazon SQS returns when the message is delivered to the queue. The ID isn’t required in order to perform any further actions on the message, but it’s useful for tracking whether a particular message in the queue has been received. When you receive a message from the queue, the response includes a receipt handle, which you must provide when deleting the message.
Q: How are unsuccessfully processed messages handled?
Amazon SQS provides the ability to configure Dead Letter Queues (DLQs). A DLQ is a queue which you configure to receive messages from other queues - referred to as “source queues.” Typically, you set up a DLQ to receive messages after a max number of processing atttempts have been reached. DLQs provides the ability to isolate messages that could not be processed for later analysis.
A DLQ is just like any other queue. Messages can be sent to it and received from it like any other queues. You can create a DLQ from either the SQS API or the SQS console.
You can learn about how to create and use DLQs from the SQS Developer Guide.
Q: Does Amazon SQS provide first-in-first-out (FIFO) access to messages?
Amazon SQS makes a best effort to preserve the order of messages, but due to the distributed nature of the queue, we cannot guarantee you will receive messages in the exact order you sent them (FIFO). If your system requires that order be preserved, we recommend you place sequencing information in each message so you can reorder the messages upon receipt.
Q: Does Amazon SQS provide "at least once" delivery of messages?
Yes. Amazon SQS stores copies of your messages on multiple servers for redundancy and high availability. On rare occasions, one of the servers storing a copy of a message might be unavailable when you receive or delete the message. If that occurs, the copy of the message will not be deleted on that unavailable server, and you might get that message copy again when you receive messages. This is known as "at least once" delivery. Because of this, you must design your application to be idempotent (i.e. it must not be adversely affected if it processes the same message more than once).
Q: What is the maximum limit for message visibility?
Amazon SQS supports up to 12 hours maximum visibility timeout.
Q: Does SQS support message metadata?
Yes. Amazon SQS allows you to send up to 10 attributes on each message. With message attributes, you can separate the body of a message from the metadata that describes it. This allows you to process and store information with greater speed and intelligence because your applications no longer have to inspect an entire message before understanding what processing steps are necessary.
SQS message attributes take the form of name-type-value triples. Types supported include: String, Binary and Number (including integers, floating point, and doubles).
Please refer to the SQS Documentation for additional information.
Q: How can a developer determine “time-in-queue”?
Developers can request the “SentTimestamp” attribute when receiving a message. Subtracting that value from the current time results in the “time-in-queue” value.
Q: What is the typical latency for SQS?
Typical latencies for SendMessage, ReceiveMessage and DeleteMessage API requests are in the tens or low hundreds of milliseconds.
Q: What is the “SenderId” attribute value of a message in the case of anonymous access?
Amazon SQS provides the IP address when the AWS account ID is not available such as when an anonymous user sends a message.
Q: What is SQS Long Polling?
SQS long polling is a way to retrieve messages from your SQS queues. While the regular SQS short polling returns immediately, even if the queue being polled is empty, SQS long polling doesn’t return a response until a message arrives in the queue, or the long poll times out. SQS long polling makes it easy and inexpensive to retrieve messages from your SQS queue as soon as they are available. Using long polling may reduce the cost of using SQS, as you can reduce the number of empty receives.
Q: Is there any additional charge for using SQS long polling?
No, long polling ReceiveMessage API calls are billed exactly the same as short polling ReceiveMessage API calls.
Q: When should I use SQS long polling, and when should I use SQS short polling?
In almost all cases, SQS long polling is preferable to SQS short polling. Long polling requests allow your queue consumers to receive messages as soon as they arrive in your queue, while reducing the number of empty ReceiveMessageResponses you encounter. Combined together, SQS long polling results in higher performance at reduced cost for the majority of use cases. However, if your application is written to expect an immediate response from a ReceiveMessage call, you may not be able to take advantage of long polling without some application modifications. For example, if your application has a single thread polling multiple queues, switching from short polling to long polling will likely not work, as the single thread will wait for the long poll timeout on any empty queues, delaying the processing of any queues which may contain messages. In such an application, it is recommended that a single thread be used to process only one queue, allowing for the application to take advantage of the benefits SQS long polling provides.
Q: What value should I use for my long polling timeout?
In general, the maximum long poll timeout of 20 seconds should be used. Higher long poll timeout values will reduce the number of empty ReceiveMessageResponses returned, so try to set your long poll timeout as high as possible. If the 20 second maximum does not work for your application (see example in previous question), you can choose a shorter long poll timeout, down to as low as 1 second. All of the AWS SDK’s work with 20 second long polls by default. If you are not using an AWS SDK to access SQS, or if you’ve specially configured your AWS SDK to have a shorter timeout, you may need to modify your SQS client to allow for longer requests or use a shorter long poll timeout.
Q: What is the AmazonSQSBufferedAsync client ?
The AmazonSQSBufferedAsync client provides an implementation for the AmazonSQSAsync client interface that adds several important features. First, the AmazonSQSBufferedAsync client supports automatic batching of multiple SendMessage, DeleteMessage or ChangeMessageVisibility requests into batches of each type, without any changes required by the application. Second, the AmazonSQSBufferedAsync client supports prefetching of messages into a local buffer, which allows your application to immediately process messages from SQS without waiting for them to be retrieved from SQS. Combined together, automatic batching and prefetching increases the throughput and reduces the latency of your application, while saving you money by making fewer SQS requests.
Q: How do I download the AmazonSQSBufferedAsync client?
You can download the AmazonSQSBufferedAsync client as part of the AWS SDK for Java, available at http://aws.amazon.com/sdkforjava/.
Q: Which languages does the AmazonSQSBufferedAsyncClient support?
At this time, the AmazonSQSBufferedAsyncClient is only supported for Java.
Q: Do I have to rewrite my application to use the AmazonSQSBufferedAsync client?
No, the AmazonSQSBufferedAsync client is implemented as a drop in replacement for the existing AmazonSQSAsync client. You can update your application to use the latest AWS SDK, change your client construction to use the AmazonSQSBufferedAsync client instead of the AmazonSQSAsync client, and your application will receive the added benefits of automatic batching and prefetching.
Q: How can I subscribe SQS queues to receive notifications from SNS topics?
Select the SQS queue in the SQS console, and from the ‘Queue Actions’ in the menu bar, select ‘Subscribe Queue to SNS Topic’ from the drop-down list. In the subscribe dialog box, select the topic from the ‘Choose a Topic’ drop-down list, and click the ‘Subscribe’ button. For complete step-by-step instructions, please refer to the Amazon SQS documentation.
Q: How can I fanout identical messages to multiple SQS queues?
Create an SNS topic first using SNS. Then create and subscribe multiple SQS queues to the SNS topic. Now whenever a message is sent to the SNS topic, the message will be fanned out to the SQS queues, i.e. SNS will deliver the message to all the SQS queues that are subscribed to the topic.
Q: Can I delete all the messages in a queue without deleting the queue itself?
Yes, you can delete all the messages in an SQS queue using the PurgeQueue action. When you purge a queue, all the messages previously sent to the queue will be deleted. Since your queue and its attributes will remain, there is no need to reconfigure the queue to continue using it. If you only need to delete specific messages, you can use the DeleteMessage or DeleteMessageBatch actions.
Q: How reliably is my data stored in Amazon SQS?
Amazon SQS stores all queue and message information in Amazon’s network of highly reliable, highly available data centers. All messages are stored redundantly on multiple servers and in multiple data centers, which means that no single computer or network failure renders SQS messages inaccessible.
Q: How can I secure the messages in my queues?
Authentication mechanisms are provided to ensure that messages stored in Amazon SQS are secured against unauthorized access. You can control who can send messages to a queue, and who can receive messages from a queue.
Amazon SQS has its own resource-based permissions system that uses policies written in the same language used for AWS Identity and Access Management (IAM) policies. This means that you can achieve the same things with Amazon SQS policies that you can with IAM policies, such as using variables in IAM policies. For the details of how to use this, please see the Amazon SQS Developer Guide.
For additional security, you can build your application to encrypt messages before they are placed in a queue.
Q: How does Amazon SQS allow multiple readers to access the same message queue, without losing messages or processing them many times?
Every Amazon SQS queue has a configurable visibility timeout. For the designated amount of time after a message is read from a queue, it will not be visible to any other reader. As long as the amount of time that it takes to process the message is less than the visibility timeout, every message will be processed and deleted. In the event that the component processing the message fails or becomes unavailable, the message will again become visible to any component reading the queue once the visibility timeout ends. This allows you to have many components all reading messages from the same queue, with each working to process different messages.
Q: Why are there separate ReceiveMessage and DeleteMessage operations?
When Amazon SQS returns a message to you, that message stays in the queue, whether or not you actually received the message. You are responsible for deleting the message; the delete request acknowledges that you’re done processing the message. If you don’t delete the message, Amazon SQS will deliver it again on another receive request. See "Visibility Timeout" in the Amazon SQS Developer Guide for more information.
Q: Can a deleted message be received again?
Yes, under rare circumstances you might receive a previously deleted message again. This can occur in the rare situation in which a DeleteMessage operation doesn’t delete all copies of a message because one of the servers in the distributed Amazon SQS system isn’t available at the time of the deletion. That message copy can then be delivered again. You should design your application so that no errors or inconsistencies occur if you receive a deleted message again.
Q: What happens if I issue a DeleteMessage request on a previously deleted message?
SQS returns a “success” response.
Q: Is Amazon SQS PCI DSS certified?
Amazon SQS is PCI DSS Level 1 certified. See http://aws.amazon.com/compliance/pci-dss-level-1-faqs for more details.
Q: Is Amazon SQS HIPAA compliant?
Amazon SQS is not certified for HIPAA compliance. However, you can utlize the Extended Client Library to send SQS message payloads via S3. S3 is HIPAA compliant. You can achieve HIPAA compliance in this manner, since no PII is transfered via SQS.
Q: How long can I keep my messages in Amazon SQS queues?
The SQS message retention period is configurable and can be set anywhere from 1 minute to 2 weeks. The default is 4 days and once the message retention limit is reached your messages will be automatically deleted. The option for longer message retention provides greater flexibility to allow for longer intervals between message production and consumption.
Q: How do I configure SQS to support longer message retention?
To configure the message retention period, set the MessageRetentionPeriod attribute using the Management Console or using the SetQueueAttributes method. This attribute is used to specify the number of seconds a message will be retained by SQS. Currently the default value for the message retention period is 4 days. Using the MessageRetentionPeriod attribute, the message retention period can be set anywhere from 60 seconds (1 minute), up to 1209600 seconds (14 days). Please consult the Amazon SQS API Reference Guide for more detail about how to work with this message attribute.
Q: How do I configure the maximum message size for SQS?
To configure the maximum message size, set the MaximumMessageSize attribute using the Management Console or the SetQueueAttributes method. This attribute specifies the limit on how many bytes an SQS message can contain. It can be set anywhere from 1024 bytes (1KB), up to 262144 bytes (256KB). Please consult the Amazon SQS API Reference Guide for more detail about how to work with this message attribute.
To send messages larger than 256KB, you can use the Amazon SQS Extended Client Library for Java. This library enables you to send a message in SQS that contains a reference to a message payload in Amazon S3. The message payload in S3 can be as large as 2GB.
Q: What kind of data can go in a message?
Amazon SQS messages can contain up to 256KB of text data, including XML, JSON and unformatted text. The following Unicode characters are accepted:
#x9 | #xA | #xD | [#x20 to #xD7FF] | [#xE000 to #xFFFD] | [#x10000 to #x10FFFF]
(according to http://www.w3.org/TR/REC-xml/#charsets).
Q: How big can Amazon SQS queues be?
A single queue may contain an unlimited number of messages, and you can create any number of queues.
Q: Are there restrictions on the names of Amazon SQS queues? Is there a size limit on the name of Amazon SQS queues? Can a queue name be reused?
Queue names are limited to 80 characters. Alphanumeric characters plus hyphens (-) and underscores (_) are allowed. Queue names must be unique within an AWS account and region. After you delete a queue, you can reuse the queue name.
Q: What happens if there is no activity against a queue for an extended period of time?
We reserve the right to delete a queue if none of the following requests have been issued against the queue for more than 30 consecutive days: SendMessage, ReceiveMessage, DeleteMessage, GetQueueAttributes and SetQueueAttributes. You should design your application with this in mind. Note that queues that are used as Dead Letter Queues will not be deleted as long as any of their source queues still exist.
Q: How do I share a queue?
A developer associates an access policy statement (specifying the permissions being granted) with the queue to be shared. Amazon SQS provides APIs to create and manage the access policy statements: AddPermission, RemovePermission, SetQueueAttributes and GetQueueAttributes. Refer to the latest API specification for more details.
Q: Who pays for shared queue access?
The queue owner pays for shared queue access.
Q: How do I identify another AWS user?
The Amazon SQS API uses the AWS account number to identify AWS users.
Q: What do I need to provide other users in order to share a queue with them?
You’ll need to provide the full URL from the queue to share. The CreateQueue and ListQueues operations return this URL in their response.
Q: Does Amazon SQS support Anonymous access?
Yes – a developer can set an access policy that allows anonymous users access to a queue.
Q: When would I use the Permissions API?
The Permissions API provides a simple interface for developers to share access to a queue but it cannot allow for conditional access and more advanced use cases.
Q: When would I use SetQueueAttributes with JSON objects?
The SetQueueAttributes operation supports the full access policy language. Using the policy language, access to a queue can be restricted by IP address and time of day for instance. Refer to the access policy documentation in the latest Developer Guide for more details.
Q: What regions is Amazon SQS available in?
Please refer to the AWS Global Infrastructure Region Table.
Q: Can messages be shared between queues in different regions?
No – Amazon SQS in each region is totally independent in message stores and queue names.
Q: Is there a pricing difference between regions?
Amazon SQS pricing is the same for all regions, except Asia Pacific (Tokyo). See Pricing for more details.
Q: What is the pricing structure between various regions?
Data transferred between Amazon SQS and Amazon EC2 within a single region is free of charge. Data transferred between Amazon SQS and Amazon EC2 in different regions will be charged at the normal data transfer rate.