Q: What is Amazon SQS?
Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable hosted queue for storing messages as they travel between computers. By using Amazon SQS, developers can simply move data between distributed application components performing different tasks, without losing messages or requiring each component to be always available. Amazon SQS makes it easy to build an automated workflow, working in close conjunction with the Amazon Elastic Compute Cloud (Amazon EC2) and the other AWS infrastructure web services.
Q: What can I do with Amazon SQS?
Amazon SQS is a web service that gives you access to a message queue that can be used to store messages while waiting for a computer to process them. This allows you to quickly build message queuing applications that can be run on any computer on the internet. Since Amazon SQS is highly scalable and you only pay for what you use, you can start small and grow your application as you wish, with no compromise on performance or reliability. This lets you focus on building sophisticated message-based applications, without worrying about how the messages are stored and managed. You can use Amazon SQS with software applications in various ways. For example, you can:
- Integrate Amazon SQS with other AWS infrastructure web services to make applications more reliable and flexible.
- Use an Amazon SQS queue as a queue of work where each message is a task that needs to be completed by a process. One or many computers can read tasks from the queue and perform them.
- Have Amazon SQS help a browser-based application receive notifications from a server. The application server can add the notifications to a queue, which the browser can poll—even if there is a firewall between them.
- Keep notifications of significant events in a business process in an Amazon SQS queue. Each event can have a corresponding message in a queue, and applications that need to be aware of the event can read and process the messages.
Q: How can I get started using Amazon SQS?
To sign up for Amazon SQS, click the “Sign up for This Web Service” button on the Amazon SQS detail page. You must have an Amazon Web Services account to access this service; if you do not already have one, you will be prompted to create one when you begin the Amazon SQS sign-up process. After signing up, please visit the AWS Management Console or refer to the Amazon SQS documentation and sample code in the Resource Center to begin using Amazon SQS.
Q: What are the benefits of Amazon SQS vs. homegrown or packaged message queuing systems?
Using Amazon SQS provides several advantages over building your own software for managing message queues or using a commercial or open-source message queuing system. The alternatives require significant upfront time to develop and configure compared to integrating with an existing web service. Also, these alternatives require ongoing hardware and system administrative resources to operate. The complexity of configuring and managing these systems is compounded if they need to be configured to provide redundant storage of messages so messages are not lost if hardware fails. By contrast, Amazon SQS requires no administrative overhead and little configuration.
Q: Does Amazon use Amazon SQS for its own applications?
Yes. Developers within Amazon use Amazon SQS for a wide variety of projects and represent a large number of Amazon SQS messages per day. Applications using Amazon SQS include key business processes for the Amazon.com retail web site and Amazon Web Services.
Q: What can I do with the Amazon SQS free tier?
The Amazon SQS free tier provides 1 million requests per month at no charge. Many small scale applications may be able to operate entirely within this free tier limit. Data transfer charges still apply. The free tier is a monthly offer. Free usage does not accumulate across months.
Q: How much does Amazon SQS cost?
You pay only for what you consume, and there is no minimum fee. You pay $0.50 for every 1 million requests, plus data transfer charges for data transferred out of Amazon SQS. Data transferred between Amazon SQS and Amazon EC2 in the U.S. is free of charge. Please see the SQS pricing page for more detail.
Q: Do Amazon SQS batch operations cost more than other requests?
Batch operations which include; SendMessageBatch, DeleteMessageBatch and ChangeMessageVisbilityBatch all cost the same as other Amazon SQS requests.
Q: How will I be charged and billed for my use of Amazon SQS?
There are no set-up fees to begin using the service. At the end of the month, your credit card will automatically be charged for that month’s usage. You can view your charges for the current billing period at any time on the Amazon Web Services web site by logging into your Amazon Web Services account and clicking “Account Activity” under “Your Web Services Account”.
Q: Do your prices include taxes?
Except as otherwise noted, our prices are exclusive of applicable taxes and duties, such as VAT and applicable sales tax. Our prices for the Asia Pacific (Tokyo) Region are inclusive of Japan consumption tax.
Q: Can Amazon SQS be used with other AWS services?
Amazon SQS can be used with Amazon EC2, as well as Amazon S3 and Amazon SimpleDB, to make applications more flexible and scalable. A common use case is to create an integrated and automated workflow, where multiple components or modules need to communicate with each other, but can’t all process the same amount of work simultaneously. In this case, Amazon SQS queues carry messages to be processed in an orderly fashion by the user’s application running on Amazon EC2 instances. The Amazon EC2 instances can read the queue, process the job, and then post the results as messages to another Amazon SQS queue (possibly for further processing by another application). Because Amazon EC2 allows applications to scale up and down dynamically, application developers can easily vary the number of compute instances based on the amount of work in the SQS queues, to ensure that jobs are executed in a timely manner.
For example, here is how a video transcoding website uses Amazon EC2, Amazon SQS, Amazon S3, and Amazon SimpleDB together. End users submit videos to be transcoded to the website. The videos are stored in Amazon S3, and a message (“the request message”) is placed in an Amazon SQS queue (“the incoming queue”) with a pointer to the video and to the target video format in the message. The transcoding engine, running on a set of Amazon EC2 instances, reads the request message from the incoming queue, retrieves the video from Amazon S3 using the pointer, and transcodes the video into the target format. The converted video is put back into Amazon S3 and another message (“the response message”) is placed in another Amazon SQS queue (“the outgoing queue”) with a pointer to the converted video. At the same time, metadata about the video (e.g. format, date created and length) can be indexed into Amazon SimpleDB for easy querying. During this whole workflow, a dedicated Amazon EC2 instance can constantly monitor the incoming queue and, based on the number of messages in the incoming queue, is able to dynamically adjust the number of transcoding Amazon EC2 instances to meet customers’ response time requirements.
Q: How do I interface with Amazon SQS?
Amazon SQS provides simple, standards-based SOAP and Query web services interfaces that are designed to work with any Internet-development toolkit. The operations are intentionally made simple to work with messages and queues.
Q: What are the available operations for queues?
Who can perform operations on a queue? The following set of operations is defined for Amazon SQS: CreateQueue, ListQueues, DeleteQueue, SendMessage, SendMessageBatch, ReceiveMessage, DeleteMessage, DeleteMessageBatch, ChangeMessageVisibility, ChangeMessageVisibilityBatch, SetQueueAttributes, GetQueueAttributes, GetQueueUrl, AddPermission and RemovePermission. Operations can be performed only by the AWS account owner or an AWS account that the account owner has delegated to.
Q: How are messages identified in the system?
All messages have a globally unique ID that Amazon SQS returns when the message is delivered to the queue. The ID isn’t required in order to perform any further actions on the message, but it’s useful for tracking whether a particular message in the queue has been received. When you receive a message from the queue, the response includes a receipt handle, which you must provide when deleting the message.
Q: Does Amazon SQS offer Dead Letter Queues (DLQs) ?
Yes. A DLQ is an SQS queue which you configure to receive messages from other SQS queues - referred to as “source queues.” Typically, you set up a DLQ to receive messages after a max number of processing atttempts have been reached. DLQ provides the ability to isolate messages that could not be processed for analysis out of band.
A DLQ is just like any other SQS queue. Messages can be sent to it and received from it like any other SQS queues. You can create a DLQ from the SQS API and the SQS console.
You can learn about how to create and use Dead Letter Queues from the SQS Developer Guide.
Q: Does Amazon SQS provide first-in-first-out (FIFO) access to messages?
No, Amazon SQS does not guarantee FIFO access to messages in Amazon SQS queues, mainly because of the distributed nature of the Amazon SQS. If you require specific message ordering, you should design your application to handle it.
Q: What is the maximum limit for message visibility?
Amazon SQS supports up to 12 hours maximum visibility timeout for the 2009 and 2011 WSDLs.
Q: What new attributes are supported?
Amazon SQS now supports additional attributes such as when a message was sent and who sent the message. Please refer to the lateast Amazon SQS documentation for all of the attributes supported.
Q: How can a developer determine “time-in-queue”?
Developers can request the “SentTimestamp” attribute when receiving a message. Subtracting that value from the current time results in the “time-in-queue” value.
Q: What is the “SenderId” attribute value of a message in the case of anonymous access?
Amazon SQS provides the IP address when the AWS account ID is not available such as when an anonymous user sends a message.
Q: What is SQS Long Polling?
SQS long polling is a new way to retrieve messages from your SQS queues. While the traditional SQS short polling returns immediately, even if the queue being polled is empty, SQS long polling doesn’t return a response until a message arrives in the queue, or the long poll times out. SQS long polling makes it easy and inexpensive to retrieve messages from your SQS queue as soon as they are available.
Q: Is there any additional charge for using SQS long polling?
No, long polling ReceiveMessage API calls are billed exactly the same as short polling ReceiveMessage API calls.
Q: When should I use SQS long polling, and when should I use SQS short polling?
In almost all cases, SQS long polling is preferable to SQS short polling. Long polling requests allow your queue consumers to receive messages as soon as they arrive in your queue, while reducing the number of empty ReceiveMessageResponses you encounter. Combined together, SQS long polling results in higher performance at reduced cost for the majority of use cases. However, if your application is written to expect an immediate response from a ReceiveMessage call, you may not be able to take advantage of long polling without some application modifications. For example, if your application has a single thread polling multiple queues, switching from short polling to long polling will likely not work, as the single thread will wait for the long poll timeout on any empty queues, delaying the processing of any queues which may contain messages. In such an application, it is recommended that a single thread be used to process only one queue, allowing for the application to take advantage of the benefits SQS long polling provides.
Q: What value should I use for my long polling timeout?
In general, the maximum long poll timeout of 20 seconds should be used. Higher long poll timeout values will reduce the number of empty ReceiveMessageResponses returned, so try to set your long poll timeout as high as possible. If the 20 second maximum does not work for your application, you can choose a shorter long poll timeout, down to as low as 1 second. All of the AWS SDK’s work with 20 second long polls by default. If you are not using an AWS SDK to access SQS, or if you’ve specially configured your AWS SDK to have a shorter timeout, you may need to modify your SQS client to allow for longer requests or use a shorter long poll timeout.
Q: What is the AmazonSQSBufferedAsync client ?
The AmazonSQSBufferedAsync client provides a new implementation for the AmazonSQSAsync client interface that adds several important features. First, the AmazonSQSBufferedAsync client supports automatic batching of multiple SendMessage, DeleteMessage or ChangeMessageVisibility requests into batches of each type, without any changes required by the application. Second, the AmazonSQSBufferedAsync client supports prefetching of messages into a local buffer, which allows your application to immediately process messages from SQS without waiting for them to be retrieved from SQS. Combined together, automatic batching and prefetching increases the throughput and reduces the latency of your application, while saving you money by making fewer SQS requests.
Q: How do I download the AmazonSQSBufferedAsync client?
You can download the AmazonSQSBufferedAsync client as part of the most recent AWS SDK for java, available at http://aws.amazon.com/sdkforjava/.
Q: Which languages does the AmazonSQSBufferedAsyncClient support?
At this time, the AmazonSQSBufferedAsyncClient is only supported for Java. We will be releasing the AmazonSQSBufferedAsyncClient in other languages in the near future.
Q: Do I have to rewrite my application to use the AmazonSQSBufferedAsync client?
No, the AmazonSQSBufferedAsync client is implemented as a drop in replacement for the existing AmazonSQSAsync client. You can update your application to use the latest AWS SDK, change your client construction to use the AmazonSQSBufferedAsync client instead of the AmazonSQSAsync client, and your application will receive the added benefits of automatic batching and prefetching.
Q: How can I subscribe SQS queues to receive notifications from SNS topics?
Select the SQS queue in the SQS console, and from the ‘Queue Actions’ in the menu bar, select ‘Subscribe Queue to SNS Topic’ from the drop-down list. In the subscribe dialog box, select the topic from the ‘Choose a Topic’ drop-down list, and click the ‘Subscribe’ button. For complete step-by-step instructions, please refer to the Amazon SQS documentation.
Q: How can I fanout identical messages to multiple SQS queues?
Create an SNS topic first using SNS. Then create and subscribe multiple SQS queues to the SNS topic. Now whenever a message is sent to the SNS topic, the message will be fanned out to the SQS queues, i.e. SNS will deliver the message to all the SQS queues that are subscribed to the topic.
Q: How reliably is my data stored in Amazon SQS?
Amazon SQS stores all queue and message information in Amazon’s network of highly reliable, highly available data centers. All messages are stored redundantly on multiple servers and in multiple data centers, which means that no single computer or network failure renders SQS messages inaccessible.
Q: How can I secure the messages in my queues?
Authentication mechanisms are provided to ensure that messages stored in Amazon SQS queues are secured against unauthorized access. Only the AWS account owners can access the queues they create.
Amazon SQS uses proven cryptographic methods to authenticate your identity, either through the use of your Access Key ID and request signature, or through the use of an X.509 certificate. For the details of how to use either of these authentication mechanisms with Amazon SQS, please see the Amazon SQS Developer Guide.
For additional security, you could build your application to encrypt messages before they are placed in a queue.
Q: How does Amazon SQS allow multiple readers to access the same message queue, without losing messages or processing them many times?
Every Amazon SQS queue has a configurable visibility timeout. For the designated amount of time after a message is read from a queue, it will not be visible to any other reader. As long as the amount of time that it takes to process the message is less than the visibility timeout, every message will be processed and deleted. In the event that the component processing the message fails or becomes unavailable, the message will again become visible to any component reading the queue once the visibility timeout ends. This allows you to have many components all reading messages from the same queue, with each working to process different messages.
Q: How many times will I receive each message?
Amazon SQS is engineered to provide “at least once” delivery of all messages in its queues. Although most of the time each message will be delivered to your application exactly once, you should design your system so that processing a message more than once does not create any errors or inconsistencies.
Q: Why are there separate ReceiveMessage and DeleteMessage operations?
When Amazon SQS returns a message to you, that message stays in the queue, whether or not you actually received the message. You are responsible for deleting the message; the delete request acknowledges that you’re done processing the message. If you don’t delete the message, Amazon SQS will deliver it again on another receive request.
Q: Can a deleted message be received again?
Yes, under rare circumstances you might receive a previously deleted message again. This can occur in the rare situation in which a DeleteMessage operation doesn’t delete all copies of a message because one of the servers in the distributed Amazon SQS system isn’t available at the time of the deletion. That message copy can then be delivered again. You should design your application so that no errors or inconsistencies occur if you receive a deleted message again.
Q: What happens if I issue a DeleteMessage request on a previously deleted message?
SQS returns a “success” response.
Q: How long can I keep my messages in Amazon SQS queues?
The SQS message retention period is configurable and can be set anywhere from 1 minute to 2 weeks. The default is 4 days and once the message retention limit is reached your messages will be automatically deleted. The option for longer message retention provides greater flexibility to allow for longer intervals between message production and consumption.
Q: How do I configure SQS to support longer message retention?
To configure the message retention period, set the MessageRetentionPeriod attribute using the SetQueueAttributes method. This attribute is used to specify the number of seconds a message will be retained by SQS. Currently the default value for the message retention period is 4 days. Using the MessageRetentionPeriod attribute, the message retention period can be set anywhere from 60 seconds (1 minute), up to 1209600 seconds (14 days). Please consult the Amazon SQS API Reference Guide for more detail about how to work with this new message attributes.
Q: How do I configure SQS to support larger message sizes?
To configure the maximum message size, set the MaximumMessageSize attribute using the SetQueueAttributes method. This attribute specifies the limit on how many bytes an SQS message can contain. It can be set anywhere from 1024 bytes (1kB), up to 65536 bytes (64kB). Please consult the Amazon SQS API Reference Guide for more detail about how to work with this new message attributes.
Q: What kind of data can go in a message?
Amazon SQS messages can contain up to 64 KB of text data, including XML, JSON and unformatted text. The following Unicode characters are accepted:
#x9 | #xA | #xD | [#x20 to #xD7FF] | [#xE000 to #xFFFD] | [#x10000 to #x10FFFF]
(according to http://www.w3.org/TR/REC-xml/#charsets).
Q: How big can Amazon SQS queues be?
A single queue may contain an unlimited number of messages, and you can create an unlimited number of queues.
Q: Are there restrictions on the names of Amazon SQS queues?
Is there a size limit on the name of Amazon SQS queues? Can a queue name be reused? Queue names are limited to 80 characters. Alphanumeric characters plus hyphens (-) and underscores (_) are allowed. Queue names must be unique within an AWS account. After you delete a queue, you can reuse the queue name.
Q: What happens if there is no activity against a queue for an extended period of time?
We reserve the right to delete a queue if none of the following requests have been issued against the queue for more than 30 consecutive days: SendMessage, ReceiveMessage, DeleteMessage, GetQueueAttributes and SetQueueAttributes. You should design your application with this in mind.
Q: How do I share a queue?
A developer associates an access policy statement (specifying the permissions being granted) with the queue to be shared. Amazon SQS provides APIs to create and manage the access policy statements: AddPermission, RemovePermission, SetQueueAttributes and GetQueueAttributes. Refer to the latest API specification for more details.
Q: Who pays for shared queue access?
The queue owner pays for shared queue access.
Q: How do I identify another AWS user?
The Amazon SQS API uses the AWS account number to identify AWS users.
Q: What do I need to provide other users in order to share a queue with them?
You’ll need to provide the full URL from the queue to share. The CreateQueue and ListQueues operations return this URL in their response.
Q: Does Amazon SQS support Anonymous access?
Yes – with the WSDL 2009 and later APIs, a developer can set an access policy that allows anonymous users access to a queue.
Q: When would I use the Permissions API?
The Permissions API provides a simple interface for developers to share access to a queue but it cannot allow for conditional access and more advanced use cases.
Q: When would I use SetQueueAttributes with JSON objects?
The SetQueueAttributes operation supports the full access policy language. Using the policy language, access to a queue can be restricted by IP address and time of day for instance. Refer to the access policy documentation in the latest Developer Guide for more details.
Q: What are the Amazon SQS service access points in each region?
The US East (Northern Virginia) end-point is: http://sqs.us-east-1.amazonaws.com
The US West (Oregon) end-point is: http://sqs.us-west-2.amazonaws.com
The US West (Northern California) end-point is: http://sqs.us-west-1.amazonaws.com
The EU (Ireland) end-point is: http://sqs.eu-west-1.amazonaws.com
The Asia Pacific (Singapore) end-point is: http://sqs.ap-southeast-1.amazonaws.com
The Asia Pacific (Tokyo) end-point is: http://sqs.ap-northeast-1.amazonaws.com
The Asia Pacific (Sydney) end-point is: http://sqs.ap-southeast-2.amazonaws.com
The South America (Sao Paulo) end-point is: http://sqs.sa-east-1.amazonaws.com
Q: Can messages be shared between queues in different regions?
No – Amazon SQS in each region is totally independent in message stores and queue names.
Q: Is there a pricing difference between regions?
No – Amazon SQS pricing is the same for all 8 regions.
Q: What is the pricing structure between various regions?
Data transferred between Amazon SQS and Amazon EC2 within a single region is free of charge. Data transferred between Amazon SQS and Amazon EC2 in different regions will be charged at the normal data transfer rate.