SQS: Super Queue Service

There were some questions/concerns regarding the Amazon’s Simple Queue Service: What can it be used for? What has it been designed for? What types of applications can use this type of service ? and so forth.

Thomas Howe wrote an excellent post about Amazon SQS. His innovative application idea triggered me to write this post.

In this post, I will try to explain the power and potential of Amazon SQS. What makes it a “Super Queue Service” :) Hopefully, after reading this post, your “thought liquids” might be spurred enough that you will use the comments section to give me more ideas. Leverage the Power of Community!

First some theory and then we go into more juicy stuff
The key to building scalable systems is to have loosely coupled components in our architecture, components that do not have strong dependencies between each other, so that if one component were to die (fail), sleep (not respond) or remain busy (slow to respond) for some reason, the other components in the system continue to work seamlessly as if nothing has happened.

The key to building loosely coupled components in our architecture is Messaging Queues. Why? Because if a queue/buffer is used to “wire” any two components together, it can now abstract out and hide the load/demand thats coming in or going out from components. So if one component dies or becomes unavailable, it will buffer the messages and get them processed when the component comes back up.

Now think of Amazon SQS as a “transient buffer” between producers and consumers. On one side, there are these inhumane monstrous producers that are continuously shoving-in messages/requests/jobs and on the other side there are ravenous consumers eagerly waiting and ready to consume those jobs.  Lets analyze: What if producers get more inhumane or more producers join the gang? No problem! The Queue will expand on-demand and keep on collecting/buffering those messages. What if consumers/processors die or become unavailable? No Problem! Same answer. This is called Asynchronous Application Integration which basically means queuing leads to asynchrony which ultimately leads to a highly-scalable system.

If you are smart (like Amazonians ;)), you will probably want to use your consumers wisely to its full utilization and save some money because consumers are expensive resources. For example, Producers keep on producing messages continuously during the day and  Consumer “wakes up” for 2 hours a day, consumes/processes the messages/jobs and goes back to sleep. Or conversely, if you care for faster processing, simply add more hungry consumers. This is called Extensible/Scalable Integration which gives you the flexibility of adding/removing nodes because they are loosely coupled.

Now think of Amazon SQS as an “Unified Interface” between two components. Imagine if all the applications in the world had just two interfaces: send, receive and delete. And all the applications in the world interacts with other applications in these two interfaces (think big!, common) Because you can shove-in messages in any self-describing formats like XML/JSON, you only need to worry about is the “language” (Message format and What the message elements mean). This glues two different types of application frameworks that are running completely independent to one-another at their own will. This is called heterogeneous data integration.

Now think of Amazon SQS as a “Service Bus“. Queuing allows and encourages applications to become truly service-oriented. You will be automatically making the right architecture decision of making your applications service-oriented and independent to others. Moreover, imagine if consumers’ can be physically located anywhere in the world. Because queues are represented by a simple URI, Consumers can find queues dynamically and process the messages, independent of where they are located. For e.g. two consumers processing messages from US during US day time and other two consumers processing the same messages from Japan during Japan day time.

Now think of Amazon SQS as a “Workflow enabler“. Imagine if all the workflow systems like WWF/WCF, jBPM, openWFE, BizTalk, Agila have Amazon SQS plug-ins. They could simply talk to one another. In fact, smart developers have already built plug-in for OpenWFru and Transport channel for Microsofts WCF

So I just thought of few ideas:
Media processing pipelines can be enabled and automated by queues.

• Document processing pipelines that format documents, perform ETL (extract, transform, load) operations and save it in different formats.
• Image processing pipelines that crop, size adjust brightness/contrast/sharpness, create thumbnails images etc.
• Video/Audio Processing pipelines that append header/footer advertisement frames in an existing video, transcode it in different formats, increase or decrease quality/size etc.

  Store all big documents/images/video on Amazon S3 and relay them via Amazon SQS.

Two or more disparate applications, from different parts of the world, can talk to each other:

• Billing system thats being built and hosted in the US can talk to order processing system which is built and maintained in Africa in a secure/reliable way.
• Central Accounts Payable system in Chicago can now take requests from any department in any city
• Implement a website that runs 9AM-5PM PST and processes customers orders in china in china day time.

Get more juice out of your bureaucratic applications (applications that just take more time for processing or are just slow in responding, causing a bottleneck)

• Integrate Banking systems
• Integrate with Large computation/rendering systems
• Enable and connect various supply chain components across companies (customers, partners, distributors, suppliers)

Use the comments section to brainstorm more SQS ideas.

— Jinesh

Modified 3/10/2021 – In an effort to ensure a great experience, expired links in this post have been updated or removed from the original post.