Top 10 Uses For A Message Queue
Overview
We’ve been working with, building, and evangelizing message queues for the last year, and it’s no secret that we think they’re awesome. We believe message queues are a vital component to any architecture or application, which is why we've put so much work into IronMQ. As you consider these 10 reasons, look into the ways we can help you tie your distributed systems together with more speed than any other solution.
Achieve Cloud Elasticity with Iron
Request a demo of IronMQ and see how our lightning fast message queue can improve your distributed systems.
Decoupling
At the start of a project, it’s extremely difficult to predict what the future needs of the project will be. By introducing a layer between processes, message queues create an implicit, data-based interface that both processes implement. This allows you to extend and modify these processes independently, by simply ensuring they adhere to the same interface requirements.
Redundancy
Sometimes processes fail when processing data. Unless that data is persisted, it’s lost forever. Queues mitigate this by persisting data until it has been fully processed. The put-get-delete paradigm, which many message queues use, requires a process to indicate explicitly that it has finished processing a message before removing it from the queue, ensuring your data is kept safe until you’re done with it.
Scalability
Because message queues decouple your processes, it’s easy to scale up the rate at which messages are added to the queue or processed; simply add another process. No need to change code; no need to tweak configurations. Scaling is as simple as adding more power.
Elasticity & Spikability
When your application hits the front page of Hacker News, you’re going to see unusual levels of traffic. Your application needs to keep functioning with this increased load, but the traffic is an anomaly, not the standard; it’s wasteful to have enough resources on standby to handle these spikes. Message queues will allow beleaguered components to struggle through the increased load, instead of getting overloaded with requests and failing completely. Check out our spikability blog post for more information about this.
Iron.io Serverless Tools
Speak to us to learn how IronWorker and IronMQ are essential products for your application to become cloud elastic.
Thanks for subscribing! Please check your email for further instructions.
Resiliency
When part of your architecture fails, it doesn’t need to take the entire system down with it. Message queues decouple processes, so if a process that is processing messages from the queue fails, messages can still be added to the queue to be processed when the system recovers. This ability to accept requests that will be retried or processed at a later date is often the difference between an inconvenienced customer and a frustrated customer.
Delivery Guarantees
The redundancy provided by message queues guarantees that a message will be processed eventually, so long as a process is reading the queue. On top of that, IronMQ provides an only-delivered-once guarantee. No matter how many processes pull data from the queue, each message will only be processed a single time. This is possible because retrieving a message “reserves” that message, temporarily removing it from the queue. Unless the client specifically states that it’s finished with that message, the message will be placed back on the queue to be processed after a configurable amount of time.
Ordering Guarantees
In a lot of situations, the order with which data is processed is important. Message queues are inherently ordered, and capable of providing guarantees that data will be processed in a specific order. IronMQ guarantees that messages will be processed using FIFO (first in, first out), so the order in which messages are placed on a queue is the order in which they’ll be retrieved from it.
Buffering
In any non-trivial system, there are going to be components that require different processing times. For example, it takes less time to upload an image than it does to apply a filter to it. Message queues help these tasks operate at peak efficiency by offering a buffer layer — the process writing to the queue can write as fast as it’s able to, instead of being constrained by the readiness of the process reading from the queue. This buffer helps control and optimize the speed at which data flows through your system.
Understanding Data Flow
In a distributed system, getting an overall sense of how long user actions take to complete and why is a huge problem. Message queues, through the rate with which they are processed, easily help to identify under-performing processes or areas where the data flow is not optimal.
Asynchronous Communication
A lot of times, you don’t need to process a message immediately. Message queues enable asynchronous processing, which allows you to put a message on the queue without processing it immediately. For long-running API calls, SQL reporting queries, or any other operation that takes more than a second, consider using a queue. Queue up as many messages as you like, then process them at your leisure.
Do More with Your Message Queues
We believe these 10 reasons make queues the best form of communication between processes and applications. We’ve spent a year building and learning from IronMQ, and our customers are doing amazing things with message queues. Queues are the key to the powerful, distributed applications that can leverage all the power that the cloud offers.
Unlock the Cloud with Iron.io
If you’d like to get started with an efficient, reliable, and hosted message queue today, check out IronMQ. Read about the difference between IronMQ and Amazon SQS if you’re considering both.
If you’d like to connect with our engineers about how queues could fit into your application, schedule a call today!
UPDATE: We posted a related article recently on the uses of a worker system titled Top 10 Uses of IronWorker (although the uses can apply to any worker system/task queue). They often work in concert with message queues and are used for background processing, schedule jobs, event processing, as a mobile compute cloud, and for many other uses.
Awesome, awesome post. I’ve only recently discovered message queues after a tech talk I went to. I wrote about it here when I was still really excited:
https://blog.hartleybrody.com/asynchronous/
but the code I had to write (Python) to use them always seemed a bit more complex than I liked (Pika library). From looking at the super simple API of your client library, I might just have to start using them again!
https://github.com/iron-io/iron_mq_python
There’s a nifty helper library for Pika which reduces the boilerplate code quite a bit. It is called pikacon. You can find it from PyPI.
+1 on scalability. I just spawn another process and it is working perfect without any extra effort.
While you named 10 advantages, it would be useful if you can provide some sample projects on how to use and explain they the use of MQ is better.
Honestly, most of the examples I see is in fact just one thing: to build a chat room..
What do you mean just chat rooms? One major example is that most medium or large companies have many internal apps and system and this sort of MQ allows for them all to communicate in a managed way with a common protocol.
Valid point! This is a more abstract, general list, but there’s no reason we can’t follow up with a list of specific examples and samples for using a message queue.
Great article.. once upon a time , i felt that MSMQ is used onlty to transfer small amount of data.. But this article is really clear
in the pics it looks more like swans rather than ducks.
@urkurk
Per Paddy’s response to a similar comment in HackerNews:
—–
jjm 26 days ago | link
Side note, I think the picture are of White Geese and not ducks.
—–
paddyforan 26 days ago | link
I had the same thought. However.
https://www.flickr.com/photos/david-hilgart/4142396713/
If the photographer calls them ducks, who am I to argue?
—–