IronMQ Push Queues – Reliable Message Delivery for the Cloud
Overview
IronMQ Push Queues adds notification capabilities to your queues allowing you to send messages to other systems in a scalable, flexible, and cost-effective way. You simply add subscribers, which are HTTP endpoints, to your IronMQ queues then any message you post to your queue will be reliably delivered to the subscribers.
Table of Contents:
Achieve Cloud Elasticity with Iron
Speak to us to find how you can achieve cloud elasticity with a serverless messaging queue and background task solution with free handheld support.
Why Push Queues?
Let your application do the work
More efficient for lightly used queues
Push Queues eliminate the need to poll your queues for messages, which was the only option until now. Polling for messages is fine and effective if the queue is heavily utilized and there are always messages on the queue but can be wasteful when the queue isn’t used that much. For instance, if you pushed one message onto the queue every hour, but you need to get that message immediately, then you may poll every second which means you would check the queue and get nothing back 3,599 every hour.
More reliable message delivery if you use IronMQ as an intermediary between systems
Another reason to use push queues is to ensure reliable delivery of messages. For instance, if your application posts to an HTTP endpoint and that endpoint was down or the request failed for some reason, then what? Either the message is lost or you'd have to build in some new retry logic, logging, and reporting to ensure that the message didn't just get lost in the void. But if you post the message to IronMQ and let IronMQ deliver the message, you can ensure safe delivery to the destination because IronMQ will retry failed requests and if retries don’t work, you can find which requests have failed and do something about it. IronMQ provides a much more robust solution for communicating between systems.
If you have provided webhook support in your service, consider using IronMQ to deliver those messages so you can ensure reliable delivery.
Two different types of push queues: unicast and multicast.
Unicast pushes out to a single random subscriber. This is a worker type pattern where you can have a pool of processes (subscribers), but you want your messages processed once and only once by one subscriber.
Multicast will push to all subscribers. This is a pub/sub pattern that delivers messages to all subscribers/endpoints.
The type of push queue is defined with the "push_type" parameter when you update your queue to be a push queue. Default is "multicast".
Retries
If a subscriber doesn’t return a 2XX status code, IronMQ will retry pushing the message to the subscriber for the number of retries you specify using the “retries” parameter, with a delay between each try that you also specify with the “retries_delay” parameter. The default "retries" is 3 and "retries_delay" is 60 seconds.
Iron.io Serverless Tools
Speak to us to learn how IronWorker and IronMQ are essential products for your application to become cloud elastic.
Thanks for subscribing! Please check your email for further instructions.
Checking Status
You can check the push status for each message you post and it will tell you the status and status code (HTTP response code) for each subscriber/endpoint. For example
Example Usage
Here's an example of how to use push queues. It consists of a simple ruby web app with two endpoints that are set up to receive POST requests. Here's how to test it out (you'll need an Iron.io account and a Heroku account for this).
1) Setup a simple application on Heroku to receive messages
- git clone https://github.com/treeder/push_queue_demo_app.git
- cd push_queue_demo_app
- heroku create
- heroku addons: add iron_cache (we need this for simple storage for the demo)
- git push heroku master
- heroku logs --tail (do this so you can watch the logs and see the incoming requests)
Grab the URL you'll see in the output of the above command and open it in your browser just to ensure it's running. It should say "I like getting messages from IronMQ!" The full code for this simple app can be found here: https://github.com/treeder/push_queue_demo_app
2) Create a queue with subscribers
Now we'll create a push queue on IronMQ with two subscribers. The subscribers are two different endpoints to the application above so you'll need the URL you got from heroku above. Here is a curl snippet to set subscribers on a queue (if the queue doesn't exist, it will be created automatically):
You'll need to replace TOKEN and PROJECT_ID with your Iron.io token and project_id and change the base URL's to point to your heroku application that you got from above.
3) Post a message to the queue
Now that we've changed the queue to a push queue with multicast and set a couple subscribers on it, let's push a message to the queue and both of the endpoints on your application will receive the messages. Here is a curl example to post a message to the queue:
You should get a curl response that looks like this:
Now refresh your app in the browser and you should see that it received the messages. And you can see the requests in the heroku logs too.
4) Check delivery status (Optional)
If you want the detailed status of the delivery to each of your subscribers, you can check that too. In the curl example below, you'll need to exchange MESSAGE_ID with the id that was returned in the response above when you posted a message.
This should return a response like this:
Conclusion
Well there you have it, folks, reliable message delivery built for the cloud. Please try it out and let us know what you think, we'd love your feedback.
The IronMQ API docs have been updated with push queue support so you can find more information on how to use it there: http://dev.iron.io/mq/reference/api/
Unlock the Cloud with Iron.io
Find out how IronWorker and IronMQ can help your application obtain the cloud with fanatical customer support, reliable performance, and competitive pricing.
How many multicast endpoints can be added for a queue?
Maximum 100 endpoints.
“Unicast pushes out to a single random subscriber” – why random over say round-robin (as with AWS ELB)? Just wondering how this would work with a smallish number of subscribers.
The DevDocs illuminate some of the logic, (in the case of an error):
https://dev.iron.io/mq/3/reference/push_queues/#retries
If you’re worried about the speed of the algorithm, IronMQv3 comparison’s are here: https://ironio2016.wpengine.com//blog/2015/08/ironmq-v3-is-10x-faster-than-rabbitmq.html
Regarding the 202 response for push queues – is it possible to return a timeout in the body of the response, i.e. to allow the consumer of the message to tell iron MQ how long it’s going to take (upper limit) to process the message?