More and more technologies get involved as systems grow, and it’s sometimes hard to keep track of what’s doing what. Caching layers, message queues, serverless functions, tracing frameworks… the list goes on. Once you start sprinkling in public cloud services, you may find yourself developing your way into vendor lock-in. All of the sudden, you’re dealing with one cloud, tons of services, and having to glue everything together in order to make the services talk to each other. One of Iron’s primary goals is to make life easier for developers, and IronMQ’s little known “Push Queue” feature is one that can help prevent you from having to write the glue.
What are Push Queues?
IronMQ has a built-in feature called Push Queues, which when enabled, fire off an event any time a message gets pushed onto that queue. This comes in extremely handy when you immediately want to “do something” (or many things) with that message. With traditional message queues, you’d usually need to write another process that polls your queues for messages at a given duration. MQ’s push queues can fire off events to different types of endpoints, each extremely helpful in their own way.
What Type of events can be triggered?
HTTP
When a message gets put onto your push queue, IronMQ can make a POST request (with the message in the request body) to any URL of your choice. This is extremely handy when you want to notify other systems that some sort of event just happened or kick off another process.
MQ
Inception! You can have the delivery of a message populate another IronMQ queue. This is helpful if you want to tie multiple queues together or create a dead letter queue for example.
Worker
MQ can connect directly to IronWorker and pass its message as the payload to one of your jobs. How cool is that!? In order to exemplify how cool that actually is, we’ll run through a real-life scenario.
MQ & Worker Example
Let’s say you have a time-sensitive nightly job that processes uploaded CSV files. It needs to process all of the files uploaded during that day and finish within a set amount of time. As your system grows and there are more CSV files to process, your nightly process starts running behind schedule.
You realize that a lot of the time spent in your nightly worker is spent formatting the CSV file into the correct format. It would make sense to split this process into two distinct stages, formatting and processing. When a CSV file is received, you could send a message to your push queue which in turn will kick off a “formatting” worker job to pre-process the CSV file into the correct format. Your nightly “processing” worker job will then be able to fly through the CSV files because it no longer needs to fix any formatting issues.
The beauty here is that you can continue to add more push events to the queue. When a file is uploaded, maybe you also need to ping another worker that handles OCR or post an update to an external HTTP endpoint letting it know exactly “what” file was uploaded. Without a push queue, you’d be adding a lot of custom code to handle these requests, retries, errors, etc. IronMQ’s push queues take care of all of this for you.
How can I configure a Push Queue?
Retries
You can configure your queue to allow for a custom amount of retries, a custom delay between retries, and even provide another queue to store failed push attempts. For example, using an HTTP event, MQ will retry pushes (3 times by default) every time it receives a non-200 response code.
Timeouts
If your event never receives a response after a certain period of time (10 seconds by default), it will chalk that up as a failed attempt and retry.
Unicast or Multicast?
You can even fire off multiple events from one queue. If you need to trigger one HTTP endpoint and also fire off a Worker job, that’s not a problem.
How do I create a push queue?
Creating one is straightforward. Here’s an example cURL request that creates a multicast Push Queue with an HTTP endpoint as well as a Worker endpoint.
IronMQ has client libraries available in most languages, so you can easily create one programmatically as well. Here’s an example in PHP:
Conclusion
With one IronMQ Push Queue, you can make a lot happen. If you were to try and replicate a multicast Push Queue in a traditional message queue, for example, you’d end up writing a lot of custom code to glue everything together. You’d also have to deal with scaling your infrastructure as your message queue needs grew. With IronMQ, you can save time and money focusing on your applications business logic, and less time on glue and infrastructure. For more detailed information about Push Queues, visit the IronMQ documentation.
If you’re interested in knowing more about IronMQ, or want to chat about how we may be able to help, call us anytime at 888-501-4766 or email at support@iron.io.
