A Serverless Message Queue Without the Glue

More and more technologies get involved as systems grow, and it’s sometimes hard to keep track of what’s doing what. Caching layers, message queues, serverless functions, tracing frameworks… the list goes on.  Once you start sprinkling in public cloud services, you may find yourself developing your way into vendor lock-in.  All of the sudden, you’re dealing with one cloud, tons of services, and having to glue everything together in order to make the services talk to each other.  One of Iron’s primary goals is to make life easier for developers, and IronMQ’s little known “Push Queue” feature is one that can help prevent you from having to write the glue.

What are Push Queues?

IronMQ has a built-in feature called Push Queues, which when enabled, fire off an event any time a message gets pushed onto that queue. This comes in extremely handy when you immediately want to “do something” (or many things) with that message. With traditional message queues, you’d usually need to write another process that polls your queues for messages at a given duration. MQ’s push queues can fire off events to different types of endpoints, each extremely helpful in their own way.

What Type of events can be triggered?

HTTP
When a message gets put onto your push queue, IronMQ can make a POST request (with the message in the request body) to any URL of your choice. This is extremely handy when you want to notify other systems that some sort of event just happened or kick off another process.

MQ
Inception! You can have the delivery of a message populate another IronMQ queue. This is helpful if you want to tie multiple queues together or create a dead letter queue for example.

Worker
MQ can connect directly to IronWorker and pass its message as the payload to one of your jobs. How cool is that!?  In order to exemplify how cool that actually is, we’ll run through a real-life scenario.

MQ & Worker Example

Let’s say you have a time-sensitive nightly job that processes uploaded CSV files.  It needs to process all of the files uploaded during that day and finish within a set amount of time.   As your system grows and there are more CSV files to process, your nightly process starts running behind schedule.

You realize that a lot of the time spent in your nightly worker is spent formatting the CSV file into the correct format.  It would make sense to split this process into two distinct stages, formatting and processing.  When a CSV file is received, you could send a message to your push queue which in turn will kick off a “formatting” worker job to pre-process the CSV file into the correct format. Your nightly “processing” worker job will then be able to fly through the CSV files because it no longer needs to fix any formatting issues.

The beauty here is that you can continue to add more push events to the queue.  When a file is uploaded, maybe you also need to ping another worker that handles OCR or post an update to an external HTTP endpoint letting it know exactly “what” file was uploaded.  Without a push queue, you’d be adding a lot of custom code to handle these requests, retries, errors, etc.  IronMQ’s push queues take care of all of this for you.

How can I configure a Push Queue?

Retries
You can configure your queue to allow for a custom amount of retries, a custom delay between retries, and even provide another queue to store failed push attempts. For example, using an HTTP event, MQ will retry pushes (3 times by default) every time it receives a non-200 response code.

Timeouts
If your event never receives a response after a certain period of time (10 seconds by default), it will chalk that up as a failed attempt and retry.

Unicast or Multicast?
You can even fire off multiple events from one queue. If you need to trigger one HTTP endpoint and also fire off a Worker job, that’s not a problem.

How do I create a push queue?

Creating one is straightforward.  Here’s an example cURL request that creates a multicast Push Queue with an HTTP endpoint as well as a Worker endpoint.

IronMQ has client libraries available in most languages, so you can easily create one programmatically as well.  Here’s an example in PHP:

Conclusion

With one IronMQ Push Queue, you can make a lot happen. If you were to try and replicate a multicast Push Queue in a traditional message queue, for example, you’d end up writing a lot of custom code to glue everything together. You’d also have to deal with scaling your infrastructure as your message queue needs grew. With IronMQ, you can save time and money focusing on your applications business logic, and less time on glue and infrastructure. For more detailed information about Push Queues, visit the IronMQ documentation.

If you’re interested in knowing more about IronMQ, or want to chat about how we may be able to help, call us anytime at 888-501-4766 or email at support@iron.io.

* These fields are required.


Top 10 Uses For A Message Queue


Geese love queues.
(Image by D.Hilgart)

We’ve been working with, building, and evangelising message queues for the last year, and it’s no secret that we think they’re awesome. We believe message queues are a vital component to any architecture or application, and here are ten reasons why:
Continue reading “Top 10 Uses For A Message Queue”

OpenShift Ecosystem: Iron.io Brings a Serverless Experience to OpenShift

There has been a lot of buzz around the Serverless trend lately; what it really means and what are its merits. At the end of the day it’s really just a new way to treat certain workloads – background jobs. How does this new pattern fit in the context of developing cloud native applications and operating container platforms such as Red Hat OpenShift?

LAYING THE FOUNDATION

Delivering continuous innovation to customers often leads to continuous pressure on the developers to build and ship software… well, continuously. Smart companies are doing all they can to empower their development teams with the right culture to encourage productivity, and the right tools to make it happen. Emerging as the foundational layer for many organizations’ application development efforts is a container application platform, with OpenShift as a leading choice.

As infrastructure resources continue to be commoditized, and as services continue to be exposed as APIs, having a foundational layer is critical to bring everything together. This is especially important when dealing with multiple distributed applications and multiple distributed teams, as containerized applications, workloads, and services need a unifying environment. Continue reading “OpenShift Ecosystem: Iron.io Brings a Serverless Experience to OpenShift”

Best Practices and Anti-Patterns for Workers and MQs

Best Practices and anti-patterns for workers and message queues

Thanks to Ruth Hartnup for the base image! CC BY 2.0

If you’ve been programming for a while, it’s probable that someone, somewhere, has recommended the Gang of Four book.

The book dissects Object Oriented programming. It lists numerous ways of royally messing things up, but it’s claim to fame is that it also lists ways to do it right! These well tested paths to success often come with explanations for when to use them and why they’re good at avoiding common pitfalls.

These are design patterns. They’re embedded in the culture of programming, and they’re an amazing way to learn from others’ mistakes. At the outset of any project, a lot of paths are open to you. Design patterns illuminate the dark paths from the healthy, low stress approaches.

Today, we’re releasing our very own set of best practices and anti-patterns in the form of a white paper. It’s a quick read and will save you time as you tinker on your own workers and message queues. So, what do you have to lose?

Continue reading “Best Practices and Anti-Patterns for Workers and MQs”

See How Untappd Processes 100s of 1000s of Updates a Night

Ironio_web_HeroBanner_Untappd_V3_twitter

Untappd, a mobile check-in app for beer lovers, helps beer lovers share their passion with friends and other beer lovers from around the world.

With millions of users checking-in, tracking and sharing newly discovered beers and beer drinking locations, as well as earning points for coveted badges, the Untappd app’s processing power is tested daily.

Watch the Untappd video to see how the team was able to make the app an integrated part of many beer lovers night out. Continue reading “See How Untappd Processes 100s of 1000s of Updates a Night”

Upgrade to IronMQ v3 in Under 30 Minutes

Upgrade to IronMQ v3 Today!

Thanks to Johan Fantenberg for the base image CC BY 2.0

Earlier this week, Iron.io Customer Success Engineer JPK and I sat down with Michael Geneles and Stephen Kamenar from PitchBox to discuss their transition to IronMQ v3. Last Friday, they made the transition in less than 30 minutes! We were so impressed, we asked them for the details of their experience.

First, how does PitchBox leverage IronMQ? PitchBox helps marketers find online influencers. To accomplish that they crawl blogs and social media. Then, they run some fancy algorithms voodoo magic to process the results, and boom! Marketers are matched with their ideal influencers.

PitchBox’s dashboard is written in PHP. But, their crawler is an event driven NodeJS app. A couple thousand websites may need to be crawled and parsed at any given time.

Continue reading “Upgrade to IronMQ v3 in Under 30 Minutes”

Message Queues & Workers: the Heart of Modern Infrastructure

Message Queues + Workers

Thanks to Sonny Abesamis for the base image! CC BY 2.0

Increasingly, message queues and workers are intertwined with the language of modern infrastructure. You might rely on explicit solutions like IronMQ or IronWorker. You might not. Whether you do or don’t is irrelevant: MQs and workers are in everything these days.

MQs and workers are hidden heroes, quietly powering a lot of the technology that many of us rely on. They’re core components in programming languages, MVC frameworks, and even web servers.

As a result, when making infrastructure decisions a good understanding of both MQs and workers is essential. The white paper below will take you from a fuzzy understanding to a well-versed conceptualization for both MQs and workers.

Download the latest white paper from Iron.io: a Refresher on Message Queues & Workers.
Give it a read and let us know what you think!

Iron.io Now Available in Microsoft Azure Marketplace

azure-icons-drawing
Iron.io continues to grow its ecosystem of value-added partners. To this point, today you can now find Iron.io solutions in the Application Services within the Microsoft Azure Marketplace. Azure users can now directly leverage Iron.io within their applications to respond to application events, decouple components as independent services, offload individual workloads, and schedule regular occurring jobs.

IronWorker and IronMQ can be added by visiting the Azure Marketplace. Developers can then write and package task code for deployment to IronWorker’s processing environment within Azure. The Iron.io dashboard built into Azure provides detailed insight into the state of tasks for monitoring complete application activity and performance.

By using Azure and Iron.io, developers and operators can move individual components to the cloud, while maintaining safe application environments through improved security. Iron.io can also act as a key processing gateway to Azure component services including storage, queues, mobile services, and more, making it easy to create hybrid solutions of existing client-server applications and cloud-based microservices.

To quote Iron.io CEO Chad Arimura: “The combination of Azure and Iron.io brings flexibility, scalability, control and security – all the things Enterprises are seeking for their applications.

IronWorker and IronMQ are currently available in the West US region of Azure, and support multiple languages with native SDKs including Go, Java, Ruby, PHP, Python, Node.js, and .NET.

Squashing Bugs in The Cloud: How Airbrake uses IronMQ v3

Airbrake-Iron.io-Blog

By Reed Allman, Backend Engineer, Iron.io

Ah, message queues.

Last year we set out to build a better message queue, both for our internal use and to run as a service for customers. If you’ve ever looked into message queues yourself, you’ll know they’re all different, and, conveniently, none of them are ever quite what you need. We hope we’ve found a sweet spot, and we’ll even run it for you. After over a year of being in production, we decided it was time to grab a mic and hit the streets. Or email. Definitely just email. Have ya seen the streets in SF?

Continue reading “Squashing Bugs in The Cloud: How Airbrake uses IronMQ v3”

IronMQ v3 is 10x Faster than RabbitMQ

mideast-iran-asiatic-_horo-2NEWLast year, we announced IronMQ v3, which was rewritten from the ground up to focus on performance and easy deployment/management for on premise installations. Since then, we’ve put all of our highest volume customers on it, some doing billions of requests per day, and nobody has hit the limits yet.

As our technology continues to evolve, it’s important that we continue to measure and benchmark. Here are the results of benchmarks we’ve run comparing IronMQ to RabbitMQ. Continue reading “IronMQ v3 is 10x Faster than RabbitMQ”