Squashing Bugs in The Cloud: How Airbrake uses IronMQ v3

By Reed Allman, Backend Engineer, Iron.io

Ah, message queues.

Last year we set out to build a better message queue, both for our internal use and to run as a service for customers. If you’ve ever looked into message queues yourself, you’ll know they’re all different, and, conveniently, none of them are ever quite what you need. We hope we’ve found a sweet spot, and we’ll even run it for you. After over a year of being in production, we decided it was time to grab a mic and hit the streets. Or email. Definitely just email. Have ya seen the streets in SF?

We asked Airbrake what they were using IronMQ for, and they hit us back with (duh!):

“IronMQ helps Airbrake capture more exceptions than McDonald’s serves burgers”

Somebody (pipe up!) even spent the time to make this awesome… happy meal? (We couldn’t resist!)

We put Airbrake on IronMQ v3 last October, and ever since then they’ve averaged nearly 100 million API requests and 35 million messages per day at an average clip of just over 3ms per request.

The bumps in latency / throughput in the charts are from trying to tune HAProxy SSL termination, when really it was our own client libraries that were misconfigured (hey, look, another shop full of Gophers 😉 ;). The cipher kittens were clogging our internet tubes from handshaking all the time. Their lead developer even let us in on how it looked from their side:

“It was deployed yesterday. And the difference is huge – avg latency goes from 150ms to just 2-3 ms. Many thanks for this – going to update all our http.Client usage!”

We didn’t even touch IronMQ! Recently, we saw Airbrake’s traffic double to around 160M req / 60M messages per diem, with no noticeable bump in latency (post cipher kitty patch). This is all powered by just one IronMQ broker. Now we have the ability to scale IronMQ horizontally, ensuring that Airbrake will be able to serve burgers — errors — for a very long time to come. 

And, when you get a chance check out Airbrake.io, the super cool error and bug tracker.

about muah: Reed is currently working on building message queues at Iron.io. He spends most of his time trying to figure out why machines aren’t doing the things that he told them to do. He thinks this is enough of him talking about himself in third person.