IronMQ v3 is 10x Faster than RabbitMQ

mideast-iran-asiatic-_horo-2NEWLast year, we announced IronMQ v3, which was rewritten from the ground up to focus on performance and easy deployment/management for on premise installations. Since then, we’ve put all of our highest volume customers on it, some doing billions of requests per day, and nobody has hit the limits yet.

As our technology continues to evolve, it’s important that we continue to measure and benchmark. Here are the results of benchmarks we’ve run comparing IronMQ to RabbitMQ.

Note: If you sign up for IronMQ today for our public cluster, you’ll be on IronMQ v2 by default which is slower than what’s posted here. Send us a note if you want to try IronMQ v3.

The Setup

These benchmarks compare IronMQ v3 vs RabbitMQ in various scenarios. These tests were all run on the same hardware, one box running IronMQ or RabbitMQ and another box on same EC2 VPC running the benchmark. The tests were run on AWS EC2 with c3.8xl instance types, which have the following specs:

  • 32 vCPUs
  • 2x200GB SSDs
  • 60GB of RAM
  • 10 Gb connection

Disk persistence is enabled in all the tests unless stated otherwise, to compare apples to apples, since IronMQ is always persistent. In-memory tests with RabbitMQ and other MQs might provide different performance numbers, but we feel strongly that message persistence and data integrity are paramount and so only offer IronMQ in persistent and clustered configurations.

We used the CoreOS AMI “CoreOS-stable-681.2.0-hvm”, IronMQ version 3.1.15 and RabbitMQ version 3.5.3. Both IronMQ and RabbitMQ were set up as a single node servers.

The benchmarks are open source at iron-maiden . Times are recorded in milliseconds and rates are per second.

Produce and Consume

The following tests produce messages as fast as possible and consume them as fast as possible, in various scenarios. These are similar to RabbitMQ’s own benchmarks.

Test 1: 100 producers, 100 consumers, 1KB message size, 1 queue

As you can see, IronMQ can handle a sustained enqueue rate of ~18,000 – ~19,000 messages per second and a consume rate of ~4K per second on a single queue.

image06

RabbitMQ can handle ~950 per second in and out.

Test 2: 100 producers, 100 consumers, 1KB message size, 10 queues

This one increases the number of queues which should reduce contention on the head of the queue during dequeue.

ironmq-10-queues-1kb

The enqueue rate didn’t change much, but as you can see, the dequeue rate increased by 2.5x, from 4K per second to 10.5-11K per second. rabbit-10-queues-1kb

RabbitMQ didn’t change much by adding more queues.

Test 3: 100 producers, 100 consumers, 1KB message size, 100 queues, 100 batch

In Test 3, we increased the batch size to post 100 messages per request vs 1 per request in the previous tests and also increased the number of queues to 100.

ironmq-100queues-1kb-100batch

Getting close to 100K producing and 40K consuming on IronMQ.

rabbitmq-100queues-1kb-100batch

 

 

15-16K producing and consuming on RabbitMQ.

Test 4: 100 producers, 100 consumers, 1 byte message size, 10 queues, 100 batch

All the previous tests were using 1KB messages, this one uses small single byte messages. Typical use would probably be somewhere in the middle. In cases where throughput matters, however, as in the case of high frequency messaging or IoT workloads, smaller message size should be a desired architectural characteristic.

ironmq-10queues-1b-100batch~175K per second enqueue rate, now we’re talking. Dequeue sits around 20k. The more queues you use, the better the dequeue rate is with IronMQ.

rabbitmq-10queues-1b-100batch

In Test 4, RabbitMQ comes in around 22K per second in and out.

Rate and Latency By Message Size

Next, message rate and latency of IronMQ vs RabbitMQ was measured by message size.

And here’s average latency (lower is better):

And 99th percentile latency:

Fill and Drain

These tests produce and consume separately, a set number of messages. In this case, we push 1 million messages then drain all of the messages.

IronMQ took 39.6 seconds to push 1 million messages onto a single queue, then 1 minutes and 45 seconds to drain them.

image04

RabbitMQ without persistence took 7m28s to push 1 million messages and 7m59s to drain them.

image07
RabbitMQ with persistence on, took 9m13s to push 1 million messages  the queue and 8m17s to drain them.

image01

Bonus: XFS vs EXT4

This is another test we ran to compare IronMQ running on different file systems, Ext4 and XFS. XFS gave us at least 20% better performance overall. Gains were considerably more noticeable when using larger messages sizes. In the below graphs, each message was 32KB which shows a near doubling in performance near the end of the graph. We still suffer in a message rate drop over time due to database compaction, however the decrease is a lot less using XFS.

image05

And XFS:

image03

Conclusion

IronMQ and RabbitMQ are fairly different products with different goals. IronMQ’s goal is to be the best cloud message queue on every cloud, including behind the firewall. IronMQ is a multi-tenant, highly available, horizontally scalable, HTTP based message queue. One IronMQ cluster for your organization can support any number of employees or projects, you don’t need one instance per project.

The results of this benchmark show that IronMQ is also blazing fast, faster than RabbitMQ. IronMQ gives you the best of both worlds.

Here’s a quote that sums up the performance improvements in IronMQ, from one of our customers that switched from IronMQ v2 to v3.

Screenshot 2015-08-05 09.42.20

Thanks to Brandon Gonzalez @bg451 for building and running all these benchmarks and creating all the pretty graphs used in this post.