Last year, we announced IronMQ v3, which was rewritten from the ground up to focus on performance and easy deployment/management for on premise installations. Since then, we’ve put all of our highest volume customers on it, some doing billions of requests per day, and nobody has hit the limits yet.
As our technology continues to evolve, it’s important that we continue to measure and benchmark. Here are the results of benchmarks we’ve run comparing IronMQ to RabbitMQ.
Note: If you sign up for IronMQ today for our public cluster, you’ll be on IronMQ v2 by default which is slower than what’s posted here. Send us a note if you want to try IronMQ v3.
These benchmarks compare IronMQ v3 vs RabbitMQ in various scenarios. These tests were all run on the same hardware, one box running IronMQ or RabbitMQ and another box on same EC2 VPC running the benchmark. The tests were run on AWS EC2 with c3.8xl instance types, which have the following specs:
- 32 vCPUs
- 2x200GB SSDs
- 60GB of RAM
- 10 Gb connection
Disk persistence is enabled in all the tests unless stated otherwise, to compare apples to apples, since IronMQ is always persistent. In-memory tests with RabbitMQ and other MQs might provide different performance numbers, but we feel strongly that message persistence and data integrity are paramount and so only offer IronMQ in persistent and clustered configurations.
We used the CoreOS AMI “CoreOS-stable-681.2.0-hvm”, IronMQ version 3.1.15 and RabbitMQ version 3.5.3. Both IronMQ and RabbitMQ were set up as a single node servers.
The benchmarks are open source at iron-maiden . Times are recorded in milliseconds and rates are per second.
Produce and Consume
The following tests produce messages as fast as possible and consume them as fast as possible, in various scenarios. These are similar to RabbitMQ’s own benchmarks.
Test 1: 100 producers, 100 consumers, 1KB message size, 1 queue
As you can see, IronMQ can handle a sustained enqueue rate of ~18,000 – ~19,000 messages per second and a consume rate of ~4K per second on a single queue.
RabbitMQ can handle ~950 per second in and out.
Test 2: 100 producers, 100 consumers, 1KB message size, 10 queues
This one increases the number of queues which should reduce contention on the head of the queue during dequeue.
RabbitMQ didn’t change much by adding more queues.
Test 3: 100 producers, 100 consumers, 1KB message size, 100 queues, 100 batch
In Test 3, we increased the batch size to post 100 messages per request vs 1 per request in the previous tests and also increased the number of queues to 100.
Getting close to 100K producing and 40K consuming on IronMQ.
15-16K producing and consuming on RabbitMQ.
Test 4: 100 producers, 100 consumers, 1 byte message size, 10 queues, 100 batch
All the previous tests were using 1KB messages, this one uses small single byte messages. Typical use would probably be somewhere in the middle. In cases where throughput matters, however, as in the case of high frequency messaging or IoT workloads, smaller message size should be a desired architectural characteristic.
In Test 4, RabbitMQ comes in around 22K per second in and out.
Rate and Latency By Message Size
Next, message rate and latency of IronMQ vs RabbitMQ was measured by message size.
And here’s average latency (lower is better):
And 99th percentile latency:
Fill and Drain
These tests produce and consume separately, a set number of messages. In this case, we push 1 million messages then drain all of the messages.
IronMQ took 39.6 seconds to push 1 million messages onto a single queue, then 1 minutes and 45 seconds to drain them.
RabbitMQ without persistence took 7m28s to push 1 million messages and 7m59s to drain them.
Bonus: XFS vs EXT4
This is another test we ran to compare IronMQ running on different file systems, Ext4 and XFS. XFS gave us at least 20% better performance overall. Gains were considerably more noticeable when using larger messages sizes. In the below graphs, each message was 32KB which shows a near doubling in performance near the end of the graph. We still suffer in a message rate drop over time due to database compaction, however the decrease is a lot less using XFS.
IronMQ and RabbitMQ are fairly different products with different goals. IronMQ’s goal is to be the best cloud message queue on every cloud, including behind the firewall. IronMQ is a multi-tenant, highly available, horizontally scalable, HTTP based message queue. One IronMQ cluster for your organization can support any number of employees or projects, you don’t need one instance per project.
The results of this benchmark show that IronMQ is also blazing fast, faster than RabbitMQ. IronMQ gives you the best of both worlds.
Here’s a quote that sums up the performance improvements in IronMQ, from one of our customers that switched from IronMQ v2 to v3.
Thanks to Brandon Gonzalez @bg451 for building and running all these benchmarks and creating all the pretty graphs used in this post.