Relify adds IronWorker to their Stack ➞ Ridiculously Simple Worker Scalability
Relify, a Recommendations as a Service engine, is one of the really cool customers that Iron.io serves. They offer a simple API that eliminates the complexity of developing a recommendation engine –which means you can greatly increase your relevance to your users.
Terry Horner is the Co-Founder and CEO of Relify. A recent blog post of his details their use of a scale-out processing pattern that, in concert with IronWorker, allowed them to significantly scale their recommendations service. All in a short period of time without any infrastructure or much cost.
It’s a great read and we thought we’d highlight some portions:
Relify Blog Post
We’d come so far and had realized that there was a distinct possibility that we’d have to implement our own worker service. Then fortune smiled. The folks at Iron.io sent me a “Tips & Tricks” email on the same day I was getting ready to start coding our worker service (no joke). In that email was a link to a blog post by one of Iron.io’s Co-Founders, Travis Reeder, entitled How to Reduce a 9 Hour Job to a 9 Minute Job. Were they reading my mind?
It turns out that we weren’t the only ones with such struggles as the first line of the post so poignantly states: “A common problem developers have is how to run through a large number of tasks in a short amount of time.” PREACH ON! I kept reading.
I immediately felt better after reading the post. Why hadn’t I attempted to dig deeper into Iron.io’s blog before I threw in the towel? Or hit them up on HipChat (which I subsequently did, and they were awesome). The answer was there the whole time!
The post got me thinking about breaking work into smaller pieces to take advantage of concurrent task execution. I imagined that we could spawn additional tasks based on the total number of items in a queue and let recursion take its course until the workers burned through all of the records in each queue. The end result was a collection of workers that run on a schedule and are entirely self-contained within IronWorker.
…
This took us from having the ability to process 1,000-2,000 records/minute to 120,000/minute (2,000/second) based on our current pricing plan. Bumping ourselves up to the next plan would put us somewhere around 300,000/minute which clocks in at 5,000/second. And we didn’t bump write throughput up a single tick. Increase the write throughput and things get crazy fast!
And the best part? Everything I’ve mentioned above took less than a day to implement and costs less than $500/month (AWS costs included). FO REALZ.
Limitless scalability is real and the team at Iron.io is making it happen. IronWorker is rock solid and our stack wouldn’t be the same without it.
About the Author
Terry Horner is the Co-Founder and CEO of Spent, Inc., the parent company of Relify. Father. Husband. Triathlete. Entrepreneur. Creative thinker. Problem solver. Student of business. Tech geek. Kent State alum. Cleveland fan. Browns season ticket holder. He loves building things and has bootstrapped three profitable startups. Working on number four.
Recommendations as an Essential Service
Recommendations, personalization, and discovery are becoming core components to almost every site and service out there. These capabilities take a nice site and turn it into an essential site. Recommendations are hard to get right, though.
You’re continually getting inputs in the form of clickstreams, checkins, purchases, follows, likes, plus-ones, and other forms of signals and intent. You have to manage this huge amount of data inputs, rank them appropriately, perform continual processing on the stream, and then return genuine items of relevance. All across an array of object types (books, movies, photos, products, people, …) It’s a perfect use case for elastic message queues and scalable task queues. All these inputs need to be collected and processed continually and at huge scales.
Finding a service that is simple to use, provides good recommendations, and scales is a rare combination. Relify has something here which certainly makes them a company to watch.
