Blog

Search This Blog

Loading...

Monday, August 6, 2012

IronWorker's Most Requested Feature is Here: Max Concurrency

Too many lemmings and you've got a problem.
Being able to limit how many worker tasks can run at once has been one of the most requested features of IronWorker and it is available now! There are many reasons this feature is so important, here are a few of the more common use cases for it:
  1. Limit load on your database. If you know your database can handle X tasks at a time, set max_concurrency to X and then queue up as many tasks as you want. 
  2. Limit requests on third party API's to stay under rate limits such as the Facebook API and Twitter API. The Twitter API for instance, has a 150 requests per hour limit.
  3. Limit requests on your own API's. This is essentially the same as #1. 
  4. Limit requests on third party sites if you are web crawling or scraping so you don't get blacklisted. 
  5. Eliminate issues related to spiky behavior. For instance, let's say you got TechCrunched and your system wasn't ready for it, your app/site would go down, you're customers would be upset and you wouldn't be able to take orders or do whatever it is your site does. If you had offloaded your work to IronWorker to begin with and set a max_concurrency on the tasks, then when you got a spike from being TechCrunched or perhaps the Colbert bump, you would be perfectly fine. Smooth sailing. 
As you can see, it opens the door to a lot of new use cases.

How to Use max_concurrency


If you're using the IronWorker CLI, it's very easy, simply add the --max-concurrency flag when uploading your worker code. So if your worker is called max_worker and you had a .worker file called max_worker.worker, you would upload like this:

    > iron_worker upload max_worker --max-concurrency 100

Now just queue up as many tasks as you want for max_worker and you can rest easy knowing that a maximum of 100 of them will be running at any point in time. 

This worker has max_concurrency of 250, you can see that only 250 of 1000 tasks are running.