IronWorker’s Most Requested Feature is Here: Max Concurrency
Overview
 |
| Too many lemmings and you've got a problem. |
Being able to limit how many worker tasks can run at once has been one of the most requested features of IronWorker and it is available now! There are many reasons this feature is so important, here are a few of the more common use cases for it:
-
- Limit load on your database. If you know your database can handle X tasks at a time, set max_concurrency to X, and then queue up as many tasks as you want.
- Limit requests on third party API's to stay under rate limits such as the Facebook API and Twitter API. The Twitter API for instance has a 150 requests per hour limit.
- Limit requests on your own APIs. This is essentially the same as #1.
- Limit requests on third-party sites if you are web crawling or scraping so you don't get blacklisted.
- Eliminate issues related to spiky behavior. For instance, let's say you got TechCrunched and your system wasn't ready for it, your app/site would go down, you're customers would be upset and you wouldn't be able to take orders or do whatever it is your site does. If you had offloaded your work to IronWorker, to begin with, and set a max_concurrency on the tasks, then when you got a spike from being TechCrunched or perhaps the Colbert bump, you would be perfectly fine. Smooth sailing.
As you can see, it opens the door to a lot of new use cases.
How to Use max_concurrency
If you're using the IronWorker CLI, it's very easy, simply add the --max-concurrency flag when uploading your worker code. So if your worker is called max_worker and you had a .worker file called max_worker.worker, you would upload like this:
> iron_worker upload max_worker --max-concurrency 100
Now just queue up as many tasks as you want for max_worker and you can rest easy knowing that a maximum of 100 of them will be running at any point in time.
| This worker has max_concurrency of 250, you can see that only 250 of 1000 tasks are running. |
Unlock the Cloud with Iron.io
Find out how IronWorker and IronMQ can help your application obtain the cloud with fanatical customer support, reliable performance, and competitive pricing.
Great post!
It’s –max-concurrency in the cli –help