Iron.io Adds Longer Running Workers
 |
| Long-Running Workers are Now Available in IronWorker |
Iron.io is happy to announce long-running workers are now available within IronWorker. Up until now, workers running on the platform have been limited to 60 minutes in duration.
Users on Production and Enterprise plans or using Dedicated Clusters can now have workers that run for hours at a time. This gives users greater flexibility to handle even more extensive asynchronous workflows.
Worker systems are essential for doing transaction and event processing, background processing, and other types of distributed processing. (GitHub once estimated that over 40% of their processing takes place in the background.)
Short-Running Tasks + Longer Running Tasks
A 60-minute limit fits most use cases for a worker system and provides the right balance between processing power, time-in-queue latency, system flexibility, and responsiveness. Great benefits can result when you distribute work across a set of task-specific workers as shown here, here and here – but that’s not always feasible to do and keep task duration to under 60 minutes.
Longer running workers are the answer to these situations where workloads can’t be broken into discrete units or where monitoring and scheduling of a complicated process might extend across a few hours.
Production plans and users on Dedicated clusters can make use of this feature right away. The default for longer running workers is 2 hours but this can be extended by talking with one of our account teams. (Users on Dedicated Clusters just need to let us know what and we can make the change quickly.)
 |
|
|
Support for More Advanced Workloads
The release of this new capability joins several others we’ve released over the past few months. These new features on the Iron.io platform include:
- Dedicated Clusters (link)
- High-Memory Workers (link)
- Long-Running Workers (link)
- Larger MQ Message Size (link)
• One example where a longer-running worker might come into play is a large unbroken iteration. Many of our users use Ironworker to process large CSV files, some of which can span millions of rows. We generally suggest breaking down and parsing the files in chunks but in some cases, this can be difficult and even problematic as in the case where line items may need to be processed in order. An example here might be inventory changes or other transaction processing.
• A second example can be a large crawling and compilation operation. Using the PhantomJS stack within IronWorker, users are able to literally take thousands of images and snapshots of websites and build PDFs, gifs, and even video files of the images. In most cases, the translation and transcoding period grows linearly with the size of the required output, which means the amount of processing can quickly go above and beyond the standard 1-hour limit.
• A third example might be in the case of using a master worker to monitor a process that might take longer than 60 minutes. In general, we recommend tasks to queue up other work (master-slave pattern) and scheduled workers to monitor progress but in the case where users want a persistent worker to continually run, longer running workers now provide that capability.
To see more use cases for a worker system and get more details the examples above, check out this article on top uses of a worker system as well as some of the success stories on our site. Most of the examples center around short duration workers but longer running workers can slot in to give you that extra processing element that you might need.
Making Use of Long-Running Workers
Longer running workers are available to users on Production and Enterprise plans or operating on Dedicated Clusters.
To make use of long-running workers, we first need to enable your account – contact one of our account teams to get provisioned. Once that’s done, all you have to do is include a new timeout value when you queue a task. The maximum limit for long running workers is initially set to 2 hours (7200 seconds) but this can be extended upon request to up to 24 hours for Enterprise accounts and Dedicated Clusters.
Setting a timeout with a curl command (in seconds)
$ curl -H “Content-Type: application/json” -d ‘{"tasks": [{"code_name": "ExampleWorker", "payload": "", "timeout": 7200}]}’ https://worker-aws-us-east-1.iron.io/1/projects/<PROJECT ID>/tasks?oauth=<TOKEN>Setting a timeout with the CLI tool in IronWorker (in seconds)
$ iron_worker queue <WORKER_NAME> --timeout 7200
Getting Started
To try IronWorker for free, sign up for an account at Iron.io. We’ll even give you a trial of some of the advanced features so that you can see how processing at scale will change the way you view modern application development.
We can also connect with one of our account teams to dive into solutions at more depth or pair program with one of our developer evangelists to get up and running in minutes.
What are you waiting for? Simple, scalable, long-running processing awaits.
