Pattern: Creating Task-Level Workers at Runtime
A type of topic appearing more and more frequently in StackOverflow and Quora are questions on general architecture and app design. We came across one the other day on approaches to queuing and scheduling workers.
Table of Contents
Related Reading: Top 10 uses of IronWorker
Achieve Cloud Elasticity with Iron
Speak to us to find how you can achieve cloud elasticity with a serverless messaging queue and background task solution with free handheld support.
Fine-Grained Data | Course-Grained Workers
Instead of creating a worker for each data element, however, it's better to have each worker work on a reasonable collection of tasks or data items. We've found that having each worker process multiple tasks (20-1000 data items for example) provides a good balance between:
- optimizing worker setup (establishing a database connection for example)
- providing good introspection into the jobs
- making retries and exception handling more manageable
Iron.io Serverless Tools
Speak to us to learn how IronWorker and IronMQ are essential products for your application to become cloud elastic.
Unlock the Cloud with Iron.io
Find out how IronWorker and IronMQ can help your application obtain the cloud with fanatical customer support, reliable performance, and competitive pricing.
Leave a Comment