Pattern: Creating Task-Level Workers at Runtime

A good worker pattern – based on what we've seen and done – is to chunk the work (and create task-level workers) at runtime and not prior. In other words, use a master/slave setup that is event or time-driven and do the majority of the work using concurrent processes. When the master comes off the queue, it can slice up the work/data space into granular tasks/chunks as well as queue up slave worker jobs to each handle a collection of discreet tasks and data. (If the work or data space is especially large or complicated, then the slicing can be distributed across a set of master tasks)

The reason for waiting until runtime to create the slave workers is that viewing jobs in the schedule is much easier when done at a coarse grain level. Meaning that the jobs scheduled correspond to the units that you're tracking (such as webpages, user profiles, or blocks of data from users, sensors, or other streaming input devices). At this level, you can more easily monitor and inspect the collection of scheduled jobs because they correspond to your key metrics and inputs. Notifications have a better signal-to-noise ratio and status indicators are much more meaningful.