Anti-Pattern: Lots of Scheduled Jobs

With the cloud, elasticity, and the modern app stack (servers + workers + datastores), this scheduling flexibility opens up new possibilities in terms of handling an app’s processing needs. It’s now possible to schedule jobs and have them coordinate running thousands of jobs and processing lots of data across a number of servers — all without having to manage servers or infrastructure.

But with this power brings with it some good and not-so-good ways of doing things. We addressed a pattern for creating Master/Slave jobs in a blog not too long ago. We’ve used this approach successfully within applications we’ve written and have recommend it far and wide.

Pattern: Creating Task-Level Workers at Runtime

Over the course of the last few days, though, we’ve gotten a good look at the anti-pattern. What happens when you create a lot of scheduled jobs — one for each low level task (instead of creating a master worker that creates slave workers when it comes off schedule).

Take our word for it. It’s not pretty. Either from an application standpoint (keeping track of 10s of thousands of scheduled jobs) or from a job queueing/management standpoint (lots of processing to keep things on schedule).

After talking with a customer building a test app and doing a bit of work on our end, things are back to normal. There may have been some delays in scheduled jobs running last night. We apologize. We’ve addressed some bottlenecks and will be creating additional safeguards to make sure quality of service remains high.

Addressing this issue does, however, offer us an opportunity to provide the following lesson for app development:

Instead, schedule jobs that accesses a data store and then queues up jobs to perform the work that is needed. Much easier that way.