New Release: Job Fairness, More Generic Backend and New Gem

Hi all, we pushed out a major new release a couple of nights ago and it is a big step forward in a couple of areas:  fairness amongst all users in the system and a more generic backend to allow any Ruby code to run, more on that below.

Fairness

Given a limited set of resources to be shared among a number of users, fairness is the concept of ensuring that all users get a fair share of the system and prevents heavy users from hogging all the resources.

Previously, we didn’t really have provisions to handle this so when a heavy user queued up thousands of jobs, all other users would have to wait for those jobs to be run, sometimes hours. Basically a FIFO queue shared across all users. You could of course bump up the priority of your jobs to get to the head of the line, but the same problem could happen if someone queued up thousands of jobs in a higher priority.

This new release goes a long way towards fixing this issue by ensuring all users jobs will be run in a reasonable amount of time. In other words, light users will not have to wait behind a huge queue of jobs that were queued up by a heavy user.

Generic Backend and New Gem

We moved a bunch of the logic that was on the backend into the gem, such as loading other files and gems and setting up database connections. The backend will simply take what it gets and execute it (well it’s not quite that simple).

Why is this good you might ask? Well there are several things:

1. You no longer have to write your code following the same pattern we use to enforce. You could write any ruby code in any way, package it up, and as long as we can unpackage it and execute a file from the package, you’re good to go. This means someone else could write a new gem that works in a different way if they’d like (challenge: feel like making something better and easier to use than our official gem??).  We’ll be releasing API docs soon that will explain exactly how to do this.
2. A lot of bugs, issues and new features can be fixed in the gem (ie: client side) which means we can fix things faster and take contributions from the community.
3. This leads to multi-language support, stay tuned for more on this in the near future!

Be sure to get the latest gem to take advantage of these changes, aptly versioned 1.0.X, so do a gem update or if you don’t have it yet, sudo gem install simple_worker.

That’s all she wrote for this round. Enjoy and let us know what you think or if you have any questions or problems.