Blog

Search This Blog

Loading...

Monday, January 31, 2011

Every Web Application Needs a Background Processing Queue

Every web application needs a queue to handle background jobs. Background jobs are simply that -- jobs that run in the background, outside of the main web response loop.

Great effort goes into making web page response as fast as possible, which means getting data to the screen, completing the request, and returning control to the user. Background jobs are created to handle tasks that take time to complete or that aren't critical to displaying results on the screen.

For example, if a query might take longer than a second, developers will want to consider running it in the background so that the web app can respond quickly and free itself up to respond to other requests. If needed, the background job can call back to the webpage when the task has been completed.

Many things that rely on external services are also suited for running as background jobs. Sending an email as confirmation, storing a photo, creating a thumbnail, or posting to social media services are jobs that don't need to be run in the front as part of the web page response. The controller in the application can put the email job, image processing, or social media posts into a delayed job queue and then return control to a user.

Jobs that run on a schedule are also considered background tasks. Also known as Cron jobs, scheduled jobs often get queued up to run on the same machine doing the tasking or on the local network, primarily because creating a truly distributed background queue is a complicated process.

Cloud computing greatly streamlines the process so that Cron jobs can run on servers far removed from the ones responding to web requests. Workers for the scheduled jobs can still be developed in the application code, but queued up from the application or from outside scripts to run in the cloud. (Which brings up the point that background queues are not only the logic to create the queue but also the servers to run the jobs on. SimpleWorker combines both -- a full-featured queuing framework and industrial-strength servers -- to make running background jobs as simple as possible.)

Even a basic or newly launched web application needs to respond quickly and do things on a schedule. Background queues are an essential component to making this happen. Fortunately, there are many options for creating them including pre-built services like SimpleWorker.


Monday, January 3, 2011

Autoload Rails Models for Super Easy Rails Integration

SimpleWorker will now include all the models in your Rails app automatically so you don't have to merge them manually. Just use them almost as if they were to be run locally and you're good to go.

eg;

class MyWorker < SimpleWorker::Base
  attr_accessor :user_id
  def run
    # User model will automatically be available, so just use it as you normally would.
    user = User.find(user_id)
    .....
  end
end