Every Web Application Needs a Background Processing Queue


Every web application has crucial tasks that need to run in the background. They run outside of the main web response loop, but ensure any number of critical jobs gets done to keep the web-based application running smoothly. This means end users enjoy an engaging app experience, without it grinding to a halt every time it needs to do something extra. Let's take a look at what happens in the background of web applications, and how IronWorker and the Docker environment can help simplify app development and deployment.

Table of Contents

Need to scale your background jobs?

Speak to us to find out how easy it is to process background jobs at scale using IronWorker with free handeld support.

Keeping it Fast

The average web browser will persevere with a web page for just 15 seconds before clicking away. That’s why it’s so important that the page loads quickly and with no glitches. Data needs to get to the screen, requests need to be completed, and the user needs to feel in control. Background jobs are created to handle tasks that take time to complete or that aren’t critical to displaying results on the screen.

For any query that might take longer than a second, developers should consider running it in the background so that the web app can respond quickly and free itself up to respond to other requests. If needed, the background job can call back to the webpage when the task has been completed.


External Services

Many tasks that rely on external services are also suitable for running as background jobs. Sending a confirmation email, storing a photo, creating a thumbnail, or posting to social media platforms are jobs that don’t need to be run in the front end of the web page response. The controller in the application can put the email, image processing, or social media post job into a delayed job queue, then return control to the user as quickly as possible.

Using a background process to send an email to a customer means less waiting around for that customer, and a boost to your business’s brand reputation. As IronWorker uses Docker containers, you could use any programming language you wanted for this. This example shows one way to set up sending an email as a background task using the open-source programming language, Ruby.

The first step is to create a Gemfile in a new directory, and input the following:

source 'https://rubygems.org'

gem 'sendgrid-ruby'

gem 'iron_worker', '>= 3.0.0'

You can then create a Ruby file, send_email.rb, that will send emails as a background task for you:

require_relative 'bundle/bundler/setup'

require 'iron_worker'

require 'sendgrid-ruby'

include SendGrid

from = Email.new(email: IronWorker.payload["from"])

to = Email.new(email: IronWorker.payload["to"])

subject = IronWorker.payload["subject"]

content = Content.new(type: 'text/plain', value: IronWorker.payload["body"])

mail = Mail.new(from, subject, to, content)

sg = SendGrid::API.new(api_key: ENV['SENDGRID_API_KEY'])

response = sg.client.mail._('send').post(request_body: mail.to_json)

puts response.status_code

puts response.body

puts response.headers

Now it’s time to create a payload.json file. This contains the actual data that gets sent to the Ruby program.


"from": "test@example.com",

"to": "you@example.com",

"subject": "It works!",

"body": "Hello from IronWorker"


The next step is installing the dependencies using Docker, zipping up the code, and uploading it to Iron.

This example uses a Ruby-based Docker image from Iron.io itself, to make things simpler. More experienced developers can use their own images or source images elsewhere, like Docker hub.

> docker run --rm -v "$PWD":/worker -w /worker iron/ruby:dev bundle install --standalone --clean

> zip -r email_worker.zip .

> iron worker upload -e SENDGRID_API_KEY=YOUR_SENDGRID_API_KEY --name email_worker --zip email_worker.zip iron/ruby ruby send_email.rb

This code, once uploaded to Iron, works with IronWorker to send email whenever required. The instruction o run the task could be as follows:

> iron worker queue --payload-file payload.json --wait email_worker

Tasks such as these are not limited by email server or provider, and will work on any system or in any environment thanks to the Docker architecture.

IronWorker Processes Background Jobs At Scale

Learn how IronWorker can run thousands of tasks in parallel and schedule jobs easily from with your applications.


Scheduled Jobs

Jobs that run on a schedule are usually considered background tasks. Scheduled jobs often get queued up to run on the same machine doing the tasking or on the local network, primarily because creating a truly distributed background queue is a complicated process.

Scheduled tasks are sometimes also called cron tasks because the cron service tends to run the automated background tasks in a Unix or Linux environment. Docker uses a Linux kernel, even though Docker containers will work on any system, so the cron service is still used for that automation and scheduling.

Various workers have their own ways of scheduling background tasks, but the most efficient will use the tools that are already there without adding to the memory used or complicating requests.

Cloud computing greatly streamlines the process so that scheduled jobs can run on servers far removed from the ones responding to web requests. Separating tasks out into Docker containers has the same effect, as each container can hold a single task's code and its dependencies without interfering with other tasks or job queues. Workers for the scheduled jobs can still be developed in the application code, but queued up from the application or from outside scripts to run in the cloud. IronWorker combines both the logic to create these queues and the servers to run the jobs on, making it a full-featured queuing framework for running background jobs as easily as possible.

Even the simplest web application needs to respond quickly and do things on a schedule. Background queues are an essential component to making this happen. Fortunately, there are many options for creating them including pre-built services like IronWorker.

IronWorker Runs Your Background Jobs

Find out why IronWorker can process your background jobs at scale with fanatical support.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.