Sharable, Open Source Workers for Scalable Processing

By Iron | November 1, 2012 |

Overview

You can now add workers to your account from a GitHub URL which enables an awesome way for people to share their workers with others. In this post, I'll show you three real-world examples workers that you can start using right away without writing a line of worker code.

The power of any worker system is splitting up your work into discrete tasks that can run in parallel. Splitting up your work means you start by running one job/task and scale to millions of tasks without changing the worker since it just does the same thing over and over and the work can be distributed across any number of machines. These examples are very common use cases that should give you an idea of how powerful workers can be and how awesome it is that they can be shared with other IronWorker users.

The Basic Steps

All shared workers have two steps:

1. Add the worker to your account using our command-line tool: iron_worker upload URL_TO_WORKER_FILE

2. Start queuing up tasks for the worker. You queue up tasks by simply making an HTTP request to the IronWorker API and passing in a payload containing the inputs the worker expects which is generally a JSON object. For instance, the payload for the email example below looks like this:

Getting Started

If you don’t have an Iron.io account, sign up for a free account and then configure your credentials after you've signed up so you can use the command line tools we'll be using in this article.

You'll also need the IronWorker gem installed too so assuming you have Ruby on your system:

$ gem install iron_worker_ng

Ok, now let's get to the fun part.

Hello Worker

Let's try a simple hello world/worker example first, this one doesn't do much, but it's quick and easy to try so you can get a feel for this.

1) The first thing we need to do is add the worker to our Iron.io account.

$ iron_worker upload https://github.com/treeder/hello_worker/blob/master/hello.worker

2) Now that it's in our account, we can start queuing up jobs/tasks using the IronWorker API, which is simply an HTTP request with a payload so we can do it from any language or even curl:

$ curl -d '{"tasks": [{"code_name": "hello", "payload": "{"foo": "bar"}"}]}' --header "Authorization: OAuth YOUR_TOKEN" --header "Content-Type: application/json" https://worker-aws-us-east-1.iron.io/2/projects/YOUR_PROJECT_ID/tasks

And here's how to queue up tasks in Ruby:

require 'iron_worker_ng'
client = IronWorkerNG::Client.new()
client.tasks.create(
    'hello', {ruby_foo: "bar"}
)

Now go look at your task list in HUD to see it run and view the log.

You can do the same with any other languages too and we have a bunch of native language clients for the most popular languages.

If you want to modify the worker, here's the code, go ahead and fork it and change it to do what you want: https://github.com/treeder/hello_worker.

Iron.io Serverless Tools

Speak to us to learn how IronWorker and IronMQ are essential products for your application to become cloud elastic.

Image Processing Worker

If you want to build the next Instagram, you need an image processing solution that scales. IronWorker is perfect for scalable image processing. And with this open-source worker, you can start using it right away and never have to worry about scaling.

1) Add the worker to your account

$ iron_worker upload https://github.com/treeder/image_processing_worker/blob/master/image_processor.worker

2) Queue up tasks for the worker

This is a Ruby example, but again you can use any language as it's just an HTTP request with a payload for the worker. This worker required a payload containing an image URL, AWS credentials to store the transformed images into s3, and which image transformation operations we want to perform.

require 'iron_worker_ng'

# Create the worker payload which has all our image manipulation functions we want to perform.
# See enqueue.rb for more operations.
payload = {
aws: {
access_key: "MY AWS ACCESS KEY",
secret_key: "MY AWS SECRET KEY",
s3_bucket_name: "MY BUCKET NAME"
},
image_url: "http://dev.iron.io/images/iron_pony.png",
operations: [
{
op: 'sketch',
format: 'jpg',
destination: "sketch.jpg"
}
]
}

client = IronWorkerNG::Client.new()
client.tasks.create(
'image_processor', payload
)

This will apply the pencil sketch transformation using ImageMagick and if you look at your log in hud, you'll see the S3 URL so you can check out the processed images. Again, feel free to fork the worker and modify it to your liking, here's the code: https://github.com/treeder/image_processing_worker. Pull requests welcome!

Email Worker

Sending emails is another common task for background processes because sometimes your SMTP provider isn't very fast and you don't want your users to wait for it. Using IronWorker, you can queue up the email very quickly and respond to your user quickly.

1) Add the worker to your account

iron_worker upload https://github.com/treeder/email_worker/blob/master/email.worker

2) Queue up tasks for the worker

require 'iron_worker_ng'
client = IronWorkerNG::Client.new()
client.tasks.create(
    'email', {
        host: "smtp.sendgrid.net",
        domain: "MY DOMAIN eg: iron.io",
        port: 587,
        username: "MY USERNAME",
        password: "MY PASSWORD",
        from: "MY EMAIL",
        to: "EMAIL TO SEND TO",
        subject: "SUBJECT",
        body: "BODY"
    }
)

Here's the code for this worker: https://github.com/treeder/email_worker

Conclusion

So there you have it, three workers can you use right now to help you build a scalable system. I hope this sparked your interest enough for you to go out and make your own worker and share it with the world! And if you do, let us know and we'll be sure to share it with our users too.

Unlock the Cloud with Iron.io

Find out how IronWorker and IronMQ can help your application obtain the cloud with fanatical customer support, reliable performance, and competitive pricing.

Get Iron

Posted in Serverless Message Queue and tagged IronWorker

About Iron

View all posts by Iron →

2 Comments

Justin England on December 1, 2012 at 6:58 pm

It would be nice if you could support git remote, I am not quite sure what benefit this adds if you still use the iron_worker client to push. I still have to install the iron_worker on all the development machines.

It should be as simple as
$ git push iron-io yourbranch:master
Travis Reeder on December 11, 2012 at 1:57 am

Hi Justin,

We plan on removing the iron_worker gem dependency, but it will probably be curl/api based rather than git based. Coming soon…