Blog

Search This Blog

Loading...

Thursday, November 1, 2012

Sharable, Open Source Workers for Scalable Processing

You can now add workers to your account from a github URL which enables an awesome way for people to share their workers with others. In this post, I'll show you three real world examples workers that you can start using right away without writing a line of worker code.

The power of any worker system is splitting up your work into discrete tasks that can run in parallel. Splitting up your work means you start by running one job/task and scale to millions of tasks without changing the worker since it just does the same thing over and over and the work can be distributed across any number of machines. These examples are very common use cases that should give you an idea of how powerful workers can be and how awesome it is that they can be shared with other IronWorker users.

The Basic Steps

All shared workers have two steps:



1. Add the worker to your account using our command line tool: iron_worker upload URL_TO_WORKER_FILE

2. Start queuing up tasks for the worker. You queue up tasks by simply making an HTTP request to the IronWorker API and passing in a payload containing the inputs the worker expects which is generally a JSON object. For instance, the payload for the email example below looks like this:


Getting Started

If you don’t have an Iron.io account, sign up for a free account and then configure your credentials after you've signed up so you can use the command line tools we'll be using in this article.

You'll also need the IronWorker gem installed too so assuming you have Ruby on your system:
$ gem install iron_worker_ng
Ok, now let's get to the fun part.

Hello Worker

Let's try a simple hello world/worker example first, this one doesn't do much, but it's quick and easy to try so you can get a feel for this.

1) The first thing we need to do is add the worker to our Iron.io account.
$ iron_worker upload https://github.com/treeder/hello_worker/blob/master/hello.worker

2) Now that it's in our account, we can start queuing up jobs/tasks using the IronWorker API, which is simply an HTTP request with a payload so we can do it from any language or even curl:
$ curl -d '{"tasks": [{"code_name": "hello", "payload": "{\"foo\": \"bar\"}"}]}' --header "Authorization: OAuth YOUR_TOKEN" --header "Content-Type: application/json" https://worker-aws-us-east-1.iron.io/2/projects/YOUR_PROJECT_ID/tasks
And here's how to queue up tasks in Ruby:
require 'iron_worker_ng'
client = IronWorkerNG::Client.new()
client.tasks.create(
    'hello', {ruby_foo: "bar"}
)
Now go look at your task list in HUD to see it run and view the log.

You can do the same with any other languages too and we have a bunch of native language clients for most popular languages.

If you want to modify the worker, here's the code, go ahead and fork it and change it to do what you want: https://github.com/treeder/hello_worker.

Image Processing Worker

If you want to build the next Instagram, you need an image processing solution that scales. IronWorker is perfect for scalable image processing. And with this open source worker, you can start using it right away and never have to worry about scaling. 

1) Add the worker to your account
$ iron_worker upload https://github.com/treeder/image_processing_worker/blob/master/image_processor.worker
2) Queue up tasks for the worker

This is a Ruby example, but again you can use any language as it's just an http request with a payload for the worker. This worker required a payload containing an image url, aws credentials to store the transformed images into s3 and which image transformation operations we want to perform.
require 'iron_worker_ng'

# Create the worker payload which has all our image manipulation functions we want to perform.
# See enqueue.rb for more operations.
payload = {
    aws: {
        access_key: "MY AWS ACCESS KEY",
        secret_key: "MY AWS SECRET KEY",
        s3_bucket_name: "MY BUCKET NAME"
    },
    image_url: "http://dev.iron.io/images/iron_pony.png",
    operations: [
        {
            op: 'sketch',
            format: 'jpg',
            destination: "sketch.jpg"
        }
    ]
}

client = IronWorkerNG::Client.new()
client.tasks.create(
    'image_processor', payload
)
This will apply the pencil sketch transformation using ImageMagick and if you look at your log in hud, you'll see the S3 URL so you can check out the processed images. Again, feel free to fork the worker and modify it to your liking, here's the code: https://github.com/treeder/image_processing_worker. Pull requests welcome!

Email Worker

Sending emails is another common task for background processes because sometimes your SMTP provider isn't very fast and you don't want your users to wait for it. Using IronWorker, you can queue up the email very quickly and respond to your user quickly.

1) Add the worker to your account

iron_worker upload https://github.com/treeder/email_worker/blob/master/email.worker
2) Queue up tasks for the worker

require 'iron_worker_ng'
client = IronWorkerNG::Client.new()
client.tasks.create(
    'email', {
        host: "smtp.sendgrid.net",
        domain: "MY DOMAIN eg: iron.io",
        port: 587,
        username: "MY USERNAME",
        password: "MY PASSWORD",
        from: "MY EMAIL",
        to: "EMAIL TO SEND TO",
        subject: "SUBJECT",
        body: "BODY"
    }
)
Here's the code for this worker: https://github.com/treeder/email_worker

Conclusion

So there you have it, three workers can you use right now to help you build a scalable system. I hope this sparked your interest enough for you to go out and make your own worker and share it with the world!  And if you do, let us know and we'll be sure to share it with our users too.