Parallelizing Ruby on the Cloud

So I'm sure we've all had the need to want to run multiple threads at once to optimize a part of our applications. There are several options like spinning up new Ruby threads (example from):

pages = %w( www.rubycentral.com
            www.awl.com
            www.pragmaticprogrammer.com
           )

    threads = []

    pages.each do |page|
        threads << Thread.new(page) { |myPage|
            h = Net::HTTP.new(myPage, 80)
            log "Fetching: #{myPage}"
            resp, data = h.get('/', nil )

# DO SOME PAGE PROCESSING AND STORE STUFF IN DATABASE
        }
    end

    threads.each { |aThread|  aThread.join }

Or you could use a library like delayed_job that works great after a bit of database configuration and starting up the job process from the command line.

But now there is another option that is actually easier and much more scalable and that is sending jobs off into the cloud using SimpleWorkr. Here is an example of the above using SimpleWorkr:

# First we create our Worker
class WebPageWorker < SimpleWorker::Base
    attr_accessor :url
    def run

h = Net::HTTP.new(myPage, 80)
        log "Fetching: #{myPage}"
        resp, data = h.get('/', nil )

# DO SOME PAGE PROCESSING AND STORE STUFF IN DATABASE

    end
end


# Now lets fire some of those workers up!
pages = %w( www.rubycentral.com
    www.awl.com
    www.pragmaticprogrammer.com
)



pages.each do |page|    worker = WebPageWorker.new

worker.page = page

worker.queue

end

That's it. Easy like Sunday morning.

Including other files in your Worker with "merge"

Previously, your worker file/class had to be self contained (ie: single file including all code for the worker).

Now you can use the new "merge" class method for instance:

class TestWorker2 < SimpleWorker::Base
  
         merge "models/model_1.rb"


This will automatically upload the merged files to the SimpleWorker service and 'require' them in your Worker class.

If you want to use the special methods like "log" in your merged classes, use the SimpleWorker::UsedInWorker mix-in, for instance:

class Model1

    include SimpleWorker::UsedInWorker


That's about it, the rest will be taken care of for you.

If you want to kick off other workers from within the first worker, use merge_worker instead:

class TestWorker2 < SimpleWorker::Base
   
         merge "models/model_1.rb"
         merge_worker "my_other_worker.rb", "MyOtherWorker" 
         # second parameter is the class name

The merge_worker does things a bit differently so it can be handled properly after uploading to the worker cloud.

Pushed 0.3.0 Gem: Much More User Friendly

After using SimpleWorkr for a long time now, I always think to myself, "there's gotta be some way to make this more intuitive". I want it to feel more like it's just like running your normal code without even really realizing that the work is actually offloaded to the cloud. So with that in mind, the gem now has a much more intuitive usage pattern, more like ActiveRecord.

The old way:


# Create a Service object
@service = SimpleWorker::Service.new(@access_key, @secret_key)
# Upload code
@service.upload(worker_class_name, path_to_file)
# Queue up jobs
@service.queue(worker_class_name, data_for_job)
# Or schedule jobs
@service.schedule(worker_class_name, data_for_job, schedule)
# Get status of jobs
@service.status(task_id)


Now, here's the shiny new way of doing it:

Some one time initialization:


SimpleWorker.configure do |config|
      config.access_key = @access_key
      config.secret_key = @secret_key
end


Now throughout your app, you can do the following:


# Instantiate a new worker
worker = MyWorker.new
# Queue directly from the object
worker.queue
# Schedule directly from the object
worker.schedule(my_schedule)
# Check status directly from it as well
worker.status


All data defined in instance variables in your worker will be sent along and usable in your run method.

Also, your code will automatically be uploaded if it has been changed!