Blog

Search This Blog

Loading...

Sunday, March 30, 2014

Using Iron.io Workers for Large Scale PHP posting with Laravel (a repost from Scott Wilcox)

Scalable Workers in Laravel
One of the fun parts about working at Iron.io is being able to connect with developers working on some great projects. 

One of these is Scott Wilcox, the developer of Tweekly.fm. His service posts Last.fm events to Twitter and has been growing in popular the last several months. Scott wrote a great post the other day on the growth of his service and the things he did to help scale the processing. 

Scott built his service on Laravel, a popular and growing PHP framework. He used Iron.io's IronWorker service in conjunction with Laravel to power and scale the posting to Twitter. 

Below is an excerpt of his post. If you're interested in ways to build something to be able to scale easily, it's definitely worth reading.
Around October 2013 I discovered Iron.io almost by accident. I'd recently begun rewriting Tweekly.fm into the excellent Laravel framework. I was testing Laravel 4's queuing systems and noticed a reference to Iron.io. After reading more into the IronMQ product, I came across IronWorker. 
The difference IronWorker provided for Tweekly.fm cannot be understated. It allows us the create updates, package them up to be sent and then queue en-masse into an IronWorker queue. These are then processed in batches and an entire days updates can be sent out in a matter of minutes. 
Sunday is the busiest day of the week for Tweekly.fm. Regularly for a year now, we've been pushing out over 200,000 updates. That's 8,333 updates an hour or 138 a minute. This would take over 24 hours sequentially, around 18 hours with multiple curl calls and takes just over 40 minutes with IronWorker at a fraction of the cost. 
I was able to remove one of the servers and save on the hosting cost – this alone reduced our costs by half. 
The exceptional service, support and price is worth it alone. Mix that in with the fact costs were halved - I'm not too sure how you can look anywhere else when needing to run PHP workers for your large scale projects.



To read the full post, visit Scott's blog here.

To check out the Iron.io platform, sign up for a free account.

To learn more about using the Iron.io platform with Laravel, take a look at these resources:


Friday, March 14, 2014

Iron.io Enhances Web and Mobile Capabilities with CORS Support

Iron.io adds CORS support
Iron.io is pleased to announce the introduction of support for Cross-origin Resource Sharing (CORS). This capability means that developers have even easier access to the Iron.io platform from desktop and mobile browsers and other client devices.

CORS is an HTML5 feature that supports cross-domain communication across web clients and server-side domains. Without CORS, a webpage cannot access another domain, either to get data or to access a service. (Cross-domain requests are typically forbidden by web browsers under the same origin security policy.)


Why CORS Matters with the Iron.io Platform

All common browsers support CORS and so adding it to the Iron.io APIs means that developers can make calls to IronMQ, IronWorker, and IronCache from webpages and other JavaScripts clients in a very simple and secure manner. 

(An alternative to CORS is JSONP but that’s limited in that it is restricted to GET requests which means other methods such as POST, PUT and DELETE are not available.)

The use cases for connecting mobile and desktop clients to Iron.io are pretty clear. Client-side components can put messages on IronMQ, run tasks in IronWorker, or put data in IronCache without having to hit another server (and have that server hit the Iron.io APIs). This greatly streamlines app development in that you can send work to the background – whether it's to execute within your own system or to push to another service.

EnableCORS website
Iron.io handles the scaling and execution of the tasks or the pushing of the message to another endpoint. Developers simply have to make calls to the Iron.io APIs and they get production-scale message queuing, task processing, and key/value data storage. Here's an example of how you might use CORS to connect to the Iron.io platform.


How CORS Works

Specifically, CORS builds on top of the XMLHttpRequest object and allows JavaScript within a webpage or mobile app to make XMLHttpRequests to other domains. The browser adds these headers, and sometimes makes additional requests, during a CORS request on behalf of the client. The server responds in kind. 

Much of the complexity for CORS is handled by the browser and the server. As a client-side developer, you can access these new headers but are generally shielded from majority of the details on how CORS works. For deeper background, however, you can check out an article by Monsur Hossain on Using CORS. It shows how to configure clients to make cross-origin requests along with an example of an end-to-end request.

Browser Support

Here is a table of the various desktop and mobile browsers and the levels of support for CORS. 



You can view the full table of browser versions here.


Making a CORS Request for the Iron.io Platform

This section shows a simple example of how to make a cross-domain request in JavaScript to IronMQ. The project ID and OAuth token provide authentication.

  // Create the XHR object.
  function createCORSRequest(method, url) {
    var xhr = new XMLHttpRequest();
    xhr.open(method, url, true);
    return xhr;
  }
   
  // Insert your iron.io credentials here:
  var projectId = 'YOUR PROJECT ID';
  var queueName = 'YOUR QUEUE NAME';
  var token     = 'YOUR TOKEN';
  var url = 'https://mq-aws-us-east-1.iron.io/1/projects/' + projectId + '/queues/' + queueName + '/messages?oauth='+ token;
   
  // create a request and set content-type header
  var xhr = createCORSRequest('POST', url);
  // missing content-type header will lead to a 406 response
  xhr.setRequestHeader("Content-Type", "application/json");
   
  // Parse the response in the format of a JSON string
  xhr.onload = function() {
    var text = xhr.responseText;
    var resp = JSON.parse(text)
    console.log("My message is: ", resp);
  };
   
  xhr.onerror = function() {
    console.log('Woops, there was an error making the request.');
  };
   
  // Stringify your JSON request body
  xhr.send(JSON.stringify({"messages":[{"body":"your message"}]}));


Similar approaches can be used for queuing up a worker or putting an item into a cache in Iron.io except with different endpoints, worker/cache names, methods, and payloads.

Note: The above example is primarily intended for as a demonstration given that the project ID and token are exposed within the JavaScript code. We will be creating additional resources on how to connect in a more secure manner.



Visit the Iron.io Dev Center for a complete list of API methods for the services.

IronMQ API reference








To learn more about how Iron.io can help you process messages, run tasks and store key/value pairs using from desktop and mobile clients, visit Iron.io today.


Wednesday, March 12, 2014

Iron.io Launches Custom Runtime Environments for IronWorker

Custom Runtime Language Environments in IronWorker
Iron.io announced today the introduction of custom language environments within its IronWorker processing platform.

Instead of a single standard environment, developers can now define runtime environments and write workers for specific language versions.

IronWorker already supports all common languages including PHP, Ruby, Python, .NET, Node.js, Java, Go and binary files, but this release adds in finer grained support for specific language versions including Ruby 2.1, Python 3.2, Mono 3.0, and others. (The full table can be seen below.)

Greater Flexibility and Increased Reliability 

IronWorker is a highly available worker service that provides background processing, scheduled jobs, and concurrent scale-out processing. Worker systems are critical within almost every production-scale cloud application. Whether it’s processing large amounts of data, processing event streams, or handling individual jobs in the background, worker systems allow applications to scale more easily and system components to operate more independently.

User-definable runtime worker environments solve a number of problems with most worker systems. Developers needing the latest language versions can now have access to them as soon as they become available. At the same time, developers with existing workers can lock down their environments to use specific versions so as to maintain consistency.

Custom language environments let Iron.io serve both of these needs – allowing Iron.io to offer the most current environments while still providing a reliable and stable platform for production-scale systems.

Another advantage of custom runtime environments is that developers no longer have to limit themselves to a single compute stack. They can change environments worker by worker. As a result, a single application can have workers using different languages and different versions – with zero installation and zero ops. This reduces the risk of monolithic apps where language upgrades can take months, or in some cases, years. Using a flexible and scalable worker system like IronWorker means applications can be more loosely coupled and distributed and therefore easier to scale and easier to maintain.


New IronWorker Runtime Environments 

In addition to the existing language environments, the custom language versions now available in IronWorker include:

  • Ruby 1.9
  • Ruby 2.1
  • Python 2.7
  • Python 3.2
  • PHP 5.4
  • Node 0.10
  • Java 1.7
  • Scala 2.9
  • Mono 2.10
  • Mono 3.0

Additional environments can be created relatively easily. See below for how to get in touch with us if you're interested in environments not listed here or in the default environment.


How to Make Use of Custom Runtime Environments in IronWorker

To make use of a custom worker environment in IronWorker, all you need to do to is add a single line to your .worker config file. (The .worker file is included as part of the worker upload process and tells IronWorker what the executable file is, what code packages and data files are included, whether to build the worker remotely, as well as a number of other options.)

Note: For the .worker file, a worker name needs to be included with the .worker suffix. For example, sendmail_worker.worker would create a worker in the system called "sendmail_worker".


To specify a particular custom environment for your worker, add the language/version parameter as a line in your .worker file.


To use a Ruby 2.1 environment, include the following in your .worker file:

 stack 'ruby-2.1'

To specify a Java 1.7 environment, include the following:

stack 'java-1.7'


To get list of available stacks via the CLI just type in the following command:

iron_worker stacks

You'll get back a list along the lines of the following:

["scala-2.9","ruby-2.1","ruby-1.9","python-3.2","python-2.7","php-5.4",
 "node-0.10","java-1.7","mono-3.0","mono-2.10"]

Note: Make sure you update the IronWorker CLI first though.

gem update iron_worker_ng


That’s it!


Simple Addition → Powerful Results

With a simple addition to a config file, developers get custom runtime language environment running within a secure sandbox and with the same features customers love – including retries, priority settings, real-time logging, dashboard monitoring, and more.

We believe custom language environments are a huge step forward in bringing you the most powerful and flexible worker system in the world. Try it out and let us know what you think.






Behind the Scenes – Using Docker 
We're using Docker behind the scenes to create and manage the containers for the custom language environments. It provides a number of powerful capabilities including the ability to automatically assemble containers from source code and control application dependencies. We'll post an article in the near future detailing our experiences with Docker. (In short, we’re fans.) If you’d like to know more about our use of Docker, please subscribe to the Iron.io newsletter.





To Get Started

To give IronWorker custom environments a try, sign up today for free account. It’s simple to get started and powerful enough for even the heaviest demands. Go here to signup.


To Create Your Own Custom Environment

If you are operating at production-scale or will be soon and need one or more custom environments, please reach out to our sales and support teams. Contact us at 1-888-939-4623 or send us details at support@iron.io.