Blog

Search This Blog

Loading...

Tuesday, January 22, 2013

ProQuest's Scalable, No Maintenance Cloud Stack

ProQuest chose IronMQ as a key part of their stack while building the website for WorldRiderz, a documentary on the Discovery Channel. They needed a robust, elastic and scalable architecture. Due to the nature of the project, traffic was always going to be patchy, with periods of heavy traffic, followed by idle periods, so controlling costs and being able to scale quickly was important. IronMQ was used to free the front end Heroku dynos from any heavy back end processing, essentially decoupling the front end from the back end.

"I've been around the developer block for 15 years and this combination of technologies rocks." said Fergus Kelledy, Solutions Architect for ProQuest. “Since deployment and go live WorldRiderz has experienced 100% uptime and 0% data loss.”

“The reason we were keen to create a very robust infrastructure for WorldRiderz was because we knew that none other than Lance Armstrong was going to tweet the website to his [4 million] followers.” said Clive Roberts of ProQuest.

The Cloud Stack:

More at Heroku.

Wednesday, January 16, 2013

Common Actions on Messages in a Queue

In a previous post, we talked about our ten favourite uses for a message queue. The uses laid out the architectural benefits of putting a message queue in place as a key component within a distributed system.

The writers at HighScalability did us one better with their description of a message queue in their repost of the article. Here’s how they described it:
"[Y]ou can find a message queue in nearly every major architecture profile on HighScalability. Historically they may have been introduced after a first generation architecture needed to scale up from their two tier system into something a little more capable (asynchronicity, work dispatch, load buffering, database offloading, etc). If there's anything like a standard structural component, like an arch or beam in architecture for software, it's the message queue."
In this piece, we wanted to take a different route and talk about some of the concrete things you can do with a queue. It’s helpful to talk about separating components, creating non-blocking flows, and arriving at an independent, loosely-coupled architecture, but that still leaves things a bit too abstract. Talking about message and event flow will hopefully provide a bit more grounding.

We’ve targeted this post at cloud developers who may not have used message queues very much but who are looking for ways to create more robust applications or connect systems together. Other developers may find it useful, though, as a quick refresher. Think of it as Message Queues 201.


Friday, January 11, 2013

IronMQ Push Queues - Reliable Message Delivery for the Cloud

IronMQ Push Queues adds notification capabilities to your queues allowing you to send messages to other systems in a scalable, flexible and cost-effective way. You simply add subscribers, which are HTTP endpoints, to your IronMQ queues then any message you post to your queue will be reliably delivered to the subscribers.

Why Push Queues?


There are many reasons where pushing a message is useful, here are a few big ones.

Let your application do the work

Instead of having processes running that pick up messages from your queue for processing, you can let your application process the queue messages simply by defining an endpoint/route in your application. When your queue receives a message, IronMQ will post the message to your application endpoint, then your application can process it. This makes it really easy to build workers since you don't have to think about much beyond your application. 

Monday, January 7, 2013

Peek, Touch and Release

IronMQ now supports three new operations: peek, touch and release. These operations give you greater control over handling your messages and here's a brief explanation of each.

A little background on the message lifecycle


First off, let's discuss the typical message lifecycle.
  1. POST message to a queue. 
  2. GET message from the queue - this reserves it.  
  3. DELETE message from the queue once you're done with it.
During the 2nd phase of the lifecycle, the GET phase, a message is reserved for a period of time defined by the "timeout" parameter (default is 60 seconds). If the message is not DELETEd before the end of the timeout, it will be placed back on the queue and become available again for the next GET.

Peek

Peek allows you to get messages off the queue without reserving them. Peek will not reserve the message and it will therefore remain available for the next peek or get operation. You can use this if you need to see if any messages are available.

Docs for peek.

Touch

Touching a reserved message extends its timeout by the duration specified when the message was created / POSTed. This is useful if your processing time is taking longer than expected and you want some more time to finish.

Docs for touch.

Release

Releasing a reserved message un-reserves the message and puts it back on the queue as if the message had timed out. This is useful if for some reason you can't process the message right away, you can put it back on the queue for another process to pick up. You can also supply a "delay" parameter.

Docs for release.


Wednesday, January 2, 2013

Queue Webhook Events with IronMQ Webhook Support

You can now use IronMQ as a webhook endpoint to collect webhook events/messages from third parties. This new feature supports arbitrary HTTP POSTs to an IronMQ queue and it will store the body of the POST in your queue.

Now why would you want to do this? There's plenty of good reasons, here are a few:

  • Ensure reliable delivery - if you need to ensure that you get each and every webhook message, you can use IronMQ to ensure that no message gets lost and you can guarantee that the message gets processed before it gets removed from the queue. 
  • Collect webhook events and deal with them later such as in a nightly batch run. 
  • Use multiple queues to collect messages for different processes/systems
  • Don’t worry about your service getting bombarded with webhook requests that it has to deal with synchronously, your workers can grab stuff off the queue when they’re ready and available.  

Let's Try It!


To use IronMQ webhooks, you just need to get your webhook URL for your queue.

https://mq-aws-us-east-1.iron.io/1/projects/MY_PROJECT_ID/queues/MY QUEUE NAME/messages/webhook?oauth=MY_TOKEN

Replace MY_PROJECT_ID and MY_TOKEN with a project id and token from HUD. Replace MY_QUEUE_NAME with whatever you want to name the queue.

Let’s quickly post something to our webhook endpoint with curl:
curl -d 'I love queues' \ 
"https://mq-aws-us-east-1.iron.io/1/projects/MY_PROJECT_ID/queues/curl-webhook-queue/messages/webhook?oauth=MY_TOKEN"

Replace MY_PROJECT_ID and MY_TOKEN with your project id and token then run that command. Go ahead, give it a try.

Now let’s grab that message off the queue:
curl \
"https://mq-aws-us-east-1.iron.io/1/projects/MY_PROJECT_ID/queues/curl-webhook-queue/messages?oauth=MY_TOKEN"
(notice I removed “webhook” from the URL, otherwise it’s the exact same as above).

You should get a response like this:
{"messages":[{"id":"5828196657047927116","body":"I love queues","timeout":60,}]}

Now if we want to acknowledge that we've handled the message, we need to delete it (if we don't delete it, it will go back on the queue after a timeout, this ensures messages have been processed). Copy the message id you got in the response above and use it in a curl delete:
curl -X DELETE \
"https://mq-aws-us-east-1.iron.io/1/projects/MY_PROJECT_ID/queues/curl-webhook-queue/messages/5828196657047927116?oauth=MY_TOKEN"

Now that we've tried posting an arbitrary http post to our queue, let’s try it from a real service.

Example: Posting GitHub Service Hooks to IronMQ


Let's try a real world example using GitHub service hooks. All we need to do is add the queue's webhook URL to the Webhook URLs section of GitHub service hooks. Go to your GitHub repository, click "Settings", then click "Service Hooks". Change the QUEUE_NAME in your queue URL to github-webhook-queue to keep it separate from our tests above, so the new URL would be:
https://mq-aws-us-east-1.iron.io/1/projects/MY_PROJECT_ID/queues/github-webhook-queue/messages/webhook?oauth=MY_TOKEN

Paste that URL into the GitHub interface:

Click "Update Settings" then test it by clicking "Test Hook". Since there's not much of a response in the GitHub interface when you do this, you can check to see that you got a message by looking at your queues list in HUD.  If it says 1, then it's all good so let’s grab the message from the queue using curl.
curl https://mq-aws-us-east-1.iron.io/1/projects/MY_PROJECT_ID/queues/github-webhook-queue/messages?oauth=MY_TOKEN

The response you get will look pretty ugly because unfortunately GitHub posts a big chunk of json as an HTTP form parameter called "payload" so we have to decode it first. Here’s a Ruby example to decode and get the proper values:

For more information on what's contained in the GitHub payload, check here.


Conclusion

There are a bunch of reasons to put a message queue between your systems and this feature enables you to use a queue between your systems and a third party. The only thing the third party needs to do is support webhooks which is becoming more and more common. Sign up for IronMQ now to try it out.

Webhooks are the events, message queues are the glue.