GridIron: The Power of Webhook-Centric Architectures

A suite of examples showcasing the power
of webhooks using Iron.io and SendGrid.

At Iron.io, we believe strongly that there is a new model of application architectures that is becoming not only viable, but the most powerful way to build your application.

We believe that more and more applications will be built not using static servers and direct integrations, but instead using workers that come into existence when you need them and shut down when they’re no longer necessary.

We believe the future will be built on webhooks and callbacks, an evented model that creates loose, flexible architectures.

Using Webhooks Today

So many of the most popular cloud APIs are using webhooks to enable this loose communication paradigm, that future isn’t far away. That’s why we built webhook endpoints into IronWorker, so you can spin up a worker using the webhooks that have become the lingua franca of the web. That’s why we built webhook endpoints into IronMQ, so you can create a message on a queue using nothing but a webhook, without writing a single line of code.

That’s why we’re so excited that SendGrid offers their Inbound Parse API, which lets you turn emails into webhooks. This kind of interaction is the future of service-oriented architectures, and we’re really excited that so many APIs are supporting it already. It ushers in a new era of cloud applications. EC2 and other hosts require you to set up endpoints to receive webhooks, and there’s so much overhead involved, it’s a major process to connect two APIs. SES and similar solutions don’t offer a way to receive email, which means you need to set up an email server to process those emails, you need to worry about its uptime, and you need to scale it to match your volume of emails. It’s super hard to go from someone sending you an email to being able to programmatically access that email. SendGrid’s Inbound Parse API and their embracing of webhooks allows you to enable that interaction in the amount of time it takes your DNS to resolve.

Introducing GridIron, A Reference Example For Webhook Integrations

At one of the hackathons we ran into SendGrid at, we started talking about all the awesome ways that Iron.io and SendGrid can be used together. As we talked more and more about it, we thought it would be a good idea to share this information with our customers, to help them get the most out of the web.

So today we’re introducing GridIron, a set of examples that shows off the power of SendGrid and Iron.io when used in concert. Consider it a case study of how to use webhooks to make powerful interactions with very little code.

GridIron consists of five main examples right now:

• Hello, SendGrid: a simple demonstration of using IronWorker to send emails using SendGrid
• Queue-Based Emailing: an evolution of Hello, SendGrid that uses a pull queue to give you a more scalable, robust email sending solution that gives you full control over how quickly emails are sent out.
• Fanout: an example that shows how to use IronMQ’s push queues to fan out data to multiple processors. In this case, two IronWorker workers receive the data, and they send a text message through Twilio and send an email through SendGrid.
• Github Notifications: an example that shows how to use Github’s webhooks to trigger IronWorker workers. The workers send emails to the repo owner whenever specified branches are committed to, and ignores the other notifications.
• Markov Bot: the most powerful demonstration we could think of, the Markov Bot will accept an email from the Inbound Parse API, use it to queue up a worker, and process that email. The bot will read your first line, and try to guess the next word you’d use, just like your phone does. It then emails back your line, with its guess appended to the end. It’s an email bot that was written in about 200 lines of Python.

The cloud is about breaking down the barriers that developers have to building cool things. It’s about abstracting away everything that they shouldn’t need to worry about, everything that’s not their core business offering. And we think GridIron is a great demonstration of the types of interactions we can enable very easily.

Fork GridIron on Github. As always, feel free to stop by our chatroom for help!

GoSF Meetup: Canonical + Go 1.1 + a growing Go community

Travis Reeder from Iron.io kicks off the GoSF Meetup (5/9)

Iron.io is the lead organizer of GoSF – which is a meetup specially centered around Go programming language. GoSF has been around for just over a year and it's apparent it's becoming a key group within the developer community.

We base this view not just on the growing attendance (our most recent meetup was packed) but also by the large number of well-known companies lining up to give presentations on their use of Go.



The latest GoSF meetup was this past Thursday, 5/9.

Packed house at Heavybit Industries.

David Cheney from Canonical gave the first talk and addressed the rewrite of Juju from Python to Go, including details on the refactoring process plus the gains they've seen. As background, David is member of the Juju team at Canonical and a committer on the Go project.

Andrew Gerrand talks Go 1.1

Andrew Gerrand followed up in the second talk telling the group about all the good stuff coming in Go 1.1. His talk definitely piqued the interest of the people in the audience, especially when he addressed the performance gains they're seeing with the new version – upwards of 30% in some cases. Andrew works in Google Sydney on Go and is one of the language's top evangelists and a core contributor to Golang.org blog.

The talks can be seen on YouTube here.

  .

For details on Iron.io's use of Go, check out Travis' post on How We Went from 30 Servers to 2 using Go.

And while you're at it, sign up for the GoSF and become part of a growing community of top Go developers.

Blacksmith – an IronMQ client for .NET

Blacksmith is a .NET client for IronMQ that was created by Khalid Abuhakmeh from Aqua Bird Consulting. Khalid recently posted a great article on the reasons behind it and how to use it.

Blog Post on BlackSmith (Tech.Pro)

What's interesting is that Aqua Bird added some capabilities of their own into the client including serialization specific to .Net as well as automatic deletes (IronMQ has a get-delete paradigm to allow for messages to timeout and get placed back on the queue.) While there should be some caution around certain customizations (i.e. automatic deletes take away from the benefits that message timeouts provide), we can't argue with people that make the effort to customize clients to fit their needs. In fact, we're pretty psyched that Aqua Bird took this on.

Here are some excerpts from the post.


The Basics
IronMQ is all based on a REST API. You can perform the following actions:

• Publish a message
• Consume a message(s)
• Defer a message to be published (my favorite)
• Have messages Webhooked to your application (second favorite)
  

Queues are super simple with IronMQ. They are just URL endpoints.

      /projects/{Project ID}/queues/{Queue Name}

All data in IronMQ is JSON, so it offers a compact serialization format that is supported by all languages at this point.
...

If you've wanted to play with queues but didn't want to setup local servers or your development environment, then IronMQ is a super simple no risk way to get started. Blacksmith was built specifically to help you get started, but supports all scenarios provided by IronMQ, so you can grow into the feature set as you need it.

Blacksmith Source Code / NuGet Installation

The Blacksmith library source code can be found on GitHub but you can also jump right in at NuGet.org.

IronMQ Client for .Net Developers (Nuget.org Installation) 

About Khalid Abuhakmeh
Khalid has over 8 years of C# .Net development working with startups and corporate environments. Curator and author of many open source software packages. Follow on twitter @AquaBirdConsult.



About AquaBird Consulting
Aqua Bird Consulting aims to provide the best in Microsoft .Net C# consulting, with current skills in both Microsoft licensed technologies and Open Source frameworks.

Iron.io @ HackTheMidwest

Iron.io will be an API sponsor at the upcoming HackTheMidwest on June 15-16th, 2013 in Kansas City. The event is put on by Kansas City IT Professionals, a grass-roots group of 10,000 strong (the heavy lifting is largely the efforts of Michael Gelphman, hackathon/tech event producer extraordinaire.)

The goal is to bring small teams of developers and designers together and give them the setting and inspiration to build something really powerful.

We're in good company in terms of API sponsors. The set – Context.io, Dwolla, Mashery, SendGrid, Iron.io, TokBox and Twilio – all provide powerful capabilities for building applications quickly.

We especially like the theme that HackTheMidwest is using – Build Something Epic. Can't wait to see what the teams in the midwest produce.

Iron.io @ Civic Data Challenge

Iron.io is thrilled to be a sponsor of the Civic Data Challenge. It's both a cool tech challenge – its prime goal is to turn raw data around ‘civic health’ into useful applications and visualizations AND a worthwhile cause – providing a great forum to directly impact public decision-making. It's put on by the NCoC (National Conference on Citizenship) and the Knight Foundation and has a number of noted civic data partners involved.

The challenge extends from May to November and has a couple different phases to it. The current phase is the ideation phase (people submit ideas, gather input, and build team members and momentum).

Civic Data Challenge 2013 - Steps to Win

Iron.io is providing some prizes to the winners but more importantly some free computing resource to help the teams do some big things with processing and presenting data. IronMQ, IronWorker, and IronCache were made for this type of challenge.

Here are just a few things our platform can be used for:

• Running lots of concurrent jobs (without having to mess with hadoop)
• Collecting streaming data onto a queue for continual processing
• Orchestrating message and event flows via message queues
• Pushing work to the background or to other processes/endpoints
• Processing work on a schedule

Great things can happen when technology meets civic mindedness, especially when you add big data and visualization to the picture. We're expecting to see great things from the teams in this challenge.

Laravel 4 + IronMQ Push Queues = Insane Goodness

Laravel is an increasingly popular PHP framework with a strong and growing following.

If you're a PHP developer, it's a framework to seriously explore if not dig in and put to good use. The design principles are centered around a clean framework that uses a simple, expressive syntax.

About Laravel

There are a number of ways that make Laravel a special framework for PHP developers. Some of the features include:

• Bundles – Bundles are Laravel's modular packaging system. The Laravel Bundle Repository is populated with quite a few features that can be easily added to any application. Developers can either download a bundle repository to their bundles directory or use the "Artisan" command-line tool to automatically install them.
• Eloquent ORM – The Laravel object-relationship mapping is the most advanced PHP ActiveRecord implementation available. With the capacity to easily apply constraints to both relationships and nested eager-loading, developers have complete control over their data with all of the conveniences of ActiveRecord. Eloquent natively supports all of the methods from Laravel's Fluent query-builder.
• Application Logic  – Application logic can be implemented within applications by either using controllers or by injecting directly into route declarations by using syntax similar to the Sinatra framework.
  
  view more >>

Laravel + IronMQ: The Benefits

Using a message queuing service with PHP applications is a great way to speed up your app (by pushing work to the background) as well as gain greater agility (by decoupling your code and making processes more independent). This solid 3-tier architectural underpinning can be achieved right from the start using IronMQ. (IronMQ is a cloud-based service that is instantly and always on. Which means you don't have to set-up or manage servers or worry about things like high availability or redundancy. All that is managed by Iron.io.)

Offloading Tasks

By moving tasks to the background – things like sending emails, posting data, transforming images, running reports, accessing accounts, or performing calculations – you can return control back to the user much more quickly and provide a much more responsive experience.

For many tasks, there may not be a user acknowledgement; for others, a call-back or a non-intrusive front-end polling cycle can retrieve the information once the task is done and the information is ready to be presented.

Decoupling Processes

By decoupling processes, you make the front-end more independent from the processing layer and make each process independent from each other. In this way, changes in one type of worker (sending an email for example or creating a thumbnail) will have little to no impact on any other parts of the application.

It also makes it much easier to add new features. If you want to add another action to an event, you can add another queue or in the case of push queues, add another subscriber. The process sending the event/message doesn't need to know about the new subscriber and neither do the other subscribers (unless you want them to in which case you can use a chain of queues to sequence the work).

Laraval 4 and IronMQ: Using Push Queues

Push queues allow you to post messages/data to a queue and have that queue push the data to one or more URLs. Laravel 4 and the IronMQ bundle provides an easy way to marshal that request into the queuing process. This means that you can do lots of asynchronous processing without a lot of overhead or added work.

Here's documentation on using queues within Laravel 4.

      Laravel 4 : Documentation : Queues

Here's a short video that goes through the process from creating a queue to push messages to a URL.

Using Iron.io push queues in Laravel 4

Iron.io Joins OpenStack to Drive Open Cloud Message Queues

Iron.io is now an OpenStack supporter. This may not appear all that unusual – given the top companies originally behind the initiative plus the growing numbers joining – but it is noteworthy for a cloud services company. 

With most cloud services, it shouldn’t really matter what the underlying IT stack or even what cloud you’re running on. API-driven services essentially abstract the VMs, infrastructure, and scaling away to provide elastic utility-based computing. Just plug into the APIs and eliminate the need to provision resources based on current or anticipated loads. 

Cloud Services Everywhere

But with the types of services that Iron.io provides – message queuing, task processing, data caching, and job scheduling – the locality of the services is important. Developers want these types of services close to their applications and in different configurations and combinations. 

Given that we believe message queueing, task processing, caching, and job scheduling are critical components within any production-scale application, we want Iron.io to be a part of every application built within every cloud platform running on every cloud infrastructure. OpenStack’s thought-leadership, traction, and increasingly prominent support makes it a clear bet to drive innovation within public and private clouds. And so it only makes sense for us to be a part of this effort. 

OpenStack Open Message Queuing Protocol

One key aspect with our support is not just in laying a marker on the cloud infrastructure layer but also in rolling up our sleeves and helping to define open cloud messaging protocols – ones that we believe will define the next generation of messaging platforms. 

We’ve spoken a bit about messaging protocols a bit before in this post. Current messaging protocols such as AMQP aren’t going away. They have many years behind them and many referenceable installations. It’s just that we don’t believe AMQP or any existing standard will be the messaging protocol in common use within the cloud going forward.
What's needed instead is a simple, secure, reliable messaging protocol that is HTTP-based, supports JSON out of the box, and is designed to support widely distributed and massively scalable sending and receiving. Older protocols could be adapted to this world, but coming up with something native to the cloud that's based on cloud application patterns will appeal to cloud developers now and the many that will follow in years to come.

OpenStack Marconi Project

This effort towards the future of Internet-scale messaging is encapsulated in the Marconi project. Kurt Griffiths from Rackspace is driving the effort forward. +Travis Reeder and +Evan Shaw from Iron.io have been influential in defining the API. Here's a description of the project:

Marconi is a new OpenStack project to create a multi-tenant cloud queuing service. Our aim is to create an open alternative to SQS (producer-consumer) and SNS (pub-sub), for use in applications that run on OpenStack clouds. The project will define a clean, RESTful API, use a modular architecture, and will support both eventing and job-queuing semantics. Users will be able to customize Marconi to achieve a wide range of performance, durability, availability, and efficiency goals.

We will fully support the protocol and plan to have the best and most reliable implementation available. We have a wealth of experience in this arena.  IronMQ is a high availability message queuing service that runs on several public clouds (AWS and Rackspace). It's tried, tested, and true – serving over 250M API requests a day.

Compute + Storage + ...

To put the importance of the project in context, much of the emphasis with cloud protocols to date has been on the Compute and Storage aspects of the system. They are huge areas that form the core of any cloud application. But production-scale applications with lots of asynchronous activity need ways to connect processes within a system, interface with other systems, buffer activity to databases, and power service oriented architectures. To do this, you need a reliable messaging layer.

Here’s how HighScalability describes the importance of message queuing:

"[Y]ou can find a message queue in nearly every major architecture profile on HighScalability. Historically they may have been introduced after a first generation architecture needed to scale up from their two tier system into something a little more capable (asynchronicity, work dispatch, load buffering, database offloading, etc). If there's anything like a standard structural component, like an arch or beam in architecture for software, it's the message queue."

And so to us, the real equation in a cloud system is:

Compute + Storage + Message

If you believe a cloud-native messaging protocol will be at the center of the next wave in connecting systems – and that the connections will not just be point-to-point within a closed system but on globally distributed and massive scale – then OpenStack and the Marconi project are efforts to pay attention to. 

Great things are happening in this space and we’re thrilled to be a part of OpenStack and helping to drive open cloud messaging forward.

IronMQ Handling 250M Requests Per Day, IronWorker Executing Over 500K Jobs

This month, Iron.io hit two major milestones in our usage numbers for two of our services.

IronMQ is now handling over 250 million requests per day (== 174K per minute == 2,900 per second). This is up from 100 million just three months ago.

Also, IronWorker is now executing over 500,000 jobs (tasks) per day. That means we are executing your code across our farm of worker servers, half a million times per day.

When we built our first product, IronWorker, we thought we had built something that other developers would find useful, but we had no idea how much people would enjoy using our services to build amazing things.

"You guys rock! Your service seems great so far :-) Keep killing it."

"IronMQ makes me want to add messaging to everything I build." src: Twitter

"We just switched out message queue backends for @besnappy (to @getiron) in 5 minutes." src: Twitter

Even when we have issues:

"Glad to hear it was just temporary.  Your uptime has been so consistent that getting text messages from my monitoring systems about you being down was noteworthy."

And my personal favorite:

"wow. sh*t. holy f**k ... awesome. shut my mouth. F**KING AWESOME. you got to be sh*tting me. I love you"

That was from one of our customers that was managing their own worker servers, costing them $12k per month in server costs alone plus all the time they had to spend dealing with them. After switching to IronWorker, they now spend less than $1000 and zero time managing servers. The quote above was when he first got his stuff running on IronWorker and had the a-ha moment.

So here's a big THANK YOU to all our users. We love you too.

Relify adds IronWorker to their Stack ➞ Ridiculously Simple Worker Scalability

Relify, a Recommendations as a Service engine, is one of the really cool customers that Iron.io serves. They offer a simple API that eliminates the complexity of developing a recommendation engine –which means you can greatly increase your relevance to your users.

Terry Horner is the Co-Founder and CEO of Relify. A recent blog post of his details their use of a scale-out processing pattern that, in concert with IronWorker, allowed them to significantly scale their recommendations service. All in a short period of time without any infrastructure or much cost.

It's a great read and we thought we'd highlight some portions:

Relify Blog Post

We’d come so far and had realized that there was a distinct possibility that we’d have to implement our own worker service. Then fortune smiled. The folks at Iron.io sent me a “Tips & Tricks” email on the same day I was getting ready to start coding our worker service (no joke). In that email was a link to a blog post by one of Iron.io’s Co-Founders, Travis Reeder, entitled How to Reduce a 9 Hour Job to a 9 Minute Job. Were they reading my mind? 

It turns out that we weren’t the only ones with such struggles as the first line of the post so poignantly states: “A common problem developers have is how to run through a large number of tasks in a short amount of time.” PREACH ON! I kept reading. 

I immediately felt better after reading the post. Why hadn’t I attempted to dig deeper into Iron.io’s blog before I threw in the towel? Or hit them up on HipChat (which I subsequently did, and they were awesome). The answer was there the whole time! 

The post got me thinking about breaking work into smaller pieces to take advantage of concurrent task execution. I imagined that we could spawn additional tasks based on the total number of items in a queue and let recursion take its course until the workers burned through all of the records in each queue. The end result was a collection of workers that run on a schedule and are entirely self-contained within IronWorker. 

... 

This took us from having the ability to process 1,000-2,000 records/minute to 120,000/minute (2,000/second) based on our current pricing plan. Bumping ourselves up to the next plan would put us somewhere around 300,000/minute which clocks in at 5,000/second. And we didn’t bump write throughput up a single tick. Increase the write throughput and things get crazy fast!

And the best part? Everything I’ve mentioned above took less than a day to implement and costs less than $500/month (AWS costs included). FO REALZ.

Limitless scalability is real and the team at Iron.io is making it happen. IronWorker is rock solid and our stack wouldn’t be the same without it.

About the Author

Terry Horner is the Co-Founder and CEO of Spent, Inc., the parent company of Relify. Father. Husband. Triathlete. Entrepreneur. Creative thinker. Problem solver. Student of business. Tech geek. Kent State alum. Cleveland fan. Browns season ticket holder. He loves building things and has bootstrapped three profitable startups. Working on number four.

Recommendations as an Essential Service

Recommendations, personalization, and discovery are becoming core components to almost every site and service out there. These capabilities take a nice site and turn it into an essential site. Recommendations are hard to get right, though.

You're continually getting inputs in the form of clickstreams, checkins, purchases, follows, likes, plus-ones, and other forms of signals and intent. You have to manage this huge amount of data inputs, rank them appropriately, perform continual processing on the stream, and then return genuine items of relevance. All across an array of object types (books, movies, photos, products, people, ...) It's a perfect use case for elastic message queues and scalable task queues. All these inputs need to be collected and processed continually and at huge scales.

Finding a service that is simple to use, provides good recommendations, and scales is a rare combination. Relify has something here which certainly makes them a company to watch.

IronWorker API Outages - Post Mortem

Starting around 9:45 PT on April 6th, the IronWorker API started having issues where requests to the API would not receive a response and timeout. The issues were intermittent over the next few days with short outages followed by hours of normal operation. The cause of the issues were a couple infrequently used queries that weren’t using indexes, more information on that below. The issue was finally resolved around 2:45 PT on April 8th.

Note: this only affected the IronWorker API. IronMQ, IronCache, the IronWorker scheduler, and the IronWorker backend were not affected.

Why did it take so long to resolve?

The hardest part about resolving this was that the outages were very brief, 5-10 minutes, and occurred at random times with hours in between. The first time it happened, we thought it may have just been one of those random issues that can happen once in a blue moon and after reviewing all our monitoring tools, it didn’t seem like anything was wrong. The second time it happened, a couple hours later, we knew something was wrong so we dug in deeper. The team worked night and day to try to resolve this, but again, the hard part was that it was short outages with hours of uptime in between so every attempt to resolve the issue took hours to see if it worked. Also, we weren’t able to reproduce this on our staging environment.

Steps we took to resolve the issue

Here are the steps we took:

• Reviewed our monitoring tools
• Nothing seemed out of the ordinary here, our databases looked fine and all CPU/Disk/Memory metrics looked fine too. 
• Reviewed code
• We reviewed our code. A couple of days before, April 4th, we had a pushed out a new release of the API with a very minor change (literally one line of code). 
• We reviewed the code of our dependencies, meaning we went through commit by commit on Github to see if anything major changed. 
• Rolled back recent changes
• We rolled back our code. This was unlikely to fix it due to the minor changes and having been tested thoroughly on staging before rolling out, but you can never be too sure. 
• We rolled back our AMI to a previous AMI we’d used without issue. 
• Reviewed how the IronWorker backend (which runs the actual jobs) was interacting with the API and how it had changed recently. 
• We even rolled this back. 
• Dug into database logs 
• We checked our query logs and found two queries that were taking seconds to complete (which is waaaay too long). Bingo. 
• These queries were locking up our database. 
• We created indexes for these two queries. 
• Problem solved. 

The Resolution

After creating those indexes, the issue went away. It turns out these queries were rarely used which is why we hadn’t seen this issue for the past 2+ years that IronWorker has been running. But when suddenly they are used in a high load system, you have yourself a good old fashioned problem. As I always say, 90% of the issues you’re going to have with any application/service is going to be in your database. I should have listened to myself over the past few days and we probably would have resolved this sooner.

What We’re Doing About It

We always learn from our mistakes and we take outages, very, very seriously. We have already added what we’ve learned from this to our emergency response plan and we are also going to start automating our response plan wherever we can to remove the human element and increase our ability to catch these things faster and with more precision. We will also be doing a full code review to ensure all queries are covered by indexes.

We sincerely apologize for any inconvenience this caused our customers. Please feel free to reach out if you have any concerns, you can reach me directly, through our chatroom, twitter or Google+.


[edit 4/15]
Thanks to a few suggestions we are also committed to better informing customers in realtime during incidents, least of which has been the rollout of our brand new status.iron.io site. You can choose to be updated by SMS/email for all or selected events.