7 Reasons Webhooks Are Magic

By Iron | September 25, 2013 |

Around the Iron.io office, I’m known as a bit of a webhook enthusiast. When asked what my favourite features of the Iron.io platform are, our webhook support tend to top my list. (Did you know you can create a message or a task using webhooks, and use push queues as webhooks?)

I love the flexible, open architecture webhooks enable. They remind me of Unix pipes: pass in some data as a string, and any program that knows how to get the data out of the string automatically gains the ability to use that data. That’s so cool. But sometimes, I forget that not everyone knows how amazing webhooks are. I go to hackathons and show them to people, and can almost see their brain as it explodes. I watch people poll APIs or create convoluted connections, and I cry a little on the inside.

I want to show off the power of webhooks for any doubters out there. I want everyone to see the awesome potential this architecture enables.

For the uninitiated, webhooks are simply a pattern for an evented web. All they are is a promise by an application or API: “when this thing happens, I’ll send this HTTP request with this data.” That simple promise opens up a lot of opportunities, though. It enables the web to start being aware of events, to respond to things without user interaction. It makes the web capable of pushing information to its users, instead of waiting for users to ask for information.

I think the easiest way to explain this is with examples:

Table of Contents

1. Handling Text Messages

The Twilio API is one of my favourite APIs of all time, because they make it super easy to marry the physical world with the software world. I love that a few lines of code can make the phone in my pocket start buzzing. I love that I can interact with my application the same way I do with most my friends: text message!

But the best part of all is that Twilio lets you set up webhooks for when you get a text message or phone call. This means that when someone texts or calls your Twilio number, an HTTP POST request will get sent to a URL you define with information about the text or call, letting you process, handle, or reply to it automatically.

Setting up Twilio webhooks for SMS.

If Twilio didn’t have webhooks (the horror!) you’d need to either ask them every few seconds if there was a new text message or phone call – which, let’s face it, your users would notice – or you’d need to set up a dedicated, always-on TCP connection that Twilio could pass information down. Nothing about that sounds fun.

2. Handling Emails

Sending and receiving emails is hard. SMTP, IMAP, POP3, DKIM, spam… you really need to be an expert to get it right. Fortunately, there are experts for hire. SendGrid, MailChimp, and Mailgun, among others, provide simple APIs for you to send and receive email, and their experts take care of making sure the mail actually reaches the other person’s inbox. Neat.

But most of these providers also provide webhooks for when you receive an email. For example, you can set up paddy+loveswebhooks@iron.io to send a POST request to a URL you define whenever an email is sent to it. Better yet, it will include information about the email, making receiving email exactly the same as receiving HTTP requests. A lot of these services can also send webhooks when an email is opened, or a link in the email is clicked. You can build some great user experiences around this kind of evented web. Experiences that don’t rely on users browsing the site, but instead can go to the user wherever they are.

Setting up SendGrid webhooks.

If they didn’t have webhooks, you’d need to set up your own email server software and manage it yourself. Which, believe me, you absolutely do not want to do. Or you’d have to write your code to use IMAP or POP and SMTP, which is no fun. If you’re writing a web application, you’re already using HTTP. No sense in adding another protocol.

Your only other option would be polling their API to see if there are emails to process yet. Polling is no fun. Don’t poll, if you can avoid it.

3. Responding to Changes in Your Code

Everyone loves GitHub. What’s not to love? Version control, wikis, issues, pull requests… GitHub is great. We use it extensively here at Iron.io, and pretty much every developer I know uses it for their personal projects (fork my repos!). One of the best things about GitHub, though, is that they have webhooks for pretty much everything that happens in their service. Want to run a bit of code after every check-in? Webhook. Want to run code whenever someone opens an issue on your repo? Webhook. Want to run code whenever a repository changes from private to public? Webhook.

GitHub exposes almost every conceivable event in their system. It’s part of what makes Github so great. You tell it what events you want to be notified about, and what URL to notify, and GitHub sends a POST request with that data whenever that event happens. Magic.

Setting up GitHub webhooks.

What if they didn’t have webhooks? You’d have to poll with their API. Nobody likes polling. Polling sucks. You could host your own repos, but that means hosting your own git server. Which means backups, no pull requests, no flashy web interface, nothing. And, of course, you have to interact with git’s internal hooks and interrogating the git database itself… It’s so much easier to just get all your information from HTTP requests.

4. Handling Payments

Accepting money online is hard. Partially because it’s scary – there’s money involved! – but partially because there are a lot of really stringent compliance requirements around it, in order to protect consumers.

Fortunately, companies like Stripe and Dwolla exist to make life easier. They expose simple APIs that let users pay you. Guess what else they expose? Yup. Webhooks! Want to run code when a payment completes? Done. Want to run code when a payment fails or is rejected? Got it covered. Want to run code when a payment is refunded? You can do that. Just tell Dwolla or Stripe what events you care about, and what URL should be notified, and they’ll send an HTTP POST request with the relevant data to that URL.

Setting up Stripe webhooks

This approach definitely beats the alternatives. Constantly asking if you have any new payments or any of your payments has been updated/failed/refunded/canceled is a waste of resources. Having users redirect through every step will only tell you about some of the events, and will make everything very fragile. Payments are inherently evented, so you want an evented architecture to deal with them.

5. Responding to Changes on the Web Itself

There’s so much data flowing into the web every minute, it’s hard to keep your application up to speed with the latest and greatest data. And nobody likes stale data. Fortunately, RSS makes it easy to publish and consume information that is meant to be updated: blogs, social media, Wikipedia, etc. The folks at Superfeedr are the experts on RSS and PubSubHubbub (an advanced specification built on top of RSS), and guess what? They offer webhooks.

Even cooler: they will monitor static pages (not just RSS feeds) and send a webhook when they change. Want to run some code when Apple’s homepage changes? You can do that. Want to run some code when there’s a new XKCD? You can do that. Want to run some code when your favourite blog is updated or a Wikipedia article changes? Yup, you can do that.

Twitter, Facebook, YouTube… Most social media platforms offer RSS feeds of their content. Superfeedr lets you get webhook notifications about all of them. Just give Superfeedr the URL of the page or feed you want to get notifications about, and the URL you want the notifications to get sent to, and they’ll send HTTP POST requests whenever that page or feed changes.

Diagram of PubSubHubbub subscriber notifications.

The alternatives are a whole lot worse. You could still rely on Superfeedr, but you’d have to poll them, asking for the latest changes. Or you could poll the pages and feeds individually, and need to deal with RSS parsing woes. Despite being a standard, a lot of people implement RSS in weird ways that break a lot of parsers.

And you need to implement backoff, so you don’t get mistaken for an attack by requesting updates about a site too much, and you should respect their robots.txt file… It’s a very complicated issue. With webhooks, it’s as simple as “yeah, tell me when this thing changes, ok? Thanks.”

6. Responding to What’s Happening On Your Servers

Servers can produce mountains and mountains of log data about literally anything that happens on the server. Error? It’s in the logs. Anything you print to STDOUT? It’s probably in the logs (depending on your setup). HTTP request? It’s probably in the logs. Services like Papertrail aggregate all these server logs for you, and let you search across them. Which, on its own, is extremely useful.

But when you combine it with the fact that they provide webhooks based on log searches, it’s a powerhouse. You can have some code run every time a specific error happens. You can have your application print to STDOUT every time a user registers, and use a webhook to have code run. You can have code run when a specific browser sends a request. Or a specific version of a browser. You can have code run when a request is likely to have come from a specific geographic location, by matching the IP address in your logs to geographic location.

The sky is the limit. Just tell Papertrail what you want to match for from your logs, and where to send notifications when it gets a new match for that. It’ll send you HTTP POST requests with the information.

Setting up Papertrail webhooks.

Without webhooks, your only option would be to poll for this data or implement your own event gateway that logs pass through, which would match the log data as it’s logged and decide whether or not to run some code. That slows everything down, takes up system resources, and is a lot of coding effort. Webhooks are a much more elegant solution.

7. Handling Files and Documents

Box is a great file storage, sharing, and backup solution. What few people know is that it has a great API that makes it easy to programmatically interact with the files and documents you’ve backed up, shared, or stored. What even fewer people know is that it has webhooks.

Want to know when a new file is uploaded? Just tell Box where to send the HTTP POST request. Want to know when a file is updated? Yup, just tell Box where to send the HTTP POST request. You can now do anything you want with those files, based on events. Want to render your markdown to HTML? Go for it. Word to PDF? Yup, you can do that. Combine a folder of images into a GIF? Yup.

Setting up Box webhooks.

Without webhooks, you’d need to poll for a list of all your files, keep track of what’s new, and keep track of when a file was last updated. And you’d have to sync that information across all the servers that are checking for those updates. It would be a mess, and a huge waste of resources. Instead, you get a nice message telling you exactly what happened, and you can respond to it as you please. Much more elegant.

These are some of my favourite webhooks. Whenever webhooks come up in conversation, these are the things I point to to demonstrate the awesome things you can do. But webhooks are more than specific examples and services, they’re a loose architecture that enables anyone to build innovative new integrations.

Webhooks at Iron.io

The internal tooling at Iron.io relies pretty heavily on webhooks; we use them to get notifications in HipChat. We use them to kick off automated processes like builds and test tools. We even use them on the backend to power parts of our services. Webhooks keep our tools flexible and powerful, the way we like them.

We build our services at Iron.io to work well with webhooks and take full advantage of this architecture, because we believe it has tremendous power and flexibility. We can power webhooks on behalf of our customers, and users can push data to us in the form of a webhook.

For example, webhook events can be turned into IronMQ messages to be processed at your convenience. If, for example, you have limits on database or other areas you need to throttle (see a prior article on uses of a queue), then using IronMQ to buffer webhook requests makes a lot of sense.

Alternatively, webhook events can be turned directly into IronWorker tasks, using the webhook data as your payload. Instead of standing up your own server to receive and process webhooks, you can scale out as you need and not have to maintain, manage, or even think in terms of servers. Just send tasks and IronWorker will process them.

Webhooks are first-class citizens in our APIs and we see that continuing and growing with time. One of my goals for our API is to expose more events in our own systems as webhooks, letting our users build flexible solutions around our APIs.

Webhooks Are Magic

The use of webhooks allows you to create a really powerful, integrated workflow across several distinct applications that know nothing about each other. No partnerships, no permissions, no official integration necessary. Just a loose architecture that lets you run code when events that interest you happen. This “evented web” architecture is something I’m a huge fan of, and my coworkers share my enthusiasm for it. In fact, every API evangelist and enthusiast I’ve spoken to has gushed a little about webhooks. The idea that integration is a technical problem, not a business problem, makes it much easier to imagine a world where it works the way it should. A world where all my applications know about each other and tell each other when things happen. A world without polling.

A world of magic.

Ready to get started?

IronMQ offers a free 14 day trial, no cards or commitments required. Signup here.

What other cool webhooks are available? What did we miss? Email support@iron.io and show us something cool. Also see Hacker News discussions here.

Posted in Programming Topics and tagged Webhooks

About Iron

View all posts by Iron →

2 Comments

Sadhana on September 22, 2015 at 8:47 am

Hi Paddy,

As a non-techie, I found this post so easy to comprehend. The examples and the use cases are clearly explained in simple language. Thank you for that!

I decided to record my learning journey and have written a post on what webhooks are and why they’re important (in a layman’s tongue). And yes, I couldn’t miss linking your post in it! 🙂

Here’s the link: https://www.chargebee.com/blog/webhooks/

It would be great if you could have a look at it and let me know how you liked it!
Once again, this is a great post! 🙂

David Multer on November 2, 2018 at 2:40 pm

A classic great article that still gets me excited about webhooks!