Around the Iron.io office, I’m known as a bit of a webhook enthusiast. When asked what my favourite features of the Iron.io platform are, our webhook support tend to top my list. (Did you know you can create a message or a task using webhooks, and use push queues as webhooks?)
I love the flexible, open architecture webhooks enable. They remind me of Unix pipes: pass in some data as a string, and any program that knows how to get the data out of the string automatically gains the ability to use that data. That’s so cool. But sometimes, I forget that not everyone knows how amazing webhooks are. I go to hackathons and show them to people, and can almost see their brain as it explodes. I watch people poll APIs or create convoluted connections, and I cry a little on the inside.
I want to show off the power of webhooks for any doubters out there. I want everyone to see the awesome potential this architecture enables.
For the uninitiated, webhooks are simply a pattern for an evented web. All they are is a promise by an application or API: “when this thing happens, I’ll send this HTTP request with this data.” That simple promise opens up a lot of opportunities, though. It enables the web to start being aware of events, to respond to things without user interaction. It makes the web capable of pushing information to its users, instead of waiting for users to ask for information.
I think the easiest way to explain this is with examples:
1. Handling Text Messages
The Twilio API is one of my favourite APIs of all time, because they make it super easy to marry the physical world with the software world. I love that a few lines of code can make the phone in my pocket start buzzing. I love that I can interact with my application the same way I do with most my friends: text message!
|Setting up Twilio webhooks for SMS.|
If Twilio didn’t have webhooks (the horror!) you’d need to either ask them every few seconds if there was a new text message or phone call – which, let’s face it, your users would notice – or you’d need to set up a dedicated, always-on TCP connection that Twilio could pass information down. Nothing about that sounds fun.
2. Handling Emails
Sending and receiving emails is hard. SMTP, IMAP, POP3, DKIM, spam… you really need to be an expert to get it right. Fortunately, there are experts for hire. SendGrid, MailChimp, and Mailgun, among others, provide simple APIs for you to send and receive email, and their experts take care of making sure the mail actually reaches the other person’s inbox. Neat.
|Setting up SendGrid webhooks.|
3. Responding to Changes in Your Code
Everyone loves GitHub. What’s not to love? Version control, wikis, issues, pull requests… GitHub is great. We use it extensively here at Iron.io, and pretty much every developer I know uses it for their personal projects (fork my repos!). One of the best things about GitHub, though, is that they have webhooks for pretty much everything that happens in their service. Want to run a bit of code after every check-in? Webhook. Want to run code whenever someone opens an issue on your repo? Webhook. Want to run code whenever a repository changes from private to public? Webhook.
|Setting up GitHub webhooks.|
What if they didn’t have webhooks? You’d have to poll with their API. Nobody likes polling. Polling sucks. You could host your own repos, but that means hosting your own git server. Which means backups, no pull requests, no flashy web interface, nothing. And, of course, you have to interact with git’s internal hooks and interrogating the git database itself… It’s so much easier to just get all your information from HTTP requests.
4. Handling Payments
Accepting money online is hard. Partially because it’s scary – there’s money involved! – but partially because there are a lot of really stringent compliance requirements around it, in order to protect consumers.
Fortunately, companies like Stripe and Dwolla exist to make life easier. They expose simple APIs that let users pay you. Guess what else they expose? Yup. Webhooks! Want to run code when a payment completes? Done. Want to run code when a payment fails or is rejected? Got it covered. Want to run code when a payment is refunded? You can do that. Just tell Dwolla or Stripe what events you care about, and what URL should be notified, and they’ll send an HTTP POST request with the relevant data to that URL.
|Setting up Stripe webhooks|
This approach definitely beats the alternatives. Constantly asking if you have any new payments or any of your payments has been updated/failed/refunded/canceled is a waste of resources. Having users redirect through every step will only tell you about some of the events, and will make everything very fragile. Payments are inherently evented, so you want an evented architecture to deal with them.
5. Responding to Changes on the Web Itself
There’s so much data flowing into the web every minute, it’s hard to keep your application up to speed with the latest and greatest data. And nobody likes stale data. Fortunately, RSS makes it easy to publish and consume information that is meant to be updated: blogs, social media, Wikipedia, etc. The folks at Superfeedr are the experts on RSS and PubSubHubbub (an advanced specification built on top of RSS), and guess what? They offer webhooks.
Twitter, Facebook, YouTube… Most social media platforms offer RSS feeds of their content. Superfeedr lets you get webhook notifications about all of them. Just give Superfeedr the URL of the page or feed you want to get notifications about, and the URL you want the notifications to get sent to, and they’ll send HTTP POST requests whenever that page or feed changes.
|Diagram of PubSubHubbub subscriber notifications.|
And you need to implement backoff, so you don’t get mistaken for an attack by requesting updates about a site too much, and you should respect their robots.txt file… It’s a very complicated issue. With webhooks, it’s as simple as “yeah, tell me when this thing changes, ok? Thanks.”
6. Responding to What’s Happening On Your Servers
Servers can produce mountains and mountains of log data about literally anything that happens on the server. Error? It’s in the logs. Anything you print to STDOUT? It’s probably in the logs (depending on your setup). HTTP request? It’s probably in the logs. Services like Papertrail aggregate all these server logs for you, and let you search across them. Which, on its own, is extremely useful.
But when you combine it with the fact that they provide webhooks based on log searches, it’s a powerhouse. You can have some code run every time a specific error happens. You can have your application print to STDOUT every time a user registers, and use a webhook to have code run. You can have code run when a specific browser sends a request. Or a specific version of a browser. You can have code run when a request is likely to have come from a specific geographic location, by matching the IP address in your logs to geographic location.
The sky is the limit. Just tell Papertrail what you want to match for from your logs, and where to send notifications when it gets a new match for that. It’ll send you HTTP POST requests with the information.
|Setting up Papertrail webhooks.|
Without webhooks, your only option would be to poll for this data or implement your own event gateway that logs pass through, which would match the log data as it’s logged and decide whether or not to run some code. That slows everything down, takes up system resources, and is a lot of coding effort. Webhooks are a much more elegant solution.
7. Handling Files and Documents
Box is a great file storage, sharing, and backup solution. What few people know is that it has a great API that makes it easy to programmatically interact with the files and documents you’ve backed up, shared, or stored. What even fewer people know is that it has webhooks.
|Setting up Box webhooks.|
These are some of my favourite webhooks. Whenever webhooks come up in conversation, these are the things I point to to demonstrate the awesome things you can do. But webhooks are more than specific examples and services, they’re a loose architecture that enables anyone to build innovative new integrations.
Webhooks at Iron.io
The internal tooling at Iron.io relies pretty heavily on webhooks; we use them to get notifications in HipChat. We use them to kick off automated processes like builds and test tools. We even use them on the backend to power parts of our services. Webhooks keep our tools flexible and powerful, the way we like them.
Webhooks are first-class citizens in our APIs and we see that continuing and growing with time. One of my goals for our API is to expose more events in our own systems as webhooks, letting our users build flexible solutions around our APIs.
Webhooks Are Magic
The use of webhooks allows you to create a really powerful, integrated workflow across several distinct applications that know nothing about each other. No partnerships, no permissions, no official integration necessary. Just a loose architecture that lets you run code when events that interest you happen. This “evented web” architecture is something I’m a huge fan of, and my coworkers share my enthusiasm for it. In fact, every API evangelist and enthusiast I’ve spoken to has gushed a little about webhooks. The idea that integration is a technical problem, not a business problem, makes it much easier to imagine a world where it works the way it should. A world where all my applications know about each other and tell each other when things happen. A world without polling.