Blog

Search This Blog

Loading...

Monday, August 25, 2014

Segment.io adds Iron.io as an endpoint for message queueing

Iron.io + Segment.io = Integration Goodness
IronMQ has just been added as an endpoint for message queueing within Segment.io. Segment.io provides a single gateway that lets developers, marketers, and analysts pipe logs, metrics, user actions, and other key app data to hundreds of other applications and tools.

The Segment.io API acts as a control for other analytics tools, allowing developers to integrate one single API in place of many.

Using Segment.io's API lets an engineer implement Segment.io tracking once within an application and then it will automatically translate/implement every third-party tag via their gateway. The net effect is that a developer or marketer can just push an "on" button to enable a new tool.

Segment.io + Message Queueing

There are a number of key advantages for using message queuing in combination with Segment.io. For example, in some situations, the data streams or webhook requests coming out of the Segment API can overload systems on the receiving end.

In these cases, IronMQ is a great solution because it can act as a buffer. It also keeps events in FIFO, has a storage duration of up to 30 days, and provides developers with the ability to build out custom workflows.

We'll be going deeper into the subject in subsequent posts and dev center articles but here are just a few of the use cases why using a message queue as a Segment.io endpoint can make a lot of sense.

•  Buffer and persist data for processing at a later time.
•  Extract, translate, and load data into tools such as AWS's Redshift, Salesforce, and other custom tools.
•  Process data on-the-fly asynchronously using IronWorker.



To learn more about how message queues can help increase system reliability and handle data spikes, check out this article on top uses of a message queue.

To integrate IronMQ into Segment.io, head to their Integration Center for simple instructions that will get you up and running in minutes.


Tuesday, August 19, 2014

How One Developer Serves Millions of Beers: Untappd + Iron.io

Untappd – The Mobile App for Beer Lovers
Untappd provides a mobile check-in application for beer lovers. Their application has been downloaded by over a million users and on any given night, they can register over 300,000 individual checkins and social sharing events.

The Untappd application lets users record their beer selections, share their likes with friends, win rewards, get recommendations, and participate in a shared passion of beer with others around the world. A solid design, a fun set of features, and a responsive application are just a few of the reasons they’re one of the fasting growing entertainment-based social communities.

What’s even more impressive about Untappd is that it’s just a two-person company – a designer and a developer, both of whom have other jobs and are doing Untappd on the side. This is a story on how they got started and the work that goes on behind the scenes to power their success.


The Untappd Story

Greg Avola and Tim Mather met over Twitter six years ago when Greg was looking for a collaborator for a Twitter mobile app. They ended up working together on the app and then proceeded to take on several other projects as a designer/developer combination. In early summer of 2010, Tim came up with the idea for a check-in system for beer drinkers. The idea mapped well with Greg’s interest in beer and so they quickly created a mobile app and got to market by the fall.

The Untappd app has evolved a lot since the early days but the main premise is the same – users check-in at locations and check-in on the beers they’re drinking. For each check-in, they become eligible to win badges and receive promotions. They can also get real-time recommendations for beers based on their location.

The team works closely with breweries and beer venues to increase the connections that users have with their favorites beers. They help breweries and other partners create badges and other promotional elements for beers and events. The badges are hugely popular and are posted and shared widely within the app and across social media.

Checking In a Beer
Given how important up-to-date information about beers is, they’ve created what will soon become one of the largest open-source databases on beers in the world. It’s moderated by over 40 volunteers who help clean up information and de-dup entries. They offer free API access for developers and have the ultimate goal of making it the most widely used libraries about beer.

Registered users top over a million and they service over 300,000 check-ins on weekend nights and have processed over 50M events (the majority of them using Iron.io). Users love the Untappd app and use it to keep track of their beer, discover new favorites, meet new people, and find new places of interest.

The Untappd app is a model that works – a fanatical user base, an app that provides rewards, hundreds of happy partners, and an almost limitless opportunity.




Behind the Scenes of the App

The app framework for Untappd is that of a mobile client, a set of app servers connected to databases, and a large async/background processing component. They make use of a LAMP stack with PHP serving as their primary language. They use MySQL as their primary database for transactions, MongoDB for their recommendation engine and activity feeds, Redis to store all the counts for beer/user/brewery/venue, and Iron.io for their background processing and as their mobile compute engine.

Recent Beers
When users check in to Untappd, there are a number of transactional events that take place. The user account gets updated and the check-in gets posted to Twitter, Facebook, and/or FourSquare. If a photo is uploaded, it gets processed. Check-in parameters get filtered for location and venue and then piped it into their MongoDB clusters that power their local recommendation capability. All in all, there can be up to 10 different events taking place for each location or beer check-in.


Before Iron.io

Initially, the check-in processes were being handled as a large batch job after hours at night. Because actions were being posted well after the actual event, the check-in process obviously wasn’t as responsive enough for their users as they needed. The Untappd team then moved these actions to the check-in response loop. That lasted for a little while as it resulted in a more responsive check-in but it quickly showed signs of strain. On heavy nights, the Untappd main app servers would start to melt because they were being used to process all the actions for each check-in, in addition to serving pages and providing query responses.

This tightly coupled serial approach also resulted in users having to wait for each process to start and finish in sequence. The delayed response times began having noticeable impacts on engagement. It was taking much longer to check-in as the app wouldn’t return for up to many seconds at a time. Users were getting frustrated and so they were not checking in for the second beer or the third.

Serial Processing Events at Check In = Slow Response Times

The general experience was also not feeling real-time enough for users because they wouldn’t see tweets until much later, and the information they were receiving from the app after a check-in was not as relevant as they might expect. Recommendations for other beers, for example, were out of date because the database wasn’t getting new beer inserts in a timely manner, and notifications of nearby trending places were not being sent out quickly enough to be relevant.

To keep user engagement high and their user base growing, they needed find a solution to their check-in problem. They turned to Iron.io to do so.


After Iron.io

To make their application more responsive and scalable, Untappd move their event processing to Iron.io as a combination of IronMQ and IronWorker. Each check-in event is sent to a queue within IronMQ and then routed to workers within IronWorker. The processing runs outside the user response loop, which speeds up check-ins as well as provides the ability to handle any and all spikes in traffic.

Trending Events
Using Iron.io, Untappd has been able to reduce the time to average check-in time from over 7 seconds to 500ms. They’ve also eliminated the need to manage infrastructure for this part of their app and given themselves an almost unlimited ability to scale their processing.

Continual Event Processing

The way the event flow works is that they put a check-in event onto a single queue and then that fans out to multiple queues – with each sub-queue controlling a different action, such as posting to social media or updating the recommendation engine. Multiple workers spin up soon after a check-in happens and so by the time the user has laid down their phone and sampled their beer, every action is either in process or has completed.
Notifications

Wednesday, August 6, 2014

Go, IronWorker, and SendGrid at Gengo (a repost)

Shawn Smith from Gengo recently wrote a post on their use of Go, the programming language that we also use at Iron.io for our backend services. (Gengo is a popular people-powered translation platform based in Tokyo and San Mateo.)

The post discusses several of the apps where they're using Go including a deployment app and several conversions of PHP and Python apps. The one that caught our attention is their use of Go with IronMQ and SendGrid to send out large volumes of emails to their user base.


Here's an excerpt from the post:
We created a service to send emails that uses Iron.io‘s queue service, IronMQ. We call this the Email Consumer, which pulls JSON payloads off of a queue before rendering and sending the email that matches the ID in the payload.  
[When] a new customer signs up on our website, our web application puts a JSON payload with an ID onto the queue. The Email Consumer consumes the payload and looks up the email subject for the email with the given ID, also rendering the template with the given data. In this case, the data is simply the user’s name. It also localizes the strings in the template based on the language code provided (in this case, it’s Japanese) and sends the email via SendGrid, welcoming the customer to Gengo.


The lines that we like come in the summary of the app:
"Moving all of our emails into one place helps us easily make consistent style and copy edits. We send over 50 different emails to customers and translators through the Email Consumer, and to date it has sent over 500,000 emails without a problem."
– Shawn Smith, Go Developer at Gengo

Doing work in the background using IronMQ and SendGrid is a great way to go about distributing work and increasing the scalability of an application. That they're using Go to do so makes it all that much better. Thanks for the inclusion, Gengo.


About Gengo 

Gengo helps businesses and individuals go global by providing fast, high quality translation powered by the crowd. Upload copy to their website or via API and then their platform springs into action, allocating work orders among thousands of qualified translators in real-time. The platform currently draws from a network of 10,000+ pre-tested translators working across 34 languages.

About the Original Author

Shawn Smith is a software developer from Boston, Massachusetts. After graduating from Northeastern University, he moved to San Francisco to work for Rackspace and now works at Gengo in Tokyo. His favorite programming language is Go.