Search This Blog


Tuesday, April 22, 2014

How Docker Helped Us Achieve the (Near) Impossible

Docker Solved a Key Problem
Ever since we started, we’ve been trying to solve a problem: how to keep our IronWorker containers up to date with newer language runtimes and Linux packages. For the past two years, IronWorker has been using the same, unchanged run-time environment. That is until a couple of weeks ago when we released custom language environments.

Since the inception of our service, we have been using a single container that contained a set of language environments and binary packages – Ruby, Python, PHP, Java, .NET, and the other languages we support as well as code libraries such as ImageMagick, SoX, and others.

This container (and the strategy behind it) was showing signs of aging with things like Ruby 1.9.1, Node 0.8, Mono 2, and other older language versions in the default stack. As time went on, this problem obviously got worse as people started using newer things and then were forced to change their worker code to work with older versions of their languages.

Limited to a Single LXC Container...

IronWorker uses LXC containers to isolate resources and provide secure run-time task environments. LXC works great as a run-time component but was falling short when it came to creating and loading environments within the runners we use to process tasks. We were at an impasse when it came to creating the runtime environments. On the one hand, we couldn’t just update versions in the existing container or else we'd risk breaking a fair number of the million-plus tasks that run each day. (We tried that once way back at the onset of the service and it wasn't pretty.)

We also couldn't keep different LXC containers around with various language versions as they contained full copies of the operating system and libraries (which means they would clock in at ~2 GB per image). This might actually work fine in a typical PaaS environment like Heroku where the processes run indefinitely and you could just get the right container before starting up the process. In this type of situation, you could even have large custom images for each customer if you wanted without much worry, but IronWorker is a much different animal.

IronWorker is a large shared multi-tenant task processing system where users queue up jobs and those jobs run across thousands of processors. It can be used to offload tasks from the main response loop to run in the background, continually process transactions and event streams, run scheduled jobs, or perform concurrent processing across a large number of cores. The benefit is users get on-demand processing and very large concurrency without lifting a finger.

Under the hood, the service works by taking a task from a set of queues, installing a run-time environment within a particular VM, downloading the task code, and then running the process. The nature of the service means that all machines are used by all customers, all the time. We don’t devote a machine to particular applications or accounts for an extended period of time. The jobs are typically short lived, some running for just a few seconds or minutes and with a maximum timeout of 60 minutes.

LXC did the job to a point but we kept asking ourselves, how can we update or add to our existing container while keeping things backwards compatible and not using an insane amount of disk space. Our options seemed pretty limited and so we kept putting off a decision.

... And Then Came Docker

We had heard about Docker over a year ago. We help organize the GoSF meetup group and Solomon Hykes, the creator of Docker, came to a meetup in March 2013 and gave a demo of his new project Docker, which happens to be written in Go. In fact, he released it to the public that day so it was the first time anyone had really seen it.

The demo was great and the 100+ developers in the audience were impressed with what he and his team had built. (And in the same stroke, as evidenced by one of his comment threads, Solomon started a new development methodology called Shame Driven Development.)

Alas, it was too early back then – we’re talking Day 1 early – so the project wasn’t exactly production ready but it did get the juices flowing.

Solomon Hykes and Travis Reeder hacking
at the OpenStack Summit in 2013.
A month later, I met up with Solomon at the Openstack Summit in Portland for some hack sessions to see how we could use Docker to solve our problem. (I think I only went to one session while I was there, instead spending most of the time hacking with Solomon and other developers).

I started playing with Docker and Solomon helped me wrap my head around what it can do and how it worked. You could tell right off it was not only a cool project but it also was addressing a difficult problem in a well-designed way. It didn't hurt, from my point of view at least, that it was new, written in Go, and didn't have a huge amount of technical debt.

Monday, April 21, 2014 will be at GopherCon (Apr 24th-26th)

GopherCon 2014 is a sponsor of GopherCon and we'll have several of our team in attendance. The conference is in Denver, CO and runs from April 24-26.

We jumped in early as a sponsor as we're a big user of Go and we're glad we did. The attendance will be off the charts as the latest count approaches 750 people.

 Given the list of speakers and the caliber of developers in the Go community, it promises to be one of the major technical events of the year.

Here are a few of the people who will be speaking:

  • Rob Pike - Google
  • Andrew Gerrand - Engineer at Google
  • John Graham-Cumming - Programmer at CloudFlare
  • Derek Collison - Founder, CEO at Apcera
  • Blake Mizerany - Funemployed
  • William Kennedy - Managing Partner at Ardan Studios
  • Richard Crowley - Betable
  • Brad Fitzpatrick - Gopher at Google
  • Rob Miller - Senior Engineer at Mozilla
  • Petar Maymounkov -

A Few of the Speakers at GopherCon

(If you're a member of GoSF, the Go meetup that we help organize, then you'll recognize a number of these names and faces as they've all spoken at GoSF events.)

Update: Keep an eye out for Travis Reeder and Chad Arimura as they'll both be roaming the halls and taking in sessions.  

Update: We're hosting a pre-GopherCon party along with folks from Docker, CoreOS, Pivotal, and Sourcegraph. Come join us for some beers and conversations on changing the (tech) world. 

Monday, April 14, 2014 Joins Red Hat’s OpenShift Marketplace

The OpenShift Marketplace is pleased to announce its IronMQ and IronWorker solutions are part of the OpenShift Marketplace. The marketplace is an important component to Red Hat’s OpenShift Platform-as-a-Service (PaaS) in that it lets developers combine the benefits of enterprise PaaS with tightly integrated, complementary solutions – all without losing time on technology integration. provides an enterprise grade message queuing and task processing platform. It makes it easy to connect systems, handle event processing, and perform highly concurrent task processing. IronMQ is a message queue written in Go that provides full features such as push queues, error queues, retries, and alerts along with high availability, guaranteed one-time delivery, message persistence, and multi-zone redundancy.

IronWorker is a scalable task queue / worker service that provides an easy and reliable way to process workloads asynchronously across thousands of cores. Tasks can be queued directly from web and mobile apps, triggered via webhooks, scheduled to run later, or run continuously in the background. IronWorker supports all common languages including binary packages, offers task retries, and sophisticated task monitoring.

“The OpenShift platform is an important development for The combination of Red Hat’s public cloud and on-premise options along with this new marketplace is a powerful offering and directly in line with our deployment options. Deploying our platform as a cartridge leverages the flexibility and security of OpenShift to give developers powerful ways to build applications.”
– Chad Arimura, CEO,

More on Red Hat’s OpenShift Marketplace  

As more developers use enterprise PaaS for an increasing array of applications, a key to their success is a vibrant and robust partner ecosystem. OpenShift’s current partner ecosystem uses the OpenShift Cartridge specification method to link key technologies and services into applications built on OpenShift, giving customers access to a variety of offerings from cloud industry leaders. The OpenShift Marketplace provides an ideal platform to reach customers and developers in one easy, interactive location, enabling ISVs to showcase their solutions and marketing material, and complete transactions in a few easy steps. 
Read more >> at the Red Hat Summit in San Francisco will be at the Red Hat Summit April 14-17th in the Partner Pavilion. The summit offers more than 150 breakout sessions and hands-on labs and they expect attendees ranging from developers, cloud enthusiasts, and enterprise architects to program managers and CxOs. If you’ll be there and want to talk distributed systems, message queuing, OpenShift, OpenStack, and a host of other topics, please stop by our booth. We’ll be right by the developers lounge.

Wednesday, April 9, 2014

How Edeva Uses MongoDB and IronMQ to Power Real-time Intelligent Traffic Systems

Edeva AB develops and markets intelligent traffic systems. Their product is a dynamic speed bump called Actibump. The selective system makes it possible to handle the compromise in accessibility, traffic flow and speed control, which is not possible with static speed bumps.

Actibump consists of one or more road modules that are mounted into a cast foundation and a radar unit that transmits information to a central control system. The road modules raise and lower in response to vehicle speed and are controlled and monitored over the Internet. Actibump can be expanded with variable speed limits, automatic traffic control (ATC), transponder systems for emergency vehicles, and real-time background data capture for statistical analysis of traffic.

From Road Trials to Production

An Actibump in Sweden
Actibump has been in operation in Linköping, Sweden in 2010 at a location at which unprotected road users cross a street used by cars and public transport. The speed of a vehicle approaching the Actibump is recorded with the aid of radar measurement or induction loops. No action is taken if the vehicle speed is within the legal limit, but if it is approaching faster than is permitted, a part of the roadway pivots downwards, creating an edge. Driving over this edge causes discomfort for the driver encouraging them to reduce speed before reaching the obstacle.)

Here's a video of an Actibump in action:

How Actibump Works
Vehicle speeds have been actively monitored before and after Actibump has been installed. The percentage of vehicles exceeding the speed limit in the three years since installation has dropped from 70 percent to 20 percent, demonstrating Actibump’s proven long-term beneficial effects. The results also show that at least 95% of passing traffic keeps to the speed limit when passing Actibump. This leads to significantly increased safety for unprotected road users.

Reducing Speeds on the Øresund Bridge

In December 2013, Edeva AB won a public procurement for variable speed impediments to the Øresund Bridge. The bridge is a double-track railway and dual carriageway bridge-tunnel across the Øresund strait between Sweden and Denmark.

Øresund Bridge between
Sweden and Denmark
The bridge runs nearly 8 kilometers (5 miles) from the Swedish coast to the artificial island of Peberholm, which lies in the middle of the strait. (The remainder of the link is by a 4 km tunnel from Peberholm to the Danish island of Amager.) Actibump will be installed in four of the eleven lanes at the toll station for Sweden for Denmark-bound traffic, and will help make the work environment safe for bridge personnel.

About Edeva AB

Edeva is led by David Eskilsson and is a spinoff from Prodelox, a leading product development company, and a member of  the LEAD business incubator in Linköping.  In response to the positive evidence of success and the visibility gained from the Øresund Bridge installation, Edeva expects to more significantly expand their opportunities in Europe during 2014.

Edeva’s Actibump solves a worldwide problem, to handle the compromise between traffic safety, accessibility, traffic flow, and work environment in public transport and do so in an intelligent and mindful manner. Edeva uses MongoDB and's IronMQ as core pieces within their infrastructure to  collect and process real-time data and create more intelligent traffic systems. Edeva’s combination of mechanical engineering and advanced data collection and processing demonstrates that the Internet of Things is real and within easy reach.

Details on How the Actibump Works

Choosing a Central Database (MongoDB)

MongoDB Documents and Collections

Edeva identified the opportunity to centrally collect and analyze traffic data in real time, enabling their customers to unlock deeper operational intelligence into traffic management. The application would initially collect data on each vehicle, including maximum speed, average speed and vehicle type, but they also intended to expand this in the future to include weather conditions, congestion levels and other metrics that would improve traffic flow and the safety of road users. 

The development team initially considered using MySQL as their central database, but quickly realized that their changing data requirements would not be supported by its rigid relational data model. With the need to ingest highly variable and rapidly changing sensor data from their Actibump modules and run analytics against it in real time, the development team chose MongoDB. As a consequence, Edeva can dynamically adjust measurements and configuration, while aggregating data to customer dashboards in real time. In addition, the engineering team can quickly evolve their application to meet new customer requirements.