Docker + Iron.io = Super Easy Batch Processing

lotsocontainers

There is a ton of use cases for batch processing and every business is probably doing it in some way or another. Unfortunately, many of these approaches take much longer than need be. In a world of ever increasing data, the old way can now hinder the flow of business. Some examples of where you’ll see batch processing used are:

  • Image/video processing
  • ETL – Extract Transform Load
  • Genome processing
  • Big data processing
  • Billing (create and send bills to customers)
  • Report generation
  • Notifications – mobile, email, etc.

We’ve all seen something that was created during a batch process (whether we like it or not).

Now, I’m going to show you how to take a process that would typically take a long time to run, and break it down into small pieces so it can be distributed and processed in parallel. Doing so will turn a really long process into a very quick one.

We’ll use Docker to package our worker and Iron.io to do the actual processing. Why Docker? Because, we can package our code and all our dependencies into an image for easy distribution. Why Iron.io?  It’s the easiest way to do batch processing. I don’t need to think about servers or deal with distributing my tasks among a bunch of servers.

Alright, so let’s go through how to do our own batch processing.

Continue reading “Docker + Iron.io = Super Easy Batch Processing”

Shepherding Containers

Shepherding Containers

Thanks to Yahoo for the base image CC BY 2.0

As one of the earliest users of Docker, I’ve had the pleasure of creating and working with multiple different platforms built on containers. Each platform has evolved in step with current ecosystem around it, and I’ve gotten the chance to really put Docker’s “batteries included, but removable” philosophy to the test.

Here at Iron.io, we have launched over 1 billion user containers in production, not to mention the containers we launch to keep our services running. The massive volume of containers we launch is enough to place great demands upon any platform that we use.

In our search for the right direction for the evolution of our platform, we’ve explored as many tools as possible. The release of Docker 1.9, combined with production-ready Docker Swarm and Docker Networking, brings a lot of value to those wanting to roll their own platforms.

Continue reading “Shepherding Containers”

Right-sizing with Docker Stats and cAdvisor

Right-sizing Docker

Thanks to Jared for the base image CC BY 2.0

Containers make life easy. Oh, you don’t have Ruby 2.2 installed? No problem, try this Docker image. Knowing what I tested on my local is exactly what’s running on production gives me warm fuzzies.

Docker gets a lot of love because it simplifies development. That’s not all though. If Docker punished infrastructure, there’d probably be a lot less love going around. Thankfully, Docker does some cool things on the infrastructure side, as well.

The biggest benefit is the “right-sizing” of compute resources. Your program might only need 200 MB of memory. Why dedicate an entire VM + OS to that? Docker insures our compute resources are neatly divided by memory and CPU between instances. Neat! There’s a lot to love about Docker on the infrastructure side, as well.

Continue reading “Right-sizing with Docker Stats and cAdvisor”

Defrag 2015: Our Favorites From Day Two

defrag-2015-day-2-fin

This post is a continuation of our Defrag 2015 coverage from yesterday. Read on to hear about our favorite talks from day two.

Talks

Where Does the Time Go? – Researching Top Activities At Work

Lisa Kamm is a Product Manager at Google. She got involved with a project to figure out how Googler’s spend their time at work. How could they make it better? Do their mobile products support their own workflows?

Kamm and a team of curious Googlers embarked on a journey to find out. The project started with a collaborative session, where Lisa posed the question: “Wait… what are the top 100 things an employee does in an average day.” Oops. By asking the question she surreptitiously also volunteered to find an answer.

The search for answers started the way you’d expect. Being a Googler, Kamm began the hunt for answers by analyzing a large set of data. Logs from mobile phone and computer usage seemed to be the easiest way to go. There were some hurdles with actually obtaining the logs, as well as with personal privacy. Kamm prevailed in the end, and was able to crunch down 2.5 billion records to get the data she needed.

Continue reading “Defrag 2015: Our Favorites From Day Two”

First Class Support for Docker: Better Service, More Legroom

First Class support for Docker

Thanks to Richard Moross for the base image CC BY 2.0

“Docker, please visit the front desk to receive your complimentary upgrade to first class seating.“

That’s right, Docker just received a first class upgrade on Iron.io. A ways back, Travis (our digital frontiers-man of a CTO) announced beta support for Docker. Today, we’re ripping off the beta tag. Docker is our preferred way to package code.

Continue reading “First Class Support for Docker: Better Service, More Legroom”

#GoSF: Identity, Safe Secrets, and IoT Friendly Languages

GoSF at Betable

The Go gopher was designed by Renee French. CC BY 3.0 US

Last night’s meetup, which was hosted by Betable, included two presentations and two lightning talks rounding out a solid evening for the GoSF group. Topics included identity on the web, safe storage of tokens (beyond ENV vars), and even the debut of a new Go-inspired embedded systems language.

Let’s take a look at each!

Continue reading “#GoSF: Identity, Safe Secrets, and IoT Friendly Languages”

2015 Container Summit notes and learnings, part 2

Containers, VMs, & Infrastructure Part 2

Yesterday we shipped part 1 of our Container Summit notes. Today is a continuation! We’ll share a few of the other talks we enjoyed.

In this post you’ll find stories from Wall Street veterans, open source giants, and nimble challengers.

Continue reading “2015 Container Summit notes and learnings, part 2”

2015 Container Summit notes and learnings, Part 1

Container Summit 2015

A day ago I joined 700+ folks at the Palace Hotel in San Francisco to attend the 2015 Container Summit. Container’s are young, but one thing this event made clear is the forebears have been around quite a while.

A favorite part of the summit was hearing war stories. That is, how containers are called on to get things done in the real world. There were plenty of looks to the past and the future, as well.

I learned quite a bit! What follows is part one of my notes and learnings from Container Summit.
Continue reading “2015 Container Summit notes and learnings, Part 1”

The Easiest Way to Develop with Go — Introducing a Docker Based Go Tool

Via Medium

While trying to make a drop dead simple example for running Go programs on IronWorker, I realized there’s just no simple way to for us to point someone at an example repo with Go code and have them up and running in a few minutes. This can be done in other languages fairly easily because they don’t have special requirements like a GOPATH, a specific directory structure, and they typically have easier dependency management.

Continue reading “The Easiest Way to Develop with Go — Introducing a Docker Based Go Tool”

An Easier Way to Create Tiny Golang Docker Images

Docker_Golang_IronAtlassian just posted a blog post about creating Static Go binaries with Docker on OSX which shows how to make a tiny little Docker image (~6MB total) with your Go program inside. When I read it, I thought “this is cool… but way too complicated”. I’ve been using Docker a lot lately and knew there had to be an easier way. Here is that way. Continue reading “An Easier Way to Create Tiny Golang Docker Images”