Introducing Custom Docker Images, Private Docker Repositories and Environment Variables

customcontainer

We’re happy to announce three awesome new IronWorker features:

  • Custom Docker Images for all and Docker is now the default code packaging mechanism
  • Support for private Docker images on any Docker Registry, including Docker Hub
  • Support for custom environment variables

I’ll explain each of these features in more detail below.

Custom Docker Images for All!

Previously only available to customers on dedicated plans, this is now available to everyone! You can create custom Docker images and run them at scale on the Iron.io Platform. And as usual, you don’t need to think about servers or scaling or managing anything, you just queue up jobs/tasks. Jobs are executed using your Docker image + a message/payload that defines that job. 

To read how to make your own Docker Worker using the language of your choice, please see our Docker Worker GitHub repository for full documentation for most languages. 

There is a size limit to custom images on our small/free plans of 200MB so you’ll definitely want to use our iron/x base images that we use in our examples to keep them small. If you need bigger images, you can upgrade your account. 

Once you’ve pushed your image to Docker Hub, simply let Iron.io know about it:

Then you can start queuing up jobs for it.

You can see the full API here and client libraries for the language of your choice here.

Private Docker Repositories

Not only can you use your own custom images, you can store those images privately and still use them on IronWorker. Obviously you don’t want other people to access your code inside your image or any type of config files you might have put in the image (although we recommend using environment variables for that, see next section) so you can use a private Docker repository to keep it private.  

To use your private images on Iron.io, you need to login like you do with Docker:

Then just do everything else like normal.

Environment Variables

Instead of uploading a config file as json, we’ve added support for custom environment variables that will be passed into your Docker container at runtime. This allows you to set options that you don’t want to include in your Docker image, such as database credentials or variables that might change based on the environment (development vs production for instance). 

These are set by using -e flags on iron register, for example:

That’s it for Now

These new features give you full capability to use Docker to it’s full potential. Now that this is in place, there’s a lot of exciting new things to come that will build on this.

How To Build Your Own Docker Images

Build Your Own Docker Images

Thanks to Ugur Ceylan for the base image! CC BY 2.0

What’s with the Docker community’s love affair with Alpine Linux? Tiny containers means more compute resources left over for actual… computing! Alpine Linux is particularly tiny. It says so, right on the tin: “Alpine Linux is a security-oriented, lightweight Linux distribution based.”

Do you like saving money? I like saving money. Better resource utilization means happy bank accounts.

Let’s take a closer look at Alpine Linux on Docker. Heck, while we’re at it let’s build our own image.

Continue reading “How To Build Your Own Docker Images”

Running IronWorker on Docker + Node.js + Windows

docker-nodejs-windows

Exosphere champions best of breed cloud applications. In their own words, “We’ve set out on a mission, a quest if you will, to gather together the best small to medium applications in each class, and try to bolt them together in such a way that combined they form a powerful, user-friendly, complete core small business package.”

For the gnarly job of data synchronization exosphere found few options. Solutions for piping core application data certainly exist, but most vendors lock you down like lawn furniture.  Exosphere found IronWorker appealing, since it saved them the hassle of building their own out of the box solution.

Exosphere is built on a Node.js + Windows development stack. Today, they’ve agreed to let us share their recent post on getting IronWorker + Windows + Node.js humming in unison.

If you’re curious what twists and tweaks are required to get IronWorker going on Windows, read on!

Continue reading “Running IronWorker on Docker + Node.js + Windows”

Docker + Iron.io = Super Easy Batch Processing

Docker containers

There are a ton of use cases for batch processing and every business is probably doing it in some way or another. Unfortunately, many of these approaches take much longer than need be. In a world of ever increasing data, the old way can now hinder the flow of business. Some examples of where you’ll see batch processing used are:

    • Image/video processing
    • ETL – Extract Transform Load
    • Genome processing
    • Big data processing
    • Billing (create and send bills to customers)
    • Report processing
    • Notifications (mobile, email, etc.)

We’ve all seen something that was created during a batch process (whether we like it or not).

Now, I’m going to show you how to take a process that would typically take a long time to run, and break it down into small pieces so it can be distributed and processed in parallel. Doing so will turn a really long process into a very quick one.

We’ll use Docker to package our worker and Iron.io to do the actual processing. Why Docker? Because, we can package our code and all our dependencies into an image for easy distribution. Why Iron.io?  It’s the easiest way to do batch processing. I don’t need to think about servers or deal with distributing my tasks among a bunch of servers.

Alright, so let’s go through how to do our own batch processing.

Continue reading “Docker + Iron.io = Super Easy Batch Processing”

Shepherding Containers

Shepherding Containers

Thanks to Yahoo for the base image CC BY 2.0

As one of the earliest users of Docker, I’ve had the pleasure of creating and working with multiple different platforms built on containers. Each platform has evolved in step with current ecosystem around it, and I’ve gotten the chance to really put Docker’s “batteries included, but removable” philosophy to the test.

Here at Iron.io, we have launched over 1 billion user containers in production, not to mention the containers we launch to keep our services running. The massive volume of containers we launch is enough to place great demands upon any platform that we use.

In our search for the right direction for the evolution of our platform, we’ve explored as many tools as possible. The release of Docker 1.9, combined with production-ready Docker Swarm and Docker Networking, brings a lot of value to those wanting to roll their own platforms.

Continue reading “Shepherding Containers”

Right-sizing with Docker Stats and cAdvisor

Right-sizing Docker

Thanks to Jared for the base image CC BY 2.0

Containers make life easy. Oh, you don’t have Ruby 2.2 installed? No problem, try this Docker image. Knowing what I tested on my local is exactly what’s running on production gives me warm fuzzies.

Docker gets a lot of love because it simplifies development. That’s not all though. If Docker punished infrastructure, there’d probably be a lot less love going around. Thankfully, Docker does some cool things on the infrastructure side, as well.

The biggest benefit is the “right-sizing” of compute resources. Your program might only need 200 MB of memory. Why dedicate an entire VM + OS to that? Docker insures our compute resources are neatly divided by memory and CPU between instances. Neat! There’s a lot to love about Docker on the infrastructure side, as well.

Continue reading “Right-sizing with Docker Stats and cAdvisor”

Defrag 2015: Our Favorites From Day Two

defrag-2015-day-2-fin

This post is a continuation of our Defrag 2015 coverage from yesterday. Read on to hear about our favorite talks from day two.

Talks

Where Does the Time Go? – Researching Top Activities At Work

Lisa Kamm is a Product Manager at Google. She got involved with a project to figure out how Googler’s spend their time at work. How could they make it better? Do their mobile products support their own workflows?

Kamm and a team of curious Googlers embarked on a journey to find out. The project started with a collaborative session, where Lisa posed the question: “Wait… what are the top 100 things an employee does in an average day.” Oops. By asking the question she surreptitiously also volunteered to find an answer.

The search for answers started the way you’d expect. Being a Googler, Kamm began the hunt for answers by analyzing a large set of data. Logs from mobile phone and computer usage seemed to be the easiest way to go. There were some hurdles with actually obtaining the logs, as well as with personal privacy. Kamm prevailed in the end, and was able to crunch down 2.5 billion records to get the data she needed.

Continue reading “Defrag 2015: Our Favorites From Day Two”

First Class Support for Docker: Better Service, More Legroom

First Class support for Docker

Thanks to Richard Moross for the base image CC BY 2.0

“Docker, please visit the front desk to receive your complimentary upgrade to first class seating.“

That’s right, Docker just received a first class upgrade on Iron.io. A ways back, Travis (our digital frontiers-man of a CTO) announced beta support for Docker. Today, we’re ripping off the beta tag. Docker is our preferred way to package code.

Continue reading “First Class Support for Docker: Better Service, More Legroom”

#GoSF: Identity, Safe Secrets, and IoT Friendly Languages

GoSF at Betable

The Go gopher was designed by Renee French. CC BY 3.0 US

Last night’s meetup, which was hosted by Betable, included two presentations and two lightning talks rounding out a solid evening for the GoSF group. Topics included identity on the web, safe storage of tokens (beyond ENV vars), and even the debut of a new Go-inspired embedded systems language.

Let’s take a look at each!

Continue reading “#GoSF: Identity, Safe Secrets, and IoT Friendly Languages”

2015 Container Summit notes and learnings, part 2

Containers, VMs, & Infrastructure Part 2

Yesterday we shipped part 1 of our Container Summit notes. Today is a continuation! We’ll share a few of the other talks we enjoyed.

In this post you’ll find stories from Wall Street veterans, open source giants, and nimble challengers.

Continue reading “2015 Container Summit notes and learnings, part 2”