It’s 3/14, and that means it’s international Pi day! A day where we rejoice over the transcendental number that seems to be everywhere.
So, why am I writing about pi on the Iron.io blog? It turns out pi is the best (read: the absolute best!) way to test out computers. It’s sufficiently random, requires large amounts of memory, CPU, and is easy to check.
“I have no interest as a hobby for extending the known value of pi itself. I have a major interest for improving the performance of computation. [..] Mathematical constants like the square root of 2, e, and gamma are some of the candidates, but pi is the most effective.”
How To Make Pi
I’m on board! I want to make Pi, myself. If Pi is a great way to test any computer, why not use it to test first-class distributed computing solutions, like IronWorker?
Humans have known about Pi for a while. Which is part of what makes it a great computation. We have multiple recipes for baking the same dish. That means it’s easy to check our work (by comparing two algorithms).
Docker enables you to package up your application along with all of the application’s dependencies into a nice self-contained image. You can then use use that image to run your application in containers. The problem is you usually package up a lot more than what you need so you end up with a huge image and therefore huge containers. Most people who start using Docker will use Docker’s official repositories for their language of choice, but unfortunately if you use them, you’ll end up with images the size of the empire state building when you could be building images the size of a bird house. You simply don’t need all of the cruft that comes along with those images. If you build a Node image for your application using the official Node image, it will be a minimum of 643 MB because that’s the size of the official Node image.
I created a simple Hello World Node app and built it on top of the official Node image and it weighs in at 644MB.
That’s huge! My app is less than 1 MB with dependencies and the Node.js runtime is ~20MB, what’s taking up the other ~620 MB?? We must be able to do better.
What is a Microcontainer?
A Microcontainer contains only the OS libraries and language dependencies required to run an application and the application itself. Nothing more.
Rather than starting with everything but the kitchen sink, start with the bare minimum and add dependencies on an as needed basis.
Taking the exact same Node app above, using a really small base image and installing just the essentials, namely Node.js and its dependencies, it comes out to 29MB. A full 22 times smaller!
Try running both of those yourself right now if you’d like, docker run –rm -p 8080:8080 treeder/tiny-node:fat, then docker run –rm -p 8080:8080 treeder/tiny-node:latest. Exact same app, vastly different sizes.
Thanks to JD Hancock for the base image! CC BY 2.0
Anyone who’s ever done ETL knows it can get seriously funky. When I first started working on ETL, I was parsing data for a real estate company. Every once in awhile roofing data would appear in the pool field. “Shingles” isn’t a compelling feature for swimming pools. Go figure.
Thankfully, Node.js gives us a lingua franca for sharing cool solutions. A search for data validation shows there are more than a few options. For ETL, let’s take a look at just one of those options.
What’s with the Docker community’s love affair with Alpine Linux? Tiny containers means more compute resources left over for actual… computing! Alpine Linux is particularly tiny. It says so, right on the tin: “Alpine Linux is a security-oriented, lightweight Linux distribution based.”
Do you like saving money? I like saving money. Better resource utilization means happy bank accounts.
Let’s take a closer look at Alpine Linux on Docker. Heck, while we’re at it let’s build our own image.
Exosphere champions best of breed cloud applications. In their own words, “We’ve set out on a mission, a quest if you will, to gather together the best small to medium applications in each class, and try to bolt them together in such a way that combined they form a powerful, user-friendly, complete core small business package.”
For the gnarly job of data synchronization exosphere found few options. Solutions for piping core application data certainly exist, but most vendors lock you down like lawn furniture. Exosphere found IronWorker appealing, since it saved them the hassle of building their own out of the box solution.
Exosphere is built on a Node.js + Windows development stack. Today, they’ve agreed to let us share their recent post on getting IronWorker + Windows + Node.js humming in unison.
If you’re curious what twists and tweaks are required to get IronWorker going on Windows, read on!
There are a ton of use cases for batch processing and every business is probably doing it in some way or another. Unfortunately, many of these approaches take much longer than need be. In a world of ever increasing data, the old way can now hinder the flow of business. Some examples of where you’ll see batch processing used are:
ETL – Extract Transform Load
Big data processing
Billing (create and send bills to customers)
Notifications (mobile, email, etc.)
We’ve all seen something that was created during a batch process (whether we like it or not).
Now, I’m going to show you how to take a process that would typically take a long time to run, and break it down into small pieces so it can be distributed and processed in parallel. Doing so will turn a really long process into a very quick one.
We’ll use Docker to package our worker and Iron.ioto do the actual processing. Why Docker? Because, we can package our code and all our dependencies into an image for easy distribution. Why Iron.io? It’s the easiest way to do batch processing. I don’t need to think about servers or deal with distributing my tasks among a bunch of servers.
Alright, so let’s go through how to do our own batch processing.
A few months ago Adrian Cockroft gave a keynote at devopsdays Amsterdam. In it, he recommended the use of Gauntlt. It’s a robust ruggedization framework for APIs. In plain English, Guantlt is an easy way to test for common security issues.
Have you started using Gauntlt yet? Me neither. Cockroft is sad. Let’s change that.
It turns out, Gauntlt is just the type of tool that IronWorker is great for. It’s modular, and easy to parallelize with the worker pattern. This post is a guide for getting your own Gauntlt up and running.
Containers make life easy. Oh, you don’t have Ruby 2.2 installed? No problem, try this Docker image. Knowing what I tested on my local is exactly what’s running on production gives me warm fuzzies.
Docker gets a lot of love because it simplifies development. That’s not all though. If Docker punished infrastructure, there’d probably be a lot less love going around. Thankfully, Docker does some cool things on the infrastructure side, as well.
The biggest benefit is the “right-sizing” of compute resources. Your program might only need 200 MB of memory. Why dedicate an entire VM + OS to that? Docker insures our compute resources are neatly divided by memory and CPU between instances. Neat! There’s a lot to love about Docker on the infrastructure side, as well.
“Docker, please visit the front desk to receive your complimentary upgrade to first class seating.“
That’s right, Docker just received a first class upgrade on Iron.io. A ways back, Travis (our digital frontiers-man of a CTO) announced beta support for Docker. Today, we’re ripping off the beta tag. Docker is our preferred way to package code.
Last night’s meetup, which was hosted by Betable, included two presentations and two lightning talks rounding out a solid evening for the GoSF group. Topics included identity on the web, safe storage of tokens (beyond ENV vars), and even the debut of a new Go-inspired embedded systems language.