How to Bake Your Own Pi

Baking Your Own Pi

It’s 3/14, and that means it’s international Pi day! A day where we rejoice over the transcendental number that seems to be everywhere.

So, why am I writing about pi on the Iron.io blog? It turns out pi is the best (read: the absolute best!) way to test out computers. It’s sufficiently random, requires large amounts of memory, CPU, and is easy to check.

I first learned about this aspect of pi while reading the book Heres Looking at Euclid. There, I also learned that Pi beyond 40 digits or so isn’t all that useful. So, why do we know pi into the billions of digits? To quote the many time world record holder,

“I have no interest as a hobby for extending the known value of pi itself. I have a major interest for improving the performance of computation. [..] Mathematical constants like the square root of 2, e, and gamma are some of the candidates, but pi is the most effective.”

How To Make Pi

I’m on board! I want to make Pi, myself. If Pi is a great way to test any computer, why not use it to test first-class distributed computing solutions, like IronWorker?

Humans have known about Pi for a while. Which is part of what makes it a great computation. We have multiple recipes for baking the same dish. That means it’s easy to check our work (by comparing two algorithms).

So, what goes into pi? How can I cook this dish? Let’s check out a few of the best recipes. Continue reading “How to Bake Your Own Pi”

Microcontainers – Tiny, Portable Docker Containers

microwhale-vs-bigwhale

Docker enables you to package up your application along with all of the application’s dependencies into a nice self-contained image. You can then use use that image to run your application in containers. The problem is you usually package up a lot more than what you need so you end up with a huge image and therefore huge containers. Most people who start using Docker will use Docker’s official repositories for their language of choice, but unfortunately if you use them, you’ll end up with images the size of the empire state building when you could be building images the size of a bird house. You simply don’t need all of the cruft that comes along with those images. If you build a Node image for your application using the official Node image, it will be a minimum of 643 MB because that’s the size of the official Node image.

I created a simple Hello World Node app and built it on top of the official Node image and it weighs in at 644MB.

That’s huge! My app is less than 1 MB with dependencies and the Node.js runtime is ~20MB, what’s taking up the other ~620 MB?? We must be able to do better.

What is a Microcontainer?

A Microcontainer contains only the OS libraries and language dependencies required to run an application and the application itself. Nothing more.

Rather than starting with everything but the kitchen sink, start with the bare minimum and add dependencies on an as needed basis.

Taking the exact same Node app above, using a really small base image and installing just the essentials, namely Node.js and its dependencies, it comes out to 29MB. A full 22 times smaller!

 

pasted_image_at_2016_01_22_11_20_am
Regular Image vs MicroImage

 

Try running both of those yourself right now if you’d like, docker run –rm -p 8080:8080 treeder/tiny-node:fat, then docker run –rm -p 8080:8080 treeder/tiny-node:latest. Exact same app, vastly different sizes.

Continue reading “Microcontainers – Tiny, Portable Docker Containers”

The E.T. in ETL

The E.T. in ETL

Thanks to JD Hancock for the base image! CC BY 2.0

Anyone who’s ever done ETL knows it can get seriously funky. When I first started working on ETL, I was parsing data for a real estate company. Every once in awhile roofing data would appear in the pool field. “Shingles” isn’t a compelling feature for swimming pools. Go figure.

Thankfully, Node.js gives us a lingua franca for sharing cool solutions. A search for data validation shows there are more than a few options. For ETL, let’s take a look at just one of those options.

Continue reading “The E.T. in ETL”

How To Build Your Own Docker Images

Build Your Own Docker Images

Thanks to Ugur Ceylan for the base image! CC BY 2.0

What’s with the Docker community’s love affair with Alpine Linux? Tiny containers means more compute resources left over for actual… computing! Alpine Linux is particularly tiny. It says so, right on the tin: “Alpine Linux is a security-oriented, lightweight Linux distribution based.”

Do you like saving money? I like saving money. Better resource utilization means happy bank accounts.

Let’s take a closer look at Alpine Linux on Docker. Heck, while we’re at it let’s build our own image.

Continue reading “How To Build Your Own Docker Images”

Running IronWorker on Docker + Node.js + Windows

docker-nodejs-windows

Exosphere champions best of breed cloud applications. In their own words, “We’ve set out on a mission, a quest if you will, to gather together the best small to medium applications in each class, and try to bolt them together in such a way that combined they form a powerful, user-friendly, complete core small business package.”

For the gnarly job of data synchronization exosphere found few options. Solutions for piping core application data certainly exist, but most vendors lock you down like lawn furniture.  Exosphere found IronWorker appealing, since it saved them the hassle of building their own out of the box solution.

Exosphere is built on a Node.js + Windows development stack. Today, they’ve agreed to let us share their recent post on getting IronWorker + Windows + Node.js humming in unison.

If you’re curious what twists and tweaks are required to get IronWorker going on Windows, read on!

Continue reading “Running IronWorker on Docker + Node.js + Windows”

Docker + Iron.io = Super Easy Batch Processing

lotsocontainers

There is a ton of use cases for batch processing and every business is probably doing it in some way or another. Unfortunately, many of these approaches take much longer than need be. In a world of ever increasing data, the old way can now hinder the flow of business. Some examples of where you’ll see batch processing used are:

  • Image/video processing
  • ETL – Extract Transform Load
  • Genome processing
  • Big data processing
  • Billing (create and send bills to customers)
  • Report generation
  • Notifications – mobile, email, etc.

We’ve all seen something that was created during a batch process (whether we like it or not).

Now, I’m going to show you how to take a process that would typically take a long time to run, and break it down into small pieces so it can be distributed and processed in parallel. Doing so will turn a really long process into a very quick one.

We’ll use Docker to package our worker and Iron.io to do the actual processing. Why Docker? Because, we can package our code and all our dependencies into an image for easy distribution. Why Iron.io?  It’s the easiest way to do batch processing. I don’t need to think about servers or deal with distributing my tasks among a bunch of servers.

Alright, so let’s go through how to do our own batch processing.

Continue reading “Docker + Iron.io = Super Easy Batch Processing”

Adrian Cockroft Wants You to Run Gauntlt

gauntlt ironworker simple security

Thanks to David Goehring for the base image CC BY 2.0

Secure APIs with a robust ruggedization framework

A few months ago Adrian Cockroft gave a keynote at devopsdays Amsterdam. In it, he recommended the use of Gauntlt. It’s a robust ruggedization framework for APIs. In plain English, Guantlt is an easy way to test for common security issues.

Have you started using Gauntlt yet? Me neither. Cockroft is sad. Let’s change that.

It turns out, Gauntlt is just the type of tool that IronWorker is great for. It’s modular, and easy to parallelize with the worker pattern. This post is a guide for getting your own Gauntlt up and running.

Continue reading “Adrian Cockroft Wants You to Run Gauntlt”

Right-sizing with Docker Stats and cAdvisor

Right-sizing Docker

Thanks to Jared for the base image CC BY 2.0

Containers make life easy. Oh, you don’t have Ruby 2.2 installed? No problem, try this Docker image. Knowing what I tested on my local is exactly what’s running on production gives me warm fuzzies.

Docker gets a lot of love because it simplifies development. That’s not all though. If Docker punished infrastructure, there’d probably be a lot less love going around. Thankfully, Docker does some cool things on the infrastructure side, as well.

The biggest benefit is the “right-sizing” of compute resources. Your program might only need 200 MB of memory. Why dedicate an entire VM + OS to that? Docker insures our compute resources are neatly divided by memory and CPU between instances. Neat! There’s a lot to love about Docker on the infrastructure side, as well.

Continue reading “Right-sizing with Docker Stats and cAdvisor”

First Class Support for Docker: Better Service, More Legroom

First Class support for Docker

Thanks to Richard Moross for the base image CC BY 2.0

“Docker, please visit the front desk to receive your complimentary upgrade to first class seating.“

That’s right, Docker just received a first class upgrade on Iron.io. A ways back, Travis (our digital frontiers-man of a CTO) announced beta support for Docker. Today, we’re ripping off the beta tag. Docker is our preferred way to package code.

Continue reading “First Class Support for Docker: Better Service, More Legroom”

#GoSF: Identity, Safe Secrets, and IoT Friendly Languages

GoSF at Betable

The Go gopher was designed by Renee French. CC BY 3.0 US

Last night’s meetup, which was hosted by Betable, included two presentations and two lightning talks rounding out a solid evening for the GoSF group. Topics included identity on the web, safe storage of tokens (beyond ENV vars), and even the debut of a new Go-inspired embedded systems language.

Let’s take a look at each!

Continue reading “#GoSF: Identity, Safe Secrets, and IoT Friendly Languages”