IronWorker Archives | The Iron.io Blog

Top 10 Uses of a Worker System

A worker system is an essential part of any production-scale cloud application. The ability to run tasks asynchronously in the background, process tasks concurrently at scale, or schedule jobs to run on regular schedules is crucial for handling the types of workloads and processing demands common in a distributed application.

At Iron.io, we’re all about scaling workloads and performing work asynchronously and we hear from our customers on a continuous basis. Almost every customer has a story on how they use the IronWorker platform to get greater agility, eliminate complexity, or just get things done. We wanted to share a number of these examples so that other developers have answers to the simple question “How do I use a worker system?” or “What can I do with a task queue?” Continue reading “Top 10 Uses of a Worker System”

Batch Processing: A Tutorial on Workers, Queueing and Gelato

Worker_Queue_Gelato_V2

Batch processing is one of the earliest ways of data processing, utilized by Herman Hollerith’s Tabulating Machine in 1890. Batch processing was developed to take advantage of scarce computing resources: it avoids idling these expensive resources by queueing instructions to process data without manual user intervention, and can shift workload to times when resources are less scarce¹.

Today, we can leverage modern architectural patterns like worker systems, message queues and the cloud to level-up these advantages and simplify our code. Let’s look at an example of queueing and workers using a calorie-dense metaphor: gelato.

gelato2

Using our favorite local gelato shop as an example, we explore how architectural concepts like queueing and workers can affect a given task. We chose gelato because:

Each order takes time to set up. You must examine the menu and display, choose and order.
Each order takes time to process. The time spent preparing an order can vary based on the order size and complexity, just as job size can vary in a worker system.
Adding additional workers helps. The queue of customers will be processed faster, in the same way that adding more workers to a particular batch processing job will make the queue shrink faster.
Like your infrastructure, gelato must be kept cold, and is delicious when consumed².

Check it out:

¹ https://en.wikipedia.org/wiki/Batch_processing

²This is not true. Iron.io assumes no responsibility if you eat your infrastructure.

How to Bake Your Own Pi

It’s 3/14, and that means it’s international Pi day! A day where we rejoice over the transcendental number that seems to be everywhere.

So, why am I writing about pi on the Iron.io blog? It turns out pi is the best (read: the absolute best!) way to test out computers. It’s sufficiently random, requires large amounts of memory, CPU, and is easy to check.

I first learned about this aspect of pi while reading the book Heres Looking at Euclid. There, I also learned that Pi beyond 40 digits or so isn’t all that useful. So, why do we know pi into the billions of digits? To quote the many time world record holder,

“I have no interest as a hobby for extending the known value of pi itself. I have a major interest for improving the performance of computation. [..] Mathematical constants like the square root of 2, e, and gamma are some of the candidates, but pi is the most effective.”

How To Make Pi

I’m on board! I want to make Pi, myself. If Pi is a great way to test any computer, why not use it to test first-class distributed computing solutions, like IronWorker?

Humans have known about Pi for a while. Which is part of what makes it a great computation. We have multiple recipes for baking the same dish. That means it’s easy to check our work (by comparing two algorithms).

So, what goes into pi? How can I cook this dish? Let’s check out a few of the best recipes. Continue reading “How to Bake Your Own Pi”

The Next Frontier: Learning Microservices in the Classroom

As a Customer Success engineer here at Iron.io, I’ve been fortunate enough to see people using Iron.io in ways I never thought about. It’s actually one of my favorite parts of my job.

Recently, I was chatting with a customer who mentioned his students were using Iron.io in their final project. This peeked my interest, so I interviewed Soumya Ray, an associate professor at National Tsing Hua University in Taiwan, about his experience. Professor Ray’s Service Oriented Architecture class is an 18 week course that takes students from idea creation to final product. And, as a cherry on top, the class has students create the building blocks of their own startup with zero dollars spent. Continue reading “The Next Frontier: Learning Microservices in the Classroom”

Introducing Custom Docker Images, Private Docker Repositories and Environment Variables

We’re happy to announce three awesome new IronWorker features:

Custom Docker Images for all and Docker is now the default code packaging mechanism
Support for private Docker images on any Docker Registry, including Docker Hub
Support for custom environment variables

I’ll explain each of these features in more detail below.

Custom Docker Images for All!

Previously only available to customers on dedicated plans, this is now available to everyone! You can create custom Docker images and run them at scale on the Iron.io Platform. And as usual, you don’t need to think about servers or scaling or managing anything, you just queue up jobs/tasks. Jobs are executed using your Docker image + a message/payload that defines that job.

To read how to make your own Docker Worker using the language of your choice, please see our Docker Worker GitHub repository for full documentation for most languages.

There is a size limit to custom images on our small/free plans of 200MB so you’ll definitely want to use our iron/x base images that we use in our examples to keep them small. If you need bigger images, you can upgrade your account.

Once you’ve pushed your image to Docker Hub, simply let Iron.io know about it:

Then you can start queuing up jobs for it.

You can see the full API here and client libraries for the language of your choice here.

Private Docker Repositories

Not only can you use your own custom images, you can store those images privately and still use them on IronWorker. Obviously you don’t want other people to access your code inside your image or any type of config files you might have put in the image (although we recommend using environment variables for that, see next section) so you can use a private Docker repository to keep it private.

To use your private images on Iron.io, you need to login like you do with Docker:

Then just do everything else like normal.

Environment Variables

Instead of uploading a config file as json, we’ve added support for custom environment variables that will be passed into your Docker container at runtime. This allows you to set options that you don’t want to include in your Docker image, such as database credentials or variables that might change based on the environment (development vs production for instance).

These are set by using -e flags on iron register, for example:

That’s it for Now

These new features give you full capability to use Docker to it’s full potential. Now that this is in place, there’s a lot of exciting new things to come that will build on this.

Best Practices and Anti-Patterns for Workers and MQs

Thanks to Ruth Hartnup for the base image! CC BY 2.0

If you’ve been programming for a while, it’s probable that someone, somewhere, has recommended the Gang of Four book.

The book dissects Object Oriented programming. It lists numerous ways of royally messing things up, but it’s claim to fame is that it also lists ways to do it right! These well tested paths to success often come with explanations for when to use them and why they’re good at avoiding common pitfalls.

These are design patterns. They’re embedded in the culture of programming, and they’re an amazing way to learn from others’ mistakes. At the outset of any project, a lot of paths are open to you. Design patterns illuminate the dark paths from the healthy, low stress approaches.

Today, we’re releasing our very own set of best practices and anti-patterns in the form of a white paper. It’s a quick read and will save you time as you tinker on your own workers and message queues. So, what do you have to lose?

Continue reading “Best Practices and Anti-Patterns for Workers and MQs”

Docker + Iron.io = Super Easy Batch Processing

There is a ton of use cases for batch processing and every business is probably doing it in some way or another. Unfortunately, many of these approaches take much longer than need be. In a world of ever increasing data, the old way can now hinder the flow of business. Some examples of where you’ll see batch processing used are:

Image/video processing
ETL – Extract Transform Load
Genome processing
Big data processing
Billing (create and send bills to customers)
Report generation
Notifications – mobile, email, etc.

We’ve all seen something that was created during a batch process (whether we like it or not).

Now, I’m going to show you how to take a process that would typically take a long time to run, and break it down into small pieces so it can be distributed and processed in parallel. Doing so will turn a really long process into a very quick one.

We’ll use Docker to package our worker and Iron.io to do the actual processing. Why Docker? Because, we can package our code and all our dependencies into an image for easy distribution. Why Iron.io? It’s the easiest way to do batch processing. I don’t need to think about servers or deal with distributing my tasks among a bunch of servers.

Alright, so let’s go through how to do our own batch processing.

Continue reading “Docker + Iron.io = Super Easy Batch Processing”

See How Untappd Processes 100s of 1000s of Updates a Night

Untappd, a mobile check-in app for beer lovers, helps beer lovers share their passion with friends and other beer lovers from around the world.

With millions of users checking-in, tracking and sharing newly discovered beers and beer drinking locations, as well as earning points for coveted badges, the Untappd app’s processing power is tested daily.

Watch the Untappd video to see how the team was able to make the app an integrated part of many beer lovers night out. Continue reading “See How Untappd Processes 100s of 1000s of Updates a Night”

Defrag 2015, Our Favorites From Day One

Yesterday was the start of the 9th annual Defrag conference! What is Defrag?

Defrag describes itself as: “We explore the frontiers of technology’s intersections with society, government, education, healthcare, and commerce through discussions and sessions rooted in topics like cloud computing, APIs, mobile technologies, big data, devops, the internet of things, and next generation human computer interaction.”

Neat! Read on to hear highlights from the first day of festivities.

Continue reading “Defrag 2015, Our Favorites From Day One”

First Class Support for Docker: Better Service, More Legroom

Thanks to Richard Moross for the base image CC BY 2.0

“Docker, please visit the front desk to receive your complimentary upgrade to first class seating.“

That’s right, Docker just received a first class upgrade on Iron.io. A ways back, Travis (our digital frontiers-man of a CTO) announced beta support for Docker. Today, we’re ripping off the beta tag. Docker is our preferred way to package code.

Continue reading “First Class Support for Docker: Better Service, More Legroom”