Message Queues & Workers: the Heart of Modern Infrastructure

Message Queues + Workers

Thanks to Sonny Abesamis for the base image! CC BY 2.0

Increasingly, message queues and workers are intertwined with the language of modern infrastructure. You might rely on explicit solutions like IronMQ or IronWorker. You might not. Whether you do or don’t is irrelevant: MQs and workers are in everything these days.

MQs and workers are hidden heroes, quietly powering a lot of the technology that many of us rely on. They’re core components in programming languages, MVC frameworks, and even web servers.

As a result, when making infrastructure decisions a good understanding of both MQs and workers is essential. The white paper below will take you from a fuzzy understanding to a well-versed conceptualization for both MQs and workers.

Download the latest white paper from Iron.io: a Refresher on Message Queues & Workers.
Give it a read and let us know what you think!

Helpful hints: Pre-commit

Get going with pre-commit (video)

I had a lot of fun sharing pre-commit with everyone earlier this week. In case you missed it, here’s a link. As a follow up, I whipped together a quick How-To video.

In it, you’ll follow along as I add a pre-commit hook to a brand new Node.js project. Init a git repo locally, and you can the same as you watch. I’ll take you through adding the initial config, picking your hook, and testing to make sure pre-commit works. Continue reading “Helpful hints: Pre-commit”

Polyglot microservices need pre-commit + git

pre-commit plus git equals good for microservices
Thanks to Jeremy Keith for the base image! CC BY 2.0

 

Microservices are great! Or, at least that’s what the internet keeps telling me. There are a lot of upsides, but there are more than a few challenges too.

For example, microservices are a polyglot’s dream. Have a Rails app and a use case where Ruby seems a bit too slow? No problem, microservices and Go are a good starting point.

Sounds great! However, coding in multiple languages brings some hurt, too. A new language means a new style guide. Work at a shop with both a front end Javascript team and a backend Node.js group? Teams working in different domains are likely to enforce their own domain specific best practices.

As you might imagine, it’s a nightmare keeping track of style guides and best practices on a per service basis. If only there were a tool to help manage this mess. Oh, hey there pre-commit!

Continue reading “Polyglot microservices need pre-commit + git”

Favorites from GitHub Universe

GitHub Universe

Remember when enterprise shied away from Open Source software? I sure do. The times they aren’t a changin’.  Led to by the desire to cut costs and create better products, companies are embracing open source. Business today runs on a mix of open and closed solutions.

Nowhere was this more apparent than at the inaugural GitHub Universe conference last week.

Continue reading “Favorites from GitHub Universe”

FAQ Series: What is an overlord-minion architecture?

overlord-minion-architecture

What do you do when even batch processing can’t save you? Yaron suggests giving the overlord-minion pattern a try.

The first step to speed is usually batch processing. All that’s needed there is a server + queue. When the server is ready to work, it grabs a batch of requests from the queue. This works well for most cases! But, what if the queue grows very large? Continue reading “FAQ Series: What is an overlord-minion architecture?”

E is for Event: A Fresh Take on ETL

ETL

As a follow up to my previous post, The Workloads of the Internet of Things, I wanted to walk through a real world example that fully captures the principles of event-driven computing put forth.

Let’s set the stage first… imagine we operate a windmill farm and want to continually track optimal weather conditions to maximize energy output. What basic steps need to be taken?

  1. Sensors capture surrounding weather conditions
  2. Captured data is delivered to a backend service
  3. The service calculates the expected power generation
  4. Calculated data is delivered to an analytics system
  5. Data is presented in a variety of charts and maps

This process flow sounds similar to the common Extract, Transform, Load (ETL) pattern, however the distinction to make here is that data is pushed from the source to the backend service instead of pulled. This means we need to update our pipeline to be more reactive.

Continue reading “E is for Event: A Fresh Take on ETL”

Using IronWorker to Power Custom Service Integrations

The rise of the API economy in recent years has also given the integration economy a much needed breath of new life. What was once a painful process of dealing with proprietary formats and clunky middleware, has now become a streamlined process via openly consumable cloud-native REST APIs. As such, a new breed of services such as Zapier and IFTTT have come along to make API integrations as simple as a few clicks, while other products such as Slack have made integrations a first class citizen feature of the product itself.

The technology that powers many of these integrations behind the scenes is webhooks, essentially an event-driven callback – when this happens, notify that via HTTP POST. Webhooks are an incredibly useful feature with many services, however as developers, we’ll always find a scenario that’s beyond what’s provided out of the box. With service integrations, this often means performing custom data translations such as field mappings, or additional business logic such as filtering and tagging.

We see a lot of our customers bridge services together using IronWorker because of its direct webhook support, flexibility to run any custom logic in any language, ability to scale concurrently behind the scenes, and “serverless” environment. We say “serverless” in quotes because to a developer, there’s never a need to think or worry about provisioning resources to run and manage tasks at scale. When one of our users shared his custom integration using IronWorker on Twitter, I got in touch to hear more and share his story.

Continue reading “Using IronWorker to Power Custom Service Integrations”

How to Build an ETL Pipeline for ElasticSearch Using Segment and Iron.io

ETL is a common pattern in the big data world for collecting and consolidating data for storage and/or analysis. Here’s the basic process:

  • Extract data from a variety of sources
  • Transform data through a set of custom processes
  • Load data to external databases or data warehouses

Continue reading “How to Build an ETL Pipeline for ElasticSearch Using Segment and Iron.io”

Orchestrating PHP Dependencies with Composer and IronWorker

Package your dependencies on IronWorker using composer

 

This is a tutorial describing how to include and use the PHP package management tool Composer with IronWorker.

Composer is a tool for dependency management in PHP. It allows you to declare the dependent libraries your project needs, and it will install them in your project for you. Packagist is the main Composer repository. It aggregates all sorts of PHP packages that are installable with Composer.

Continue reading “Orchestrating PHP Dependencies with Composer and IronWorker”