IBM MQ (IBM Message Queue): Overview and Comparison to IronMQ

Wherever two or more people need to wait in line for something, you can guarantee that there will be a queue: supermarkets, bus stops, sporting events, and more.

It turns out, however, that queues are also a useful concept in computer science.

The data structure known as a queue is a collection of objects that are stored in consecutive order. Elements in the queue may be removed from the front of the queue and inserted at the back of the queue (but not vice versa).

Queues are a good choice of data structure when you have items that should be processed one at a time in first-in, first-out (FIFO) order–in particular, the message queue. In this article, we’ll discuss IBM MQ, one of the most popular solutions for implementing message queues, and see how it stacks up against Iron.io’s IronMQ software.

What is a Message Queue?

As the name suggests, a message queue is a queue of messages that are sent between different software applications. Messages consist of any data that an application wants to send, as well as a header at the start of the message that contains information about the data below it.

Message queues are necessary because different applications consume and produce information at different speeds. For example, one application may sporadically create large volumes of messages at the same time, while another application may slowly process messages, one after another.

The differing speeds at which these two applications operate can pose an issue. Because they are all produced at the same time, all but one of the first application’s messages will be lost by the second application–unless they can be temporarily stored within a message queue.

A message queue is a classic example of asynchronous communication, in which messages and responses do not need to occur at the same time. The messages that are placed on the queue do not require an immediate response, but can be postponed until a later time. Emails and text messages are other examples of asynchronous communication in the real world.

While implementing a basic message queue is a fairly straightforward task, complex IT environments may include communications between separate operating systems and network protocols. In addition, basic message queues may not be resilient when the network goes down, causing important messages to be lost.

For these reasons, many organizations have chosen to use “message-oriented middleware” (MOM): applications that make it easier for different software and hardware components to exchange messages.

There are a variety of MOM products on the market today (like Delayed Job or Sidekiq in the Ruby on Rails world), each one intended for different situations and use cases. In the rest of this article, we’ll compare and contrast two popular options for exchanging data via MOM software: IBM MQ and IronMQ.

What to Consider When Selecting a Message Queue

Message queues are essential to how different applications interact and exchange data within your IT environment. This means, of course, that choosing the right message queue solution is no easy task–picking the wrong one can drastically affect your performance.

Below, we’ll discuss some of the most important factors to consider when choosing a message queue solution.

Features

Depending on the specifics of your IT environment, you may require any number of different features from your message queue. Here’s just a small selection of potential functionality:

Pushing and/or pulling: Most message queues include support for both pushing and pulling when retrieving new messages. In the first option, new messages are “pushed” to the receiving application in the form of a direct notification. In the second option, the receiving application chooses to “pull” new messages itself by checking the queue at regular intervals.

Delivery options: You may want to schedule messages at a specific time, or send messages more than once in order to make sure that they are delivered. If so, choose a message queue that includes support for these features.
Message priorities: Some messages are more critical or urgent than others. In order to receive the information you need in a timely manner, your message queue may use some way of migrating important messages up the queue (just like letting late passengers cut in front of you at the airport).
Persistence: Messages that are persistent are written to disk as soon as they enter the queue, while transient messages are only written to disk when the system is using a large amount of memory. Persistence improves the redundancy of your messages and ensures that they will be processed even in the event of a system crash.

Scalability and Performance

The more complex your IT environment is, the more difficult it is to scale it all at once. Instead, you can scale each application independently, decoupled from the rest of the environment, and use a message queue to communicate asynchronously.

Certain message queue solutions are better-suited for improving the scalability and performance of your IT environment. Look for options that are capable of handling high message loads at a rapid pace.

Pricing

Different message queue solutions may have different price points and pricing models that lead you to choose one over the other.

“As a service” is currently the predominant pricing model for message queues. This means that customers have a “pay as you go” plan in which they are charged by the hours and the computing power that they use. However, there are also prepaid message queue plans with an “all you can eat” pricing model.

Security

With hacks and data breaches constantly in the news, maintaining the security of your message queue should be a primary concern. Malicious actors may attempt to insert fraudulent messages into the queue and use them to exfiltrate data or gain control over your system.

Message queue solutions that use the Advanced Message Queuing Protocol (AMQP) include support for transport-level security. In addition, if the contents of the message itself may be sensitive, you should look for a solution that encrypts messages while in transit and at rest within the queue.

What is IBM MQ (IBM Message Queue)?

IBM Message Queue (IBM MQ) is a MOM product from IBM that seeks to help applications communicate and swap data in enterprise IT environments. IBM MQ calls itself a “flexible and reliable hybrid messaging solution across on-premises and clouds.” It includes support for a variety of different APIs, including Message Queue Interface (MQI), Java Message Service (JMS), REST, .NET, IBM MQ Light, and MQTT.

The IBM MQ software has been around in some form since 1993. Thanks to the widespread demand for real-time transactions on the Internet, IBM MQ and other message queue solutions have enjoyed a renewed popularity in recent years.

The benefits of using IBM MQ include:

Support for on-premises, cloud, and hybrid environments, as well as more than 80 different platforms.
Advanced Message Security (AMS) for encrypting and signing messages between applications.
Multiple modes of operation, including point-to-point, publish/subscribe, and file transfer.
A variety of tools for managing and monitoring queues, including MQ Explorer, the MQ Console, and MQ Script Commands.

On websites such as TrustRadius and IT Central Station, IBM MQ users mention a few advantages and disadvantages of the software. Some of the recurring themes in these reviews are:

IBM MQ is reliable and does its job well, without any lost messages.
The software helps to improve data integrity, availability, and security.
The user interface may be a little unintuitive and challenging, especially for first-time users.
Tools such as MQ Explorer seem to be “aging” and are not as effective as third-party solutions.
IBM MQ lacks certain integrations that would be useful in a modern IT enterprise environment.

IBM MQ vs. IronMQ

There’s no doubt that IBM MQ is a robust, mature message queue solution that fits the needs of many organizations. However, it’s far from the only MOM software out there. Offerings such as Iron.io’s IronMQ are highly viable message queue alternatives, and in many cases may be superior to market leaders such as IBM MQ.

What is IronMQ?

IronMQ is a messaging queue solution from Iron.io, a cloud application services provider based in Las Vegas. According to Iron.io, the IronMQ message queue is “the most industrial-strength, cloud-native solution for modern application architecture.”

The software includes support for all major programming languages and is accessible via REST API calls. Iron.io offers a number of different monthly and annual pricing models for IronMQ, ranging from the hobbyist all the way up to the large enterprise.

The benefits of IronMQ include:

Support for both push and pull queues, as well as “long polling” (holding a pull request open for a longer period of time).
The use of multiple clouds and availability zones, making the service highly scalable and resistant to failure. In the event of an outage, queues are automatically redirected to another zone without any action needed on the part of the user.
Backing by a high-throughput key/value data store. Messages are preserved without being lost in transit, and without the need to sacrifice performance.
Flexible deployment options. IronMQ can be hosted on Iron.io’s shared infrastructure or on dedicated hardware to improve performance and redundancy. In addition, IronMQ can run on your internal hardware in cases where data must remain on-premises.

IBM MQ vs. IronMQ: The Pros and Cons

Both IBM MQ and IronMQ are cloud-based solutions, which means they enjoy all the traditional benefits of cloud computing: better reliability and scalability, faster speed to market, less complexity, and so on.

Since it was created with the cloud in mind, IronMQ is particularly well-suited for use with cloud deployments. Because IronMQ uses well-known cloud protocols and standards such as HTTP, JSON, and OAuth, cloud developers will find IronMQ exceedingly simple to work with.

IronMQ users enjoy access to an extensive set of client libraries, each one with easy-to-read documentation. The IronMQ v3 update has also made the software faster than ever for customers who need to maintain high levels of performance.

Customers who already use Iron.io’s IronWorker software for task management and scheduling will find IronMQ to be the natural choice. According to one IronMQ user in the software industry, “I can run my Workers and then have them put the finished product on a message queue – which means my whole ETL process is done without any hassle.”

On the other hand, because it’s part of the IBM enterprise software family, IBM MQ is the right choice for organizations that already use IBM applications. If you already have an application deployed on IBM WebSphere, then it will be easier to simply use it together with IBM MQ.

What’s more, IBM MQ is capable of working well in many different scenarios with different technologies, including mainframe systems. However, some customers report that IBM MQ has a clunky, “legacy” feel to it and is difficult to use in an agile IT environment.

While it’s definitely able to compete with IBM MQ, IronMQ also stacks up favorably against other message queue solutions such as RabbitMQ and Kafka. For example, RabbitMQ’s use of the AMQP protocol means that it is more difficult to use and can only be deployed in limited environments. According to various benchmarks, IronMQ is roughly 10 times as fast as RabbitMQ.

IronMQ Customer Reviews

Of course, reading long lists of software features can only go so far–you need customer feedback in order to make sure that the application really does what it says on the tin.

The good news is that IronMQ has a number of happy customers who are all too eager to share their positive experiences. John Eskilsson, technical architect at the engineering firm Edeva, raves about IronMQ in his testimonial on FeaturedCustomers:

“IronMQ has been very reliable and was easy to implement. We can take down the central server for maintenance and still rely on the data being gathered in IronMQ. When we start up the harvester again, we can consume the queue in parallel using IronWorker and be back to real-time quickly.”

In a review on G2, one user working in marketing and advertising praised IronMQ’s reliability and performance:

“My experience with the message queues was a good one. I had no issues and found the message queues to be very reliable. The website has good monitoring showing exactly what is happening in real time.”

The world’s most popular websites may receive millions of page hits per day, and more during times of peak activity. Businesses such as CNN need a robust, feature-rich, highly available message queue solution in order to get the right information to the right people. CNN is one of many enterprise clients that uses IronMQ as its message queue solution.

IBM MQ vs. IronMQ: Which is Right for You?

At the end of the day, no one can tell you which message queue solution is right for your company’s situation. Both IBM MQ and IronMQ have their advantages and drawbacks, and only one may be compatible with your existing IT infrastructure.

In order to make the final decision, draw up a list of the features and functionality that are most important to you in a message queue. These may include issues such as persistence, fault tolerance, high performance, compatibility with existing software and hardware, and more.

Fortunately, you can also try IronMQ before you buy. Want to find out why so many clients are proud to use IronMQ and other Iron.io products? Request a demo of IronMQ, or sign up today for a free, full-feature 14-day trial of the software.

Amazon SQS (Simple Queue Service): Overview and Tutorial

What’s a Queue? What’s Amazon SQS?

Queues are a powerful way of combining software architectures. They allow for asynchronous communication between different systems, and are especially useful when the throughput of the systems is unequal. Amazon offers their version of queues with Amazon SQS (Simple Queue Service).

For example, if you have something like:

System A – produces messages periodically in huge bursts
System B – consumes messages constantly, at a slower pace

With this architecture, a queue would allow System A to produce messages as fast as it can, and System B to slowly digest the messages at it’s own pace.

Queues have played an integral role in software architecture for decades along with core technology concepts like APIs (Application Programming Interfaces) and ETL/ELT (Extract, Load Transform). With the recent trend towards microservices, have become more important than ever.

Amazon Web Services

AWS (Amazon Web Services) is one of the leading cloud providers in the world, and anyone writing software is probably familiar with them. AWS offers a wide variety of “simple” services that traditionally had to be implemented in-house (eg, storage, database, computing, etc). The advantages offered by cloud providers are numerous, and include:

Better scalability – your data center is a drop in their ocean. They’ve got mind-boggling capacity. And it’s spread around the world.
Better reliability – they hire the smartest people in the world (oodles of them) to ensure these services work correctly, all the time.
Better performance – you can typically harness as much computing horsepower as you’d like with cloud providers, far exceeding what you could build in-house.
Better (lower) cost – nowadays, they can usually do all this cheaper than you could in your own data center, especially when you account for all the expertise they bring to the table. And many of these services employ a “pay as you go” model, charging for usage as it occurs. So you don’t have to pay the large up front cost for licenses, servers, etc.
Better security – their systems are always up to date with the latest patches, and all their smart brainiacs are also thinking about how to protect their systems.

If you have to choose between building out your own infrastructure, or going with something in the cloud, it’s usually an easy decision.

AWS Simple Queue Service

It comes as no surprise that AWS also offers a queueing service, simply named AWS Simple Queue Service. It touts all the cloud benefits mentioned before, and also features:

Automatic scaling – if your volume grows you never have to give a thought to your queuing architecture. AWS takes care of it under the covers.
Infinite scaling – while there probably is some sort of theoretical limit here (how many atoms are in the universe?), AWS claims to support any level of traffic.
Server side encryption – using AWS SSE (Server Side Encryption), messages can remain secure throughout their lifetime on the queues.

Their documentation is also top-notch. It’s straightforward to get started playing with the technology, and when you’re ready for serious, intricate detail, the documentation goes deep enough to get you there.

Example

Let’s walk through a simple example of using AWS SQS, using the line at the DMV (Department of Motor Vehicles) as the example subject matter. The DMV is notorious for long waits, forcing people to corral themselves into some form of a line. While this isn’t an actual use case anyone would (presumably) solve using AWS SQS, it will allow us to quickly demo their capabilities, with a real-world situation most are all too familiar with.

While AWS SQS has SDK libraries for almost any language you may want to use, I’ll be using their REST interface for this exercise (with my trusted REST side kick Postman!).

Authorization

Postman makes it easy to setup all the necessary authorization using Collections. Configure the AWS authorization in the parent collection with the Access Key and Secret Access Key found in the AWS Console:

Then reference that authorization in each request:

Using this pattern, it’s easy to quickly spin up requests and put AWS SQS through its paces.

Creating a Queue

When people first walk in the door, any DMV worth their salt will give them a number to begin the arduous process. This is your main form of identification for the next few minutes/hours (depending on that day’s “volume”), and it’s how the DMV employees think of you (“Number 14 over there sure seems a bit testy!”).

Let’s create our “main queue” now, with the following REST invocation:

Request:

GET https://sqs.us-east-1.amazonaws.com?Action=CreateQueue&DefaultVisibilityTimeout=0&QueueName=MainLine&Version=2012-11-05

Response:

https://sqs.us-east-1.amazonaws.com/612055710376/MainLine

fa178e12-3178-5318-8d90-da20904943f0

Good deal. Now we’ve got a mechanism to track people as they come through the door.

Standard vs FIFO

One important detail that should be mentioned – there are two types of queues within AWS SQS:

Standard – higher throughput, with “at least once delivery”, and “best effort ordering”.
FIFO (First-In-First-Out) – not as high throughput, but guarantees on “exactly once” processing, and preserving the ordering of messages.

Long story short, if you need things super fast, can tolerate messages out of order, and possibly sent more than once, Standard queues are the answer. If you need absolute guarantees on order of operations, no duplication of work, and don’t have huge throughput needs, then FIFO queues are the best choice.

We’d better make sure we create our MainLine queue using FIFO! While a “mostly in order” guarantee might suffice in some situations, you’d have a riot on your hands at the DMV if people started getting called out of order. Purses swinging, hair pulling – it wouldn’t be pretty. Let’s add “FifoQueue=true” to the query string to indicate that the queue should be FIFO:

Request

https://sqs.us-east-1.amazonaws.com?Action=CreateQueue&DefaultVisibilityTimeout=0&QueueName=MainLineFIFO&Version=2012-11-05&FifoQueue=true

Send Message

Now that we’ve got a queue, let’s start adding “people” to it, using the “SendMessage” action. Note that when using REST, we need to URL encode the payload. So something like this:

{
"name": "Ronnie Van Zandt",
"drivers_license_number": "1234"
}

Becomes this:

%7B%0A%20%20%20%22name%22%3A%20%22Ronnie%20Van%20Zandt%22%2C%0A%20%20%20%22drivers_license_number%22%3A%20%221234%22%0A%7D

There are many ways of accomplishing this, I find the urlencoder site to be easy and painless.

Here’s the final result:

Request

https://sqs.us-east-1.amazonaws.com/612055710376/MainLineFIFO?Version=2012-11-05&Action=SendMessage&MessageBody=%7B%0A%20%20%20%22name%22%3A%20%22Ronnie%20Van%20Zandt%22%2C%0A%20%20%20%22drivers_license_number%22%3A%20%221234%22%0A%7D

Response:

00ad4e10-4394-450f-8902-4a9cf4b96b95

b9f28edc9c6dc9fe2a86f5ae8efb2364

97a41dd4-5d15-59e0-b9f5-49e02fb4384d

After this call, we’ve got young Ronnie standing in line at the DMV. Thanks to AWS’s massive scale and performance, we can leave Ronnie there as long as we’d like. And we can add as many people as we’d like – with AWS SQS’s capacity, we could have a line around the world. But that’s horrible customer service, someone needs to find out what Ronnie needs!

ReceiveMessage

At the DMVs I’ve been to, there’s usually a large electronic sign on the counter that will display the next lucky person’s number. You feel a brief pulse of joy when your number displays, and rush to the counter on a pillow of euphoria, eager to get on with your life. How do we recreate this experience in AWS SQS?

Why, “ReceiveMessage”, of course! (Note we are invoking it using the actual QueueUrl passed back by the CreateQueue call above)

Request

https://sqs.us-east-1.amazonaws.com/612055710376/MainLineFIFO?Action=ReceiveMessage&Version=2012-11-05

Response

00ad4e10-4394-450f-8902-4a9cf4b96b95

AQEBjq8apWDfLXE0pCbpABh6Wdx70ZbszY0k38t9u8Mrny1Jz+Q522Vwvvf4xLqzQHfjoHQd56JJJEM67LJG5tQ/YSCibFSNCg8jfadyNMbqBH48/WxmpYunI3w1+GbDCL2tlKkDz/Lm9akGasgDZEBtw6U9jw1Bu6XbzNuNiw5jfVzjC99E38KSvxvZMHfmSi3Wo2XOBAcfU0oTpLmGMwccGiRUOp4XtS38nMXHhBdtKSS+U11N38cJAtlnxHQJkXmTAk7ZdvpxJNtnOrXmeGN00vtf6OSyLJzRJJieYHNtxIyxojcGZcnJQ6dTveMWQ1A1FOzschRuavl3wtftDS/YSt5sDNeBcjEOE+Y0QE+18qiWaDZc+nlaetcBvqmt6Hbt

b9f28edc9c6dc9fe2a86f5ae8efb2364

{
"name": "Ronnie Van Zandt",
"drivers_license_number": "1234"
}

6a43b589-940c-52a4-bc62-e1bde75e22e4

One thing to keep in mind – ReceiveMessage doesn’t actually REMOVE the item from the queue – the item will remain there until explicitly removed. Visibility Timeout can be used to ensure multiple readers don’t attempt to process the same message.

So how do we permanently mark the item as “processed”? By deleting it from the queue!

DeleteMessage

The DeleteMessage action is what removes items from a queue. There’s not really a good analogy with the DMV here (thankfully, DMV employees can’t “delete” us), so we’ll just go with an example. DeleteMessage takes the ReceiptHandle returned by the ReceiveMessage endpoint as a parameter (once again, encoded):

Request

https://sqs.us-east-1.amazonaws.com/612055710376/MainLineFIFO?Action=DeleteMessage&Version=2012-11-05&ReceiptHandle=AQEBjq8apWDfLXE0pCbpABh6Wdx70ZbszY0k38t9u8Mrny1Jz%2BQ522Vwvvf4xLqzQHfjoHQd56JJJEM67LJG5tQ%2FYSCibFSNCg8jfadyNMbqBH48%2FWxmpYunI3w1%2BGbDCL2tlKkDz%2FLm9akGasgDZEBtw6U9jw1Bu6XbzNuNiw5jfVzjC99E38KSvxvZMHfmSi3Wo2XOBAcfU0oTpLmGMwccGiRUOp4XtS38nMXHhBdtKSS%2BU11N38cJAtlnxHQJkXmTAk7ZdvpxJNtnOrXmeGN00vtf6OSyLJzRJJieYHNtxIyxojcGZcnJQ6dTveMWQ1A1FOzschRuavl3wtftDS%2FYSt5sDNeBcjEOE%2BY0QE%2B18qiWaDZc%2BnlaetcBvqmt6Hbt

Response

a69c7042-d0e2-546a-bdf7-2476a30b89df

And just like that, Ronnie is able to leave the DMV with his newly printed license, all thanks to AWS SQS!

DMV line — It’s time to get out of here!

IronMQ vs AWS SQS

While AWS SQS has many strengths, there are advantages to using Iron MQ that make it a more compelling choice, including:

Client Libraries

Iron MQ features an extensive set of client libraries, with clear, straightforward documentation . Getting started with Iron MQ is a breeze. After playing with both SDKs, I found the Iron MQ experience to be easier.

Speed

Iron MQ is much faster than SQS, with V3 making it faster and more powerful than ever before. And with high volume systems, bottlenecks in your messaging architecture can bring the whole system to its knees. Faster is better, and Iron MQ delivers in this area.

Push Queues

Iron MQ offers something called Push Queues, which supercharge your queueing infrastructure with the ability to push messages OUT. So rather than relying solely on services pulling messages off queues, this allows your queues to proactive send the messages to designated endpoints, recipients, etc. This powerful feature expands the communication options between systems, resulting in faster workflow completion, and more flexible architectures.

Features

Check out the comparison matrix between Iron MQ and its competitors (including SQS). It clearly stands out as the most feature-rich offering, with functionality not offered by SQS (or anyone else, for that matter).

Iron MQ offers a free 14 day trial to see for yourself how it compares to SQS. Signup here.

In Con-q-sion

Hopefully this simple walkthrough is enough to illustrate some possibilities of using AWS SQS for your queuing needs. It is easy to use, with incredible power, and their SDK supports a variety of language. And may your next trip to the DMV be just as uneventful as young Ronnie’s.

Happy queueing!

Iron at APIdays, see you there?

First off, we’re giving away a few free tickets to the SF APIdays conference on July 31st. Comment about your favorite API on this post for a chance to win a free ticket!

With the freebie out of the way, we’re huge fans of APIdays (and API’s in general) and love to reference this landscape diagram. If the landscapes weren’t moving so fast, we’d probably have a copy printed on our office wall alongside the Cloud Native Landscape diagram.

API’s are everywhere

As engineers, most of us are inherently API minded. Others, not so much. It’s only been in the last 5 or 6 years that the idea behind API’s has gained public mindshare. Following Executive Order 13571 in 2011, the Obama administration directed federal agencies to deploy Web APIs which put API’s in the public spotlight. There’s been a lot of progress in the public sector, and now we’re holding conferences about API’s in general. These are steps in the right direction.

Iron <3’s API’s

We build all of our products with API’s in mind. All of our client libraries for each of our products use our HTTP API’s, and we’ve received a lot of praise for building API centric and cloud agnostic services. Internally we rely on a lot of API’s as well. We use API management solutions like DreamFactory to coordinate data sources, RingCaptcha for SMS verification and Zapier to tie disparate services together. We obviously use all of the public cloud API’s directly as well.

What API’s do you use?

There are many others API’s that we use that I didn’t list. What are some of your favorite API’s? Comment below and you might be sent a free ticket to APIdays. If you’re already going, let us know as we’d be happy to meet up!

Iron Enters the Manifold

We’re happy to announce that Iron’s suite of products are now publicly available on the Manifold marketplace (see the bottom of this post for a coupon code you’ll want to redeem).

If you haven’t heard of Manifold before, you definitely need to check them out. They’re an application marketplace that lets you provision and manage services without being tied to a specific platform. For those of you familiar with Heroku, you can think of it as their Add-ons marketplace but without being tied to Heroku the platform. Manifold itself was founded by ex-Heroku leadership, so there’s plenty of best practice and real-world experience being brought to the table with their offering.

With Manifold, you have the freedom to deploy your apps wherever you’d like. Tight Terraform and Kubernetes integrations mean you aren’t constrained to a single platform. This is extremely important. You’re future proofing your 3rd party integrations. You’re minimizing the operational expenditure hit you’d normally face when migrating to new infrastructure. This is something we pride ourselves with at Iron as well, and a lot of our customers like having the insurance that if they move off a public cloud elsewhere, we can deploy on-premise and easily move with them.

We’re proud to be a part of the Manifold platform and highly recommend trying it. As part of this release, they’re offering $25 in Manifold credits if you use the coupon code “IRON”.

It’s extremely easy to get started and we definitely think it’s the way developers will be building applications moving forward.

3 Key Benefits to Container-Based Background Job Processing

Whether deploying applications or providing microservices, being able to get tasks done in the background without user intervention is key to operating efficiently for IT and development teams. One effective way to facilitate background job processing is with the help of containers.

Container-based background job processing comes with a whole host of benefits. Here are some of the key benefits of using container-based background job processing that IT and development teams can leverage.

Enhanced Security

With ever-increasing data breaches and ransomware threats, keeping applications secure during deployment is vital. Managing the deployment of applications often calls for working with several development teams distributed across different locations. Having more people work on these teams can create a higher risk of exposure and data breaches due to errors or vulnerabilities from mistakes by the staff.

The great news is that containers offer enhanced security. That’s because more effort has been put in place to safeguard containers. For instance, container systems and container management systems, such as Docker and Kubernetes, require container image signing to ensure your team is deploying containers from trusted resources.

Moreover, container scanning solutions also help enhance security by quickly identifying vulnerabilities that may exist in your containers, including the containers that were signed. This helps reduce security risks, including deploying unsafe containers.

Versatile Background Job Capabilities

Being able to provide on-time delivery to clients is essential for enhancing the customer’s experience. With the help of container-based background job processing, IT and development teams can manage a variety of background tasks.

For instance, tasks such as email delivery, automated scaling, calculating bandwidths or automating push notifications can be handled by containers. That’s because containers can fragment applications into smaller components while enabling communication among developer teams. This also helps facilitate speedy software development and testing. Moreover, using a container-based workload platform from development tool expert services, such as Iron IO, helps enterprises free up staff from handling background job processing so they can focus on more vital tasks, such as testing and developing their software applications.

Flexible Deployment

Thanks to the container’s shareability, enterprises can leverage flexible deployment options, including the shared, on-premise, dedicated or hybrid deployment options offered by a reliable container-based workload hosted platform, such as Iron IO’s Worker. That means enterprise leaders can choose a deployment option that’s customized to their needs.

For instance, development teams working in enterprises that often deal with classified or highly sensitive data or personal information, such as banks, hospitals or federal agencies, often have to follow several compliance regulations. Having the ability to use on-premise deployment solutions can help support background tasks in a secure manner.

At the same time, enterprises that must support staying in compliance with enterprise and federal rules while facilitating a distributed team may find a hybrid deployment approach more feasible. This deployment option is ideal for handling secure background job processing for tasks, such as scheduling and authentication, while letting development teams run their containers on-premise.

Final Thoughts

From flexible deployment options to versatile background task processing capabilities, containers offer much for development teams to leverage. While containers provide several benefits, it’s important to also use reputable platforms and professional teams that have the experience and expertise in managing and implementing containers to support container-based background jobs. By leveraging containers and the platforms that support them, enterprises can better serve their clients for an enhanced customer experience.

Iron’s East/West Coast Drink-up

A bunch of Iron employees will be out and about in April, looking to meet up with customers to chat about our up and coming platform changes. Beer (or wine, or cocktails, or <insert drink here>) will be on us! We’re sticking to the east and west coasts for now, and our current plans are:

April 5th, San Francisco
April 14th, Boston
April 15th, NYC
April 17th, Los Angeles

If you’re interested in attending, fill out the form below. We’ll be in touch with the details once we have them confirmed on our end. Cheers!

Loading…

A Serverless Message Queue Without the Glue

More and more technologies get involved as systems grow, and it’s sometimes hard to keep track of what’s doing what. Caching layers, message queues, serverless functions, tracing frameworks… the list goes on. Once you start sprinkling in public cloud services, you may find yourself developing your way into vendor lock-in. All of the sudden, you’re dealing with one cloud, tons of services, and having to glue everything together in order to make the services talk to each other. One of Iron’s primary goals is to make life easier for developers, and IronMQ’s little known “Push Queue” feature is one that can help prevent you from having to write the glue.

What are Push Queues?

IronMQ has a built-in feature called Push Queues, which when enabled, fire off an event any time a message gets pushed onto that queue. This comes in extremely handy when you immediately want to “do something” (or many things) with that message. With traditional message queues, you’d usually need to write another process that polls your queues for messages at a given duration. MQ’s push queues can fire off events to different types of endpoints, each extremely helpful in their own way.

What Type of events can be triggered?

HTTP
When a message gets put onto your push queue, IronMQ can make a POST request (with the message in the request body) to any URL of your choice. This is extremely handy when you want to notify other systems that some sort of event just happened or kick off another process.

MQ
Inception! You can have the delivery of a message populate another IronMQ queue. This is helpful if you want to tie multiple queues together or create a dead letter queue for example.

Worker
MQ can connect directly to IronWorker and pass its message as the payload to one of your jobs. How cool is that!? In order to exemplify how cool that actually is, we’ll run through a real-life scenario.

MQ & Worker Example

Let’s say you have a time-sensitive nightly job that processes uploaded CSV files. It needs to process all of the files uploaded during that day and finish within a set amount of time. As your system grows and there are more CSV files to process, your nightly process starts running behind schedule.

You realize that a lot of the time spent in your nightly worker is spent formatting the CSV file into the correct format. It would make sense to split this process into two distinct stages, formatting and processing. When a CSV file is received, you could send a message to your push queue which in turn will kick off a “formatting” worker job to pre-process the CSV file into the correct format. Your nightly “processing” worker job will then be able to fly through the CSV files because it no longer needs to fix any formatting issues.

The beauty here is that you can continue to add more push events to the queue. When a file is uploaded, maybe you also need to ping another worker that handles OCR or post an update to an external HTTP endpoint letting it know exactly “what” file was uploaded. Without a push queue, you’d be adding a lot of custom code to handle these requests, retries, errors, etc. IronMQ’s push queues take care of all of this for you.

How can I configure a Push Queue?

Retries
You can configure your queue to allow for a custom amount of retries, a custom delay between retries, and even provide another queue to store failed push attempts. For example, using an HTTP event, MQ will retry pushes (3 times by default) every time it receives a non-200 response code.

Timeouts
If your event never receives a response after a certain period of time (10 seconds by default), it will chalk that up as a failed attempt and retry.

Unicast or Multicast?
You can even fire off multiple events from one queue. If you need to trigger one HTTP endpoint and also fire off a Worker job, that’s not a problem.

How do I create a push queue?

Creating one is straightforward. Here’s an example cURL request that creates a multicast Push Queue with an HTTP endpoint as well as a Worker endpoint.

IronMQ has client libraries available in most languages, so you can easily create one programmatically as well. Here’s an example in PHP:

Conclusion

With one IronMQ Push Queue, you can make a lot happen. If you were to try and replicate a multicast Push Queue in a traditional message queue, for example, you’d end up writing a lot of custom code to glue everything together. You’d also have to deal with scaling your infrastructure as your message queue needs grew. With IronMQ, you can save time and money focusing on your applications business logic, and less time on glue and infrastructure. For more detailed information about Push Queues, visit the IronMQ documentation.

If you’re interested in knowing more about IronMQ, or want to chat about how we may be able to help, call us anytime at 888-501-4766 or email at support@iron.io.

Docker, Inc isn’t Dead

Chris Short recently wrote up a piece entitled Docker, Inc is Dead, with a prediction that the company would no longer exist sometime in 2018. It’s well written and he does a good job of running through some of Docker’s history in recent years. Although I agree with some of his sentiments, I don’t think Docker, Inc will exit the stage anytime soon. Here are some reasons I think Docker, Inc will live a healthy life in 2018.

Docker is Good Software

This was the first point in Chris’ piece, and he’s right. Docker definitely helped widen the spotlight on *n?x kernels. Discussions around namespaces, cgroups, lxc, zones, jails, etc… lit up across communities in different disciplines. Dockers’ simple interface lowered the barrier of entry for non-administrators, and the developer community immediately added it to their workflows. Docker released EE/UCP, and larger organizations jumped on board. It “is” good software for developers, SMB’s, and large organizations, and Docker, Inc isn’t slowing down development efforts.

DOCKER HAS FRIENDS

“I’m really excited to welcome Solomon and Docker to the Kubernetes community”. Brendan Burns (of Microsoft, Lead Engineer of Kubernetes) definitely made me do a double take when he said that on stage at DockerCon EU a few months ago. Many people I spoke to at the conference referenced that statement and saw this as a big blow to Docker. “Who’s joining who’s community? ” The thing is, the real purpose of Brendan’s talk was about the collaboration between companies, and the effort to make our lives as developers and administrators better. The whole “it takes a village to raise a child” saying. This village is composed of some of the brightest engineers from many of the world’s largest companies, and they’re all striving to make things better. Docker and Kubernetes worked together, and the Kubernetes integration into UCP made perfect sense.

Docker has business

They don’t have a lack of coherent leadership. They’ve received a ton of money, their marketing is great, and they’re acting like what they are; a rapidly growing company moving into the enterprise market. Were some of their keynotes awkward at DockerCon EU this year? Yes. Were there fantastic sessions from customers who shared real-life Docker success stories? Yes. Have they made some mistakes here and there? Yes. Have they moved past those and grown? Yes. If you’ve been around the block and watched small companies rapidly grow into behemoths, this is all typical. Growing isn’t easy. Their “Modernizing Enterprise Applications” mantra is perfect. There are countless technical budgets from Fortune 10,000 companies that Docker, Inc will capitalize on. The best part is that they’ll actually be making a positive difference. They are not snake-oil salesmen. These companies will probably see real ROI in their engagements.

Conclusion

Docker, Inc isn’t going to be acquired (yet) or close their doors. There is a lot going on at Docker, Inc right now but they aren’t signs of a company that is getting ready for a sale.

It’s a company that’s based on OSS with a lot of opportunity in the market. While one of the products at Iron is Docker-based, we use a wide variety of software from many companies with roots in OSS. We’re happy to pay for a higher level of support and features for OSS software backed by a business. For other projects, we often donate through Open Collective to help maintainers and small development teams. Docker’s donation of containerd was a great move and I think it is a project that fits perfectly into CNCF’s charter.

While Docker, Inc is moving upstream, they haven’t at all abandoned its real users; developers. We use Docker daily, contribute back when we can, and are optimistic about its trajectory as a business and a product. Docker, Inc has a lot of room to grow, and in 2018, it will.

Webhooks the Right Way™

If you’re a developer, dealing with webhooks is a part of your life. Nowadays almost every subscription service allows for these user-defined callbacks. For example, when a Lead is added to Salesforce, you may want to have a task that runs in the background to generate more information about the company they work for. Maybe you want to receive a request from Stripe when a customers payment fails so you can send them dunning emails? You get the drift.

The most common way to deal with webhooks is adding an endpoint to your application that handles the request and response. There are some benefits to this. No external dependencies by having all your code in one place for example. However, the cons usually outweigh the pros.

Common problems handling Webhooks

Application downtime

If your application is down, or in maintenance mode, you won’t be able to accept webhooks. Most external services will have retries built in but there are many that don’t. You’d need to be OK with missing data sent from these services.

IronMQ and IronWorker have great uptime — If your application is down, you could lose valuable data

Request queuing

What happens if you have a ton of webhooks from a bunch of different services all coming in at once? Your application/reverse proxy/etc will probably end up queuing up the requests along with other customer requests. If your application is customer facing, this could result in a degraded user experience or even full-blown timeouts.

Use IronMQ and IronWorker to prevent bad user experiences — Too many requests to your frontend could result in request queuing and negatively affect the end user experience

Thundering herds and Cache stampedes

Even if you’re able to process all of the webhooks coming in at once, your system is going to feel the effects one way or another. This could result in unwanted resource spikes (CPU/MEM/IO). Unless you’re set up to autoscale, bad things could happen to your infrastructure.

IronMQ and IronWorker can help prevent downtime caused by webhooks — Handling webhooks at scale via your application could result in infrastructure issues

At Iron, many of our customers get around these issues by using IronMQ and IronWorker in conjunction. Since IronMQ is HTTP based, highly available, and built to handle thousands of requests a second, it’s a perfect candidate for receiving webhooks. One of the great things about IronMQ is that it supports push queues. When a message is received, it can push its payload to an external HTTP endpoint, to another queue, or even to IronWorker.

IronWorker is a container based enterprise-ready background job processing system that can autoscale up and down transparently. We have customers processing 100’s of jobs concurrently one minute, while the next minute the number is up in the 100’s of thousands.

The beauty of the IronMQ and IronWorker integration is that IronMQ can push its payloads directly to IronWorker. Your work is then picked up and worked on immediately (or at a specific date and time if required). You can have a suite of different workers firing off for different types of webhooks and handling this process transparently. This is great for a number of reasons.

Handling Webhooks the Right Way

Happy application

Now your application is never involved in the process of handling webhooks. This all happens outside of your normal application lifecycle. Your application machines will never have to deal with the excessive load that could deal with infrastructure issues.

Happy users

All the work you need to do to process webhooks now happens in the background and on hardware that your users aren’t interacting with. This ensures that processing your webhooks won’t affect your user experience.

MQ and Worker to handle Webhooks — Using IronMQ and IronWorker to handle incoming Webhooks

This is a pattern that our customers are pretty happy with, and we’re constantly improving both IronMQ and IronWorker to handle their additional needs. For example, being able to programmatically validate external API signatures and the ability to respond with custom response codes are on our list. That said, similar to microservices, this level of service abstraction can also introduce its own complexities. For example, dependency and access management come to mind. We’ve had long conversations about these topics with our customers and in almost all cases, the pro’s out-weigh the cons. This approach has been a success and we’re seeing it implemented more and more.

If you have any questions or want to get started with the pattern above, contact us and we’ll be happy to help.

AWS EC2: P2 vs P3 instances

Amazon announced its latest generation of general-purpose GPU instances (P3) the other day, almost exactly a year after the launch of its first general-purpose GPU offering (P2). While the CPU’s on both suites of instance types are similar (both Intel Broadwell Xeon’s), the GPU’s definitely improved. Note that the P2/P3 instance types are well suited for tasks that have heavy computation needs (Machine Learning, Computational Finance, etc) and that AWS does provide G3 and EG1 instances specifically for graphic intensive applications.

The P2’s sport NVIDIA GK210 GPU’s whereas the P3’s run NVIDIA Tesla V100’s. Without digging too deep into the GPU internals, the Tesla V100’s are a huge leap forward in design and specifically target the needs of those running computationally intensive machine learning operations. Tesla V100’s tout “Tensor cores” which increase the performance of floating point computations and the larger of the P3 instance types support NVIDIA’s “NVLINK”, which allow multiple GPU’s to share intermediate results at high speeds.

While the P3’s are more expensive than the P2’s, they fill in the large gaps in on-demand pricing that existed when just the P2’s were available. That said, if you’re running a ton of heavy GPU computation through EC2, you might find the P3’s that offer NVLink a better fit, and picking them up off the spot market might make a lot of sense (they’re quite expensive).

When the P2’s first came out, Iraklis Mathiopoulos had a great blog post where he ran Hashcat (a popular password “recovery” tool) with GPU support against the largest instance size available… the p2.16xlarge. Just a few days ago he repeated the test against the largest of the P3 instances, the p3.16xlarge. If you’ve ever played around with Hashcat on your local machine, you’ll quickly realize how insanely fast one p3.16xlarge can compute. Iraklis’ test on the p2.16xlarge cranked out 12,275.6 MH/s (million hashes per second) while the p3.16xlarge at 59,971.8 MH/s against SHA-256. The author’s late 2013 MBP clocks in at a whopping 121.7 MH/s. The p3.16xlarge instance type is about to get some heavy usage by AWS customers who are concerned with results rather than price.

Of course, the test above is elementary and doesn’t exactly show the benefits on the NVIDIA Tesla V100 vs the NVIDIA GK210 in regard to ML/AI and neural network operations. We’re currently testing different GPU’s in our Worker product and hope to have some benchmarks we can soon share based on real customer workloads in the ML/AI space. The performance metrics and graphs that Worker produces will give a great visual on model building/teaching, and we’re excited to share our recent work with our current ML/AI customers.

While most of our ML/AI customers are on-premise, we’ll soon be looking to demonstrate Iron’s integration with P2 and P3 instances for GPU compute in public forums. In the meantime, if you are considering on-premise or hybrid solutions for ML/AI tasks, or looking to integrate the power of GPU compute, reach out and we’d be happy to help find an optimal strategy based on your needs.