Search This Blog


Monday, March 26, 2012

How to Reduce a 9 Hour Job into a 9 Minute Job

A common problem developers have is how to run through a large number of tasks in a short amount of time. For instance, sending out a nightly notification to all your users, calculating your user's bills every month or updating their Facebook graph data. These types of things are very common and almost every application needs to do it and it becomes more and more of a problem as your user base grows.

I was just helping a customer today that was using Heroku workers at full worker capacity (24) and it was taking ~9 hours per day to go through his user database of 200,000 users and send out notifications. A quick calculation explains why:

((200,000 users * 3.5 seconds per task) / 3600 seconds/hour) / 24 workers processes = 8.1 hours

We took it down to less than 9 minutes. Here's how:

The easy (naive?) way would just be to queue up 200,000 tasks on IronWorker. This would have worked just fine, but it would have taken a while just to queue up the tasks and the setup/teardown time of a task would be wasteful when the task only takes 3.5 seconds to run. Instead, we had each task process 100 users which is now only 2000 tasks. Each task should take approximately 3.5 * 100 = 350 seconds = ~6 minutes to run and we can run them all at pretty much the same time, but I'll be conservative and adjust a bit for IronWorker capacity.

((200,000 users * 3.5 seconds) / 3600 seconds/hour) / 2000 worker processes * 1.5 capacity adjustment = 0.146 hours = 8.75 minutes

Batching up the tasks into batches of 100 was easy, here's sample code:

IronWorker gives you super simple access to huge compute power and you don't even need to think about a server. This customer will never have to worry about scaling this part of his application again.

Thursday, March 22, 2012

IronMQ is now on AppHarbor

Today, we’re pleased to announce that IronMQ is now included as a supported add-on for AppHarbor, the leading cloud platform for .NET developers. It’s a great combination and we’re thrilled to be partners.
What’s great about the pairing is that the two companies share the same vision of cloud computing. AppHarbor lets developers spend their time coming up with ideas and developing applications, not patching servers, worrying about deployment, messing with configuration files, or scaling. provides hosted messaging, background processing, and scheduling -- services that a developer can get going in minutes and provides them with the ability to build apps that do massive asynchronous things. All without managing servers or dealing with infrastructure.

If you’ve followed our progress over the past few months, you’ve seen us moving rapidly to increase support for multiple languages and tie in with many platforms and clouds. The addition of AppHarbor to this list is a big milestone because it connects us with a huge body of .NET developers. One of our goals is to where developers are and is a big advance on that goal.

And what’s neat about this market is that while .NET developers may have been asking themselves the “Why cloud?” question a bit longer than Ruby or Python developers, they’re getting it and they’re getting it fast. They bring with them a deep knowledge of building complex apps with lots of moving pieces.

Which is great for, because IronMQ is designed to help developers pass data and events between different parts of their application and to external apps and systems. It's a durable tools made for professional developers building applications in the cloud. Which should explain while we’re thrilled to connect to AppHarbor and the .NET community.

To learn more about IronMQ and our other product, IronWorker, please visit our product page or our DevCenter.

To learn more about the and AppHarbor partnership, please visit the IronMQ page in the AppHarbor Add-ons section.

Tuesday, March 13, 2012

@BetaVanguard: A Few Questions and Answers

SF Beta had some questions for us the other day. Here are a few of the answers. Full set can be seen here.

Monday, March 12, 2012 Speaking at Data 2.0 Summit

One of the big uses of services is in working with cloud data at big scale. Some customers are using IronWorker to crawl the web and process the results. Others are taking data and transforming it using a host of workers. Still others are using IronMQ to connect cloud apps to legacy apps.

And so it makes sense to connect with data providers, information services, API companies, and others in the Big Data space at the Data 2.0 Summit in SF on April 3rd.

Chad Arimura will be on a panel talking about the Real-Time Data Stack. Alongside app servers, databases, and the data sources themselves, application infrastructure services like IronWorker and IronMQ are natural fits. There's lots of processing that needs to be done in the background and databases, unfortunately, don't work so well in orchestrating process flow, which is where message queues come in.
The topic of the realtime stack is right up our alley. Ken Fromm wrote a series on the realtime web three years ago that turned out to be one of ReadWriteWeb's top stories of the year. You can read it herehere, and here. Here's an excerpt.

Because of demand within the [Twitter] eco-system, quite a bit of effort is being made on storing, slicing, dicing, and disseminating information as quickly as possible. The fundamental implication of this activity (without any explicit markers being laid down) is that the velocity of information within the Web data system has just increased by an order of magnitude. 
The pipes are moving data at the same rate: the speed of your data connection has not changed (although it is getting faster because of an independent effort by cable companies, telcos, and the like). What has changed is the flow of data from machine to machine on the Web and the processing that happens as information makes its way to users. Companies are making use of data that takes seconds to be published to the Web, as opposed to hours or minutes. Years ago, pages might have been crawled by search engines daily. With the advent of RSS, new posts would flow through the system within hours. With Twitter, the flow is propagated from company to company to user in real time. 
This facet of the real-time stream is having a profound impact on the infrastructure of the Web. New storage and retrieval methods are being developed to overcome the time lags of writing not just to disks but to traditional databases. Adaptations to traditional structured query languages are being made to index items directly from the stream. Search engines and search capabilities are being modified to make use of real-time inputs to influence the search results. This isn't just a Twitter effect. This is an effect across all uses of the Web, because the expectation of access to real-time information is now permeating all websites and the infrastructure of the Web itself.

Message queues and massively scalable background processing are big parts of this infrastructure and playing bigger roles in an increasingly faster and more connected web.

Saturday, March 10, 2012 Demoing at SF Beta (March 27th) will be one of the demo tables at SF Beta on March 27th. SF Beta is a great event. It's a mixer more than anything else but we're hoping it's heavy on the hackers and developers, designers and students. A bunch of other great companies will be a part.

Demo Theme: Platforms and APIs
• SendGrid
• PubNub
• Add to Trip
• TellApart
• LaunchRock

It's at 111 Minna St in SF. If around, stop on by and grab a drink with us.