Search This Blog


Monday, July 6, 2015

How to Deliver Massive Volumes of Sports Scores and News to over 12 Million Mobile Users at Bleacher Report

Bleacher Report (B/R) is one of the leading sports news sources in the country. Second only to ESPN, Bleacher Report is a growing force for team-specific sports content, commentary, and real-time event coverage. B/R’s goal is to deliver a comprehensive experience for sports fans and foster engagement with favorite teams and topics across all major sports. The content that B/R delivers includes curated content originating from featured columnists and external sources, all pushed to fans in real-time. They provide timely and relevant news to sports fans keeping them informed and ahead of the game.

B/R Team Stream mobile app
To make this experience possible, Bleacher Report delivers real-time sports scores, news, and team updates via the Team Stream mobile app. With over 12 million downloads of the Team Stream app, users get a personalized fan experience wherever they are. Sports fans get all the latest news and alerts, all in one place on their mobile devices. In order for B/R to deliver a high volume of alerts at peak events, they needed a solution that would easily scale, and remain available in the event of a major traffic spike.

Distributing Massive Volumes of Notifications at Scale

Previously, Bleacher Report relied on a variety of self-hosted DIY open-source components to stream the latest information about fans’ favorite sports teams and events. During major events, such as the NFL Draft, World Series, and NCAA March Madness, the pace of delivering up-to-the-minute notifications would often completely overwhelm the system and create huge backlogs in their workload processing. B/R learned the hard way that delivering massive volumes of notifications at scale could negatively impact response times and overall uptime of the system. This kind of spike in messaging traffic and the potential outcome of reduced performance and reliability was simply unacceptable for B/R.

Eddie Dombrowski,
Sr. Software Engineer at B/R

“Notifications are critical to Bleacher Report. With over 12 million downloads, users look to Team Stream for breaking news alerts about their favorite teams and players,” said Eddie Dombrowski, Senior Software Engineer at Bleacher Report. “Speed of delivery is a constant focus for us. No longer worrying about infrastructure allows us to focus on delivering new features and optimizing existing ones.”

The Bleacher Report ops team has long held 99.9% uptime as a primary goal. In order to achieve this goal, changes to the infrastructure had to be made. At an early stage in their refactoring process, the ops team realized they were not only exceeding the limits of an infrastructure of 60 concurrent servers, they were also consumed by the complexity and labor-intensive monitoring they needed to do for their infrastructure. B/R had reached a point in their rapid growth where they needed to implement an asynchronous task processing system – one that was cloud-based, reliable, and could scale with the volume of notifications they were sending at any given time or event.

Deploying IronWorker for Flexibility and Scale

Bleacher Report initially started with a trial version of IronWorker to “kick-the-tires” of the Iron platform and test its capabilities as an event-driven workload processor.

“We already had a working infrastructure so we were looking for something that we could integrate relatively quickly,” recalls Dombrowski. “Because our system was already worker based, it was easy to experiment with and we had a functioning prototype copy of our existing system in less than a day. A week later and we were able to migrate our code base to use A week after that we shut down our deprecated infrastructure and never looked back.”

B/R’s initial test was to send some notifications to users of their mobile app via scheduled IronWorker events. From this test, B/R was able to realize the power and flexibility IronWorker delivers.

“Our push notification system is now on-demand and highly concurrent,” said Dombrowski. “Before refactoring our existing app code (which took less than a week) and moving to the infrastructure, we managed our own cluster of worker instances. When high throughput was needed we often found ourselves waiting for new instances to spin up.”

After B/R completed their migration to the platform, they quickly saw strong measureable benefits in both alert times and cost savings.

We noticed a dramatic performance improvement when we migrated to Also, we no longer had instances running unless there was work to do, so were able to cut our baseline expense significantly. Our only integration point was to RedisLabs, which supports Amazon Security Groups, so integration was painless,” according to Dombrowski.

The combination of a reliable, scalable cloud infrastructure plus the power of asynchronous task-processing services from IronWorker, enabled B/R to deliver high volume messages at scale, offload critical tasks, free up valuable resources to maintain a more flexible environment, and save time and money.

“We did our due diligence in preparing for high volume [of notifications], but it would sometimes stress the system in unexpected ways. Luckily, we were equipped to respond quickly. At a certain point, however, we decided that the time we were using to babysit our infrastructure would be better spent adding direct product value,” says Dombrowski.

Greater Reliability + Faster Response Times = Happy Sports Fans

The team at Bleacher Report has been so pleased with the reliability and performance of their new IronWorker-supported notification system that they have grown to a peak of 3000 concurrent workers on dedicated cloud-server resources. On normal occasions, any time B/R has to send an update or alert to their millions of users, they simply spool up 500 IronWorkers ­– as each worker can distribute thousands of push notifications to mobile app users on demand.

“We enjoyed all the obvious benefits of removing a whole stack of infrastructure, allowing our teams to focus on features while also saving money,” says Dombrowski. “One not so obvious, yet huge win for us is our ability to scale painlessly. Before we were constantly triaging production issues. After (a year later that same month), we delivered billions of push notifications with ease.”

Empowering developers to do what they do best – which is build great software that provides great service at scale. This is our mission at, as we reduce the overhead of managing infrastructure and administration of developer tools, while delivering a turn-key solution for highly scalable event-driven applications. We can’t wait to see the increasing success and growth at Bleacher Report.

About Bleacher Report

Bleacher Report, a division of Turner Sports, is the leading digital destination for team-specific sports content and real-time event coverage, and is one of the fastest-growing digital properties in the U.S. Bleacher Report's editorial and video teams, led by a growing roster of lead writers and premier contributors, create hundreds of pieces of content per day to provide fans with the most comprehensive experience for their favorite teams and topics across all major sports.

Bleacher Report is also available through Team Stream™, the top-rated, industry-leading tablet and smartphone app, and via the Bleacher Report daily sport- and team-specific email newsletters. With over 12 million downloads to date, Bleacher Report's Team Stream tablet and smartphone app service provides an unmatched personalized fan experience on mobile devices.

Getting Started with

To give a try, sign up for a free account today.

As a reward for signing up, we’ll even extend to you a 30-day trial of advanced features so that you can see how moving to the cloud will change the way you think about application development.

Tuesday, June 16, 2015

Project Thor Will Deliver First True Hybrid IronWorker Solution

Today, announced Project Thor, which is developing the world's first hybrid job processing system. This is unlike anything we've done to date. Unlike previous IronWorker technology, Project Thor is architected to deliver the power of to any server in the world in minutes.

With Project Thor, the IronWorker Docker container is deployed on a company's own servers, which then communicates with the API. This brings the power of's event-driven computing service to the Enterprise in just a few easy steps. Project Thor seamlessly integrates with existing solutions for container deployment, such as OpenShift by Red Hat, Cloud Foundry, OpenStack and Kubernetes.

Because the workloads can run behind the firewall on the Enterprise's own machines, existing security solutions can be leveraged. Meanwhile, all orchestration is provided by the IronWorker API in the cloud. For the first time, companies will have the option to use their own hardware as well as cloud options. Enterprises can now reap all the benefits of cloud, while adhering to their compliance and security protocols.

Project Thor is an architectural shift for Enterprise businesses will now be to leverage the deployment model, while maintaining the same level of control as if they built everything internally. Because this shift to more powerful APIs enables more sophisticated workloads to be utilized, it is important for both development empowerment and Enterprise innovation.

To signup for project updates or to request to participate in the Project Thor register at

Tuesday, June 9, 2015

No lock in, thanks to Docker

Flexibility and the freedom to choose are among the core tenets at For those who have been following posts for some time, we're sure you've read a number of the Docker-related developments we released over the past year, including the previous blog post that IronWorker supports custom Docker images. This is awesome not only for many reasons from a workflow perspective, it also means there’s no lock in.

You can run your code anywhere, on any infrastructure, or on any cloud platform. Additionally, IronWorker provides several features and functions that you won’t find anywhere else, including multi-language support, multiple security and authentication features, enterprise-grade scalability and availability, and an advanced management dashboard (HUD). And if you don’t like it for whatever reason, you can take it and run it somewhere else. Now that's freedom.

We believe in user control and choice when it comes to your infrastructure, which is why we support running our managed services on all the major public cloud platforms (including AWS, Azure, Rackspace, Heroku, Pivotal, and Red Hat OpenShift); you can even run IronWorker on your own bare metal hardware (on-premise) too.

WIth’s flexible architecture, it’s your choice how you manage and deploy your code. You're never locked in to writing code for a proprietary service because you can take your Docker image and run it wherever you want.

Getting Started with

To give a try, sign up for a 30-day trial today.

Friday, May 29, 2015

Treasure Data and IronWorker (repost)

Our friends at Treasure Data wrote a blog post about data collection in Ruby and how to run multiple data collection tasks in parallel (or scheduled) using IronWorker. The example from Treasure Data demonstrates what it takes to build a simple logging application in Ruby with IronWorker to manage and log the output to Treasure Data, which can then perform queries.

As noted in the blog, this example is not a complete solution but an illustration to show users what's possible when combining and Treasure Data. Big thanks to John Hammink and the Treasure Data team for their work to educate the community.

Here's an excerpt from the original post:

Managing multiple tasks with IronWorker with output to Treasure Data 

Coding up Your Ruby Task

Create a directory called /SendLog and open a text editor. You can get your API KEY fromhere.  Type in the following, and save it as SendLog.rb
This script, when run, will do exactly four things:
  1. Create a database on Treasure Data called iron_1;
  2. Create a table called login and add a single record to it. Note that there are always two values added to any user-defined values by default: v, which is a map of key value pairs containing the timestamp and any user-defined values; and time, which is the timestamp. The user-defined value in this case is uid.
  3. Create a table called follow and add a single record to it, this time with three user-defined values: uid, from and to.
  4. Finally, create a table called pay with one record, and add five user-defined values to it:  uid, item_name, category, price, and count.

Running your Ruby Task Locally

If you haven’t installed ruby gems, you should do so.   Also, you will want to install bundler.
Open up a text editor in your current /SendLog directory and create a file called Gemfile:
Next, run the following command (also in the same directory):
$ bundle install
Now, run your Ruby script once:
$ ruby SendLog.rb
Finally, log into Treasure Data console and, after connecting to iron_1 database,  issue and run each of the following queries:
  • SELECT uid FROM login
  • SELECT * FROM login ORDER BY time
  • SELECT * FROM follow
  • SELECT time from (SELECT * from follow) as sub ORDER BY time
  • SELECT uid, category, price, count, item_name FROM pay
  • SELECT count(1) from pay
For each query you ran, did you get the expected result? What did you see? Did you encounter any errors? (Note that Treasure Data console limits output to 100 records.)
You may run into errors. Sometimes, an easy way to discover any errors or bugs you may have made in your own code and/or console commands is to reread your code and examples backwards.
Since you have run the script only once, you should have only one record in each table.

Configuring your Worker

Now you’ve run your task once, but you want to see how it goes when running it in multiple instances in the cloud.  Enter!
You will need to take a few preliminary steps to get an instance up and running.
  1. Go to Register for an account and log in.
  2. Navigate to and click “New Project”. Name it “SendLog”.
  3. Click the Worker button next to SendLog project.
  4. You should be looking at the “Get Started” tab.  Do step 1  (downloadIron.json to your /SendLog directory and run $ sudo gem install iron_worker_ng).
  5. In the same /SendLog directory where your Gemfile, Iron.json, andSendLog.rb now exist, create a fourth file:  SendLog.worker.  Open up a text editor and enter the following:

Uploading, Queuing and Running a Single Instance of Your Worker

This next step is what is required to run your Ruby script – your packaged worker – in the cloud.
    1. Run the following from the command line:
      $ iron_worker upload SendLog;  iron_worker queue SendLog – -priority 2 – -wait
(note: omit the space between dashes)
  1. Once the process is complete, you should see the following output from the console:

——> Creating client
Project ‘SendLog’ with id=’554a8f2475e6cc00060000b6′
——> Creating code package
Found workerfile with path=’SendLog.worker’
Adding ruby gem dependency with name=’td’ and version=’~> 0.11.1′
Detected exec with path=’SendLog.rb’ and args='{}’
Code package name is ‘SendLog’
——> Uploading and building code package ‘SendLog’
Remote building worker
Code package uploaded with id=’554a910d0f9128000700a686′ and revision=’13’
Check ‘′ for more info
——> Creating client
Project ‘SendLog’ with id=’554a8f2475e6cc00060000b6′
——> Queueing task
Code package ‘SendLog’ queued with id=’554a9d6d6b3a88000b00f4c1′
Check ‘′ for more info
——> Getting log for task with id=’554a9d6d6b3a88000b00f4c1′
I, [2015-05-06T23:02:11.705509 #18]  INFO — : Creating table iron_1.login on Treasure Data
I, [2015-05-06T23:02:12.364480 #18]  INFO — : Creating table iron_1.follow on Treasure Data
I, [2015-05-06T23:02:12.714396 #18]  INFO — : Creating table on Treasure Data
Did it run correctly?  Did you get any errors?  If you check your tables in Treasure Data console, you should see that each one now contains two records.

Running Multiple Instances of Your Worker

Now that your job is up on, it’s easy to schedule multiple instances of the worker.
    1. In the “Scheduled tasks” tab, click the calendar icon to the top right of the task list.
      Note:  Your task list may be empty.
    2. In the “Add New Scheduled Task” dialog that appears, select your “SendLog” job in the drop down, along with Stop parameters, Run parameters, Priority (p2 jobs are generally run immediately), and the cluster.  Mem3 will be a more dedicated cluster.  (Don’t worry about payload at this point.)  Once you’re ready, click “Schedule Task”.

  1. To run many tasks in parallel, schedule multiple tasks with overlapping run times.
While this is going on — and once it’s complete — try running some of the same queries from the section “Running your Ruby Task Locally.”   You should now see the databases populated with many records, with more adding as additional tasks get run.
This is only a taste of what’s possible to do with and, by no means a complete example. There are many instances where logging messages from an worker could be useful: For instance, perhaps you want to send a diagnostic message if something goes wrong on a job (and some error code is executed), or you want to log the timestamp when a job is complete. 
We look forward to bringing you future examples and best practices on how to leverage the platform to address your asynchronous tasks and workflows. Stay tuned...

About Treasure Data

Treasure Data was founded in 2011 with the mission of building the first prebuilt cloud service for massive-scale collection, storage and analysis of big data sources, including web, mobile, IoT/sensor data, logs and events. The cloud service is monitored and supported by Treasure Data staff. This gives its customers an end-to-end big data processing capability at a fraction of the cost of building and maintaining it themselves.