How to Deliver Massive Volumes of Sports Scores and News to over 12 Million Mobile Users at Bleacher Report
Bleacher Report (B/R) is one of the leading sports news sources in the country. Second only to ESPN, Bleacher Report is a growing force for team-specific sports content, commentary, and real-time event coverage. B/R’s goal is to deliver a comprehensive experience for sports fans and foster engagement with favorite teams and topics across all major sports. The content that B/R delivers includes curated content originating from featured columnists and external sources, all pushed to fans in real-time. They provide timely and relevant news to sports fans keeping them informed and ahead of the game.
To make this experience possible, Bleacher Report delivers real-time sports scores, news, and team updates via the Team Stream mobile app. With over 12 million downloads of the Team Stream app, users get a personalized fan experience wherever they are. Sports fans get all the latest news and alerts, all in one place on their mobile devices. In order for B/R to deliver a high volume of alerts at peak events, they needed a solution that would easily scale, and remain available in the event of a major traffic spike.
Distributing Massive Volumes of Notifications at Scale
Previously, Bleacher Report relied on a variety of self-hosted DIY open-source components to stream the latest information about fans’ favorite sports teams and events. During major events, such as the NFL Draft, World Series, and NCAA March Madness, the pace of delivering up-to-the-minute notifications would often completely overwhelm the system and create huge backlogs in their workload processing. B/R learned the hard way that delivering massive volumes of notifications at scale could negatively impact response times and overall uptime of the system. This kind of spike in messaging traffic and the potential outcome of reduced performance and reliability was simply unacceptable for B/R.
“Notifications are critical to Bleacher Report. With over 12 million downloads, users look to Team Stream for breaking news alerts about their favorite teams and players,” said Eddie Dombrowski, Senior Software Engineer at Bleacher Report. “Speed of delivery is a constant focus for us. No longer worrying about infrastructure allows us to focus on delivering new features and optimizing existing ones.”
The Bleacher Report ops team has long held 99.9% uptime as a primary goal. In order to achieve this goal, changes to the infrastructure had to be made. At an early stage in their refactoring process, the ops team realized they were not only exceeding the limits of an infrastructure of 60 concurrent servers, they were also consumed by the complexity and labor-intensive monitoring they needed to do for their infrastructure. B/R had reached a point in their rapid growth where they needed to implement an asynchronous task processing system – one that was cloud-based, reliable, and could scale with the volume of notifications they were sending at any given time or event.
Deploying IronWorker for Flexibility and Scale
Bleacher Report initially started with a trial version of IronWorker to “kick-the-tires” of the Iron platform and test its capabilities as an event-driven workload processor.
“We already had a working infrastructure so we were looking for something that we could integrate relatively quickly,” recalls Dombrowski. “Because our system was already worker based, it was easy to experiment with Iron.io and we had a functioning prototype copy of our existing system in less than a day. A week later and we were able to migrate our code base to use Iron.io. A week after that we shut down our deprecated infrastructure and never looked back.”
B/R’s initial test was to send some notifications to users of their mobile app via scheduled IronWorker events. From this test, B/R was able to realize the power and flexibility IronWorker delivers.
“Our push notification system is now on-demand and highly concurrent,” said Dombrowski. “Before refactoring our existing app code (which took less than a week) and moving to the Iron.io infrastructure, we managed our own cluster of worker instances. When high throughput was needed we often found ourselves waiting for new instances to spin up.”
After B/R completed their migration to the Iron.io platform, they quickly saw strong measureable benefits in both alert times and cost savings.
“We noticed a dramatic performance improvement when we migrated to Iron.io. Also, we no longer had instances running unless there was work to do, so were able to cut our baseline expense significantly. Our only integration point was to RedisLabs, which supports Amazon Security Groups, so integration was painless,” according to Dombrowski.
The combination of a reliable, scalable cloud infrastructure plus the power of asynchronous task-processing services from IronWorker, enabled B/R to deliver high volume messages at scale, offload critical tasks, free up valuable resources to maintain a more flexible environment, and save time and money.
“We did our due diligence in preparing for high volume [of notifications], but it would sometimes stress the system in unexpected ways. Luckily, we were equipped to respond quickly. At a certain point, however, we decided that the time we were using to babysit our infrastructure would be better spent adding direct product value,” says Dombrowski.
Greater Reliability + Faster Response Times = Happy Sports Fans
The team at Bleacher Report has been so pleased with the reliability and performance of their new IronWorker-supported notification system that they have grown to a peak of 3000 concurrent workers on dedicated cloud-server resources. On normal occasions, any time B/R has to send an update or alert to their millions of users, they simply spool up 500 IronWorkers – as each worker can distribute thousands of push notifications to mobile app users on demand.
“We enjoyed all the obvious benefits of removing a whole stack of infrastructure, allowing our teams to focus on features while also saving money,” says Dombrowski. “One not so obvious, yet huge win for us is our ability to scale painlessly. Before Iron.io we were constantly triaging production issues. After Iron.io (a year later that same month), we delivered billions of push notifications with ease.”
Empowering developers to do what they do best – which is build great software that provides great service at scale. This is our mission at Iron.io, as we reduce the overhead of managing infrastructure and administration of developer tools, while delivering a turn-key solution for highly scalable event-driven applications. We can’t wait to see the increasing success and growth at Bleacher Report.
About Bleacher Report
Bleacher Report, a division of Turner Sports, is the leading digital destination for team-specific sports content and real-time event coverage, and is one of the fastest-growing digital properties in the U.S. Bleacher Report's editorial and video teams, led by a growing roster of lead writers and premier contributors, create hundreds of pieces of content per day to provide fans with the most comprehensive experience for their favorite teams and topics across all major sports.
Bleacher Report is also available through Team Stream™, the top-rated, industry-leading tablet and smartphone app, and via the Bleacher Report daily sport- and team-specific email newsletters. With over 12 million downloads to date, Bleacher Report's Team Stream tablet and smartphone app service provides an unmatched personalized fan experience on mobile devices.
Getting Started with Iron.io
To give Iron.io a try, sign up for a free account today.
As a reward for signing up, we’ll even extend to you a 30-day trial of advanced features so that you can see how moving to the cloud will change the way you think about application development.
