The Weather Company Case Study

The weather impacts everything around us and drives decisions that range from government policy on climate change to whether it’s a good day for the family to go to the beach. The Weather Company focuses on connecting people, businesses and governments with the world’s best weather forecasts. It is the parent company for The Weather Channel and other radio and television broadcast channels, as well as digital properties such as weather.com. The company creates apps for broadband and mobile platforms, including iOS and Android mobile devices. Hundreds of thousands of customers in the aviation, broadcasting, energy, insurance and other industries, as well as governments worldwide, use its weather forecasting tools and systems.

The Weather Company also uses data to provide analytic services that help companies understand the relationship between the weather and consumer behavior. Retailers can use the information to target ads and consumer messaging using weather-driven decisions. Based in Atlanta, Georgia, The Weather Company is owned by a consortium made up of NBC Universal and the private equity firms The Blackstone Group and Bain Capital.

Over the years, the company made several acquisitions and grew organically to transform from a cable television company to a global provider of weather data services. As a result, the company’s infrastructure became a collection of 13 data centers running legacy systems that included forecasting models built and run on Fortran. That environment wasn’t cost-effective or scalable for a company that provides weather forecasts for 2.4 billion locations worldwide.

The Weather Company’s solution was to re-design its big data platform, forecasting systems, and applications to run natively in a cloud environment. This strategy would allow The Weather Company to scale while maintaining control over the environment and costs. It would also allow the company to create developer-friendly APIs that product teams across the organization could use to innovate and create new products without worrying about infrastructure.

AWS is a key component of The Weather Company’s cloud strategy. “We need to be within 100 milliseconds of everyone on the planet — and we leverage the global reach and availability of AWS to deliver on that strategy,” says Bryson Koehler, Executive Vice President and CIO.

Generating four terabytes of data every hour

Weather is a volatile dataset that changes every 15 minutes or less. In order to scale quickly on a global basis, the company pushes four gigabytes per second of forecasting data through AWS services. For forecasting, the company designed a new weather forecast and data services platform powered by NoSQL-distributed databases like Basho's Riak, Redis and Cassandra. These NoSQL databases are leveraged to handle massive data movement, replication, and data distribution in a cloud-based, distributed environment. The Weather Company reaches audiences worldwide by deploying the platform across multiple Availability Zones in the US East (Northern Virginia), US West (Northern California), EU (Ireland) and Asia Pacific (Singapore) Regions. Riak stores historical data from satellites, radars, forecast models, users and weather stations.

The platform consists of the following three layers:

• Dynamic Integrated foreCast (DICast), a statistical weather forecasting engine producing “first-guess” forecast grids
  
• A forecast-on-demand layer that applies human forecaster influence to the “first-guess” forecasts in real time and publishes a consumption-ready “finished” forecast for the user’s precise location
  
• The SUN (Storage Utility Network) platform, a public-facing API caching layer
  

The Weather Company covers the entire globe with a grid of 37 million four-kilometer squares, and updates this grid of first-guess weather forecasts as new numeric model guidance is received, often several times an hour. This allows the company to report current weather conditions accurately, and predict weather for each grid point days or weeks in advance.

The platform ingests information from more than 100 different sources and generates close to one-half terabyte (TB) of data each time it updates. The information is mapped and processed into forecast points that can be retrieved in real time, based on queries coming into the system. All data is stored in Amazon Simple Storage Service (Amazon S3), leveraging the efficiency of cloud storage as opposed to an on-premises storage solution and eliminating the hassle of managing a storage platform.

Amazon Redshift is The Weather Company’s data analytics platform, providing analytical data for services such as the WeatherFX ad engine, which looks at how the weather affects retail sales.

Developers worldwide can take advantage of an API-driven platform

The weather forecast and data platform uses AWS APIs to automate application development and the launch of new environments. “Building and designing for the cloud is a different philosophy and mind set, and certainly a different technical approach,” Koehler says. “We deploy about 90 percent of our applications and systems on AWS — and we have the flexibility to easily port applications and systems as necessary for the business.”

The Weather Channel uses AWS APIs to give its developers flexibility. Any developer can use what the company refers to as data cartridges to develop services for the data platform, other cloud environments, or on-premises hardware. “This gives us extremely high volume and reach both within the company and by external developers,” says Koehler. “More than 150,000 external developers worldwide are registered to use the platform. Anyone from a hobbyist to an external developer building a large-scale system can use our APIs to build products.”  

Since joining The Weather Company, Koehler has transformed the IT organization to support the company’s business growth strategy. “AWS offers a responsive environment with massive data movement, replication and synchronization to help meet the demands of our digital transformation at scale and on a global distribution level,” he says. “The Weather Company’s business strategy is focused on four S’s: science, safety, storytelling and services. Using AWS allows us to focus on the most important aspects in our business in a cost-effective manner.”

By launching an API-driven platform on AWS, The Weather Company has been able to bring products and services to market faster. “Using AWS means that we’re not hamstrung by infrastructure,” says Koehler. “When we wanted to launch a new forecasting system, the requirements for doing that internally would have taken as much time, money and effort as it would to write the code. We started the project in April 2013. Now our weather API is one of the world’s most-used public APIs. The platform is robust enough to handle between 10 and 15 billion transactions each day at 100,000 to 150,000 per second, depending on the weather.”

The Weather Company’s cloud strategy has reduced its on-premises environments from 13 to six data centers. “The less my engineers have to worry about power, cooling, racking, stacking, and other operations tasks, the more they can focus on the business,” says Koehler. “They can focus on being application engineers, building resiliency into the apps, and improving network and application efficiencies. Those sorts of savings are hard to measure, but you can’t go from 13 data centers down to six without realizing significant savings in time and management.”

Running on AWS has also been cost-effective for The Weather Company. “Anyone who thinks running on the cloud is more expensive hasn’t really totaled up all the true costs,” says Koehler. “A forecasting system like ours is a steady-state load; it runs all the time. That’s not a traditional cloud computing value proposition, but by running on AWS instead of on-premises, we’re saving almost $1 million per year on just one application. If I can get a win on a forecasting application that runs consistently 24/7, I believe that anyone can get a win on applications, especially those with spikes in demand.”