Founded in 1986, Weathernews is one of the world's largest private meteorological companies with sales and operations bases in major cities around the world, serving approximately 50 countries worldwide. The company's services ranges from shipping and aviation weather to rail, road, and retailer weather. It is also known for its "Weathernews" mobile application.

For the most part, the meteorological industry obtains its base information from government agencies. Since it was established, Weathernews has built its own infrastructure for observation, communication networks, image processing, and distribution. In 2005, the company developed the Original Weather Numerator (OWN) as its own weather forecast model. To process the data it built an on-premises high performance computing (HPC) system, which had been continuously enhanced by increasing the number of servers. This enabled them to forecast at one-hour intervals up to three days ahead. In addition, applying artificial intelligence (AI) technology to radar and weather reports received from application users enabled forecasting at 10-minute intervals up to three hours ahead, and at 1-hour intervals up to 15 hours ahead.

"However, it was pointed out that at one-hour intervals, ‘rainclouds suddenly became blurry and difficult to make out’. In recent years, there has been an increase in rapid weather changes such as thunderstorm or localized downpour. This has corresponded to an increased need for high-resolution forecasting over longer periods of time. Also, being able to make detailed weather forecasts further into the future has long been a dream for us meteorologists" said Tomohiro Ishibashi, Managing Director and Executive Officer at Weathernews.

The company then started looking for a service capable of forecasting at 10-minute intervals up to 15 hours ahead. Its aim was to improve the previous forecasting interval from every six hours (four times a day) to every three hours (eight times a day).

The main challenge was to procure the large amounts of computational resources needed for forecasting. Forecast Center Development Team leader, Kohei Sakamoto reflected, "Adding on-premises resources, as we did previously, would require a huge investment," and as the numbers of servers increase, so do concerns of failure and higher operational burdens. In addition, June to October experience more frequent typhoons and thunderstorms or localized downpours in Japan, and thus require more computational resources than during other seasons. It was difficult for Weathernews to respond to such load fluctuations in a flexible manner using their on-premises environment.

In 2018, Weathernews began exploring a next-generation OWN cloud implementation utilizing AWS ParallelCluster. After thorough validation, the decision to adopt AWS ParallelCluster was made in April 2020. "By performing actual model calculations, we validated performance using the relationship between the number of AWS ParallelCluster instances, the effectiveness of Elastic Fabric Adapter (EFA), and changes in processing speed depending on the instance type," said Kazunari Takahashi from the Forecast Center Development Team. Initially, there were some concerns about scalability when using MPI (Message Passing Interface) in cloud HPC, but actual measurements showed speed improvements in the range of 5,000 vCPU. Utilizing EFA, a low-latency network adapter for workloads that require high bandwidth inter-node communications, like MPI based workloads, increased calculation speeds 25%.

To ensure reliability, the company is building a main system and a sub-system in two separate AWS Regions. Processing will normally be carried out on the main system in the Northern Virginia Region, but in the event of failure, it will revert to the sub-system in the Tokyo Region for reprocessing. In addition, the main system environment uses Amazon EC2 Spot Instances. Amazon EC2 Spot Instances are spare compute capacity available at up to a 90% discount over On-Demand Instance pricing. The company worked with AWS Solution Architects to configure the system to meet availability, cost, and performance requirements of the next generation of systems.

Kohei Sakamoto said, "Using an AWS multi-Region architecture has significantly reduced our initial investment, as well as our operational costs. I estimate we will lower our total cost of ownership by one-third over five years."

"At the same time, we have also managed to reduce the operational workload," said Kazunari Takahashi. Servers in the on-premises environment went down every two to three months, and remedying this required on-site visits by staff. This isn't required with AWS ParallelCluster. If processing fails on the main systems, the system notifies the administrator and automatically migrates processing to the sub-system.

Adding computing resources with AWS ParallelCluster let Weathernews achieve their initial target of forecasting at 10-minute intervals up to 15 hours ahead. As of July 2020, this has been provided as a paid service through the "Weathernews" smartphone app. While this achieves forecasting over shorter periods of time, spatial resolution has also been improved from a 1 to 5-km area down to a 250-m area. These higher resolutions leverage newly developed AI technologies, which were trained on AWS.

Being able to forecast at 10-minute intervals up to 15 hours ahead has resonated with users given that it is more intuitive to understand, and there has also been an increase in the number of paid users. "For example, you can clearly see the rotation of a typhoon. You can see how the weather will change, built upon past weather patterns. This information has not been seen by anyone, and it is having quite a considerable impact." (Tomohiro Ishibashi)

This also increases the accuracy of forecasting. By 2020, forecast accuracy of 91% (daily rainfall capture rate) had already been achieved, but the company plans to reach 94% accuracy during 2021. The ability to dynamically procure the required compute resources has also sped up improvements to the predictive model.

With AWS, Weathernews was able to achieve longer-term, more accurate weather forecasts. However, Tomohiro Ishibashi says that this is not the only benefit of using AWS.
"From a business perspective, we expect that having our meteorologists and engineers play with their own ideas will enable a type of forecasting that wouldn’t have been possible otherwise. This is because forward-thinking ideas that will shape the future have their genesis in play. Naturally, if this requires a large amount of investment, then taking that first step will be difficult. However, by using AWS the risk is lower, and there are minimal sunk costs if things do not go well."

AWS also enables easier external collaboration and incorporation of new technologies. What kinds of innovations will emerge from these new challenges? The future of Weathernews deserves close attention.

Return to the list of customer case studies in Japan »