Solcast: Solar irradiance forecasting for the solar powered future

Dr. Nick Engerer, CTO of Solcast, standing next to solar panels

Solcast is a global solar forecasting and solar irradiance data modelling company, delivering its data products via an Amazon Web Services (AWS) hosted API and online web platform: the Solcast API Toolkit. Solcast’s mission is to deploy the data and tools needed to build a solar powered future, by making it easy for those in the solar industry to access the information they need to get the job done.

As part of the Amazon Sustainability Data Initiative, we invited Dr. Nick Engerer, Chief Technology Officer at Solcast, to share how their organization is using open data and AWS to provide global coverage of operational solar forecasting data via application programming interface (API).

A guest post by Dr. Nick Engerer, Chief Technology Officer at Solcast

Capturing and processing large volumes of satellite data

Solcast produces real-time, historical and forecast estimates of the available solar radiation resources around the globe. Altogether, this requires processing imagery from five geostationary weather satellites, which take snapshots of the Earth from space every 5-15 minutes. These images can be as fine as 1 kilometer in spatial resolution and are re-captured and transmitted as frequently as every five minutes – a large volume of data. A single third-generation weather satellite in orbit over the United States such as the GOES-16 generates nearly two terabytes (TB) of data each month.

Each of these images needs to be processed through geo-coding, quality control and solar irradiance calculation algorithms in a matter of tens of seconds before being ingested into our forecasting algorithms. Following this step, Solcast computes nearly 600 million forecasts every hour via a probabilistic ensemble, to capture a range of outcomes for the near-term availability of cloud cover (up to four hours ahead). The large volumes of input data require a lot of processing power and storage. In addition, all the generated products need to be made available to the API service for the end users to access in real-time (meaning it needs to be fast, reliable and robust).

AWS and operational solar forecasting: fit for purpose

AWS has been instrumental in enabling Solcast to make the above system an operational reality. By leveraging AWS, we have deployed an operational system that provides global solar radiation and solar photovoltaic (PV) forecasts that are updated every 5-15 minutes.

Our systems (including our API) use several Amazon services. Amazon Relational Database Service (Amazon RDS) gave us a turn-key solution to high availability data storage with Postgres (and PostGIS extension) – if we were to host this ourselves it would have taken a lot longer and cost a lot more as AWS was able to scale with us as we learnt about our customer needs. Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Elastic Container Service (Amazon ECS) enabled us to create a zero-down-time deployment strategy for our API while also taking advantage of spot instances for cost reductions. Amazon ECS has proven to be a great platform for Docker hosting, minimizing complexity.

Overall, what we have learned from our journey with AWS is the remarkable progress made in cloud computing technologies and services over the last decade. It is now possible to process “big data” affordably in the cloud, in real-time, while running memory intensive algorithms that demand resources in a flexible fashion. The scale of the computing challenge we have been able to tackle through AWS-based services is really quite amazing, and by going that route, we can get the work done at much lower cost and with greater robustness than we would be able to do on our own.

Geostationary satellite BIG data

One particular problem we encountered was the storage of large amounts of historical satellite imagery data. That’s 2TB of data each month, piling up continuously. Amazon Simple Storage Service (Amazon S3) gives us the flexibility to store terabytes of satellite imagery next to the compute, using cost-effective storage. Looking to the future, these challenges may become even more manageable because of the growing collaboration between key data providers and AWS. For example, as NOAA is staging the GOES data on AWS through the AWS Public Dataset Program, many users may no longer have to cover the cost of acquisition and storage for those datasets accessible in Amazon S3 buckets. Utilizing these cost-effective storage options on AWS services has given us the confidence to scale up our operations to include long-term historical datasets for solar resource assessment, including historical time-series and typical meteorological year (TMY) data, now available via the Solcast API Toolkit.

Serving customers on six continents

Today, our global solar forecasting system hosted on AWS is delivering live and forecast solar radiation and power data to customers in six continents. Solcast has become an operational services provider for the grid operator of Vietnam (EVN), the Australian Energy Market Operator and a wide array of solar farm operators around the world. We now have over 3,000 users of our API services, who are using the Solcast API Toolkit to set up their own solar forecasting sites, big and small, along with researchers who are given free access to our technologies on sign-up.

Looking forward, we hope to enable the next big idea in solar. We need to connect with the individuals and organizations planning, deploying, and operating solar installations everywhere in order to make a solar-powered future a reality.