AWS for Industries

Unlocking sustainable power using Stem’s AI-driven clean energy platform on AWS

In response to the mounting impacts of climate-driven extreme weather, and continued global and regional mandates, industries and businesses worldwide are increasingly investing in sustainability, environmental, social, and governance (ESG), as well as Net-Zero strategies. This includes setting goals to reduce businesses’ carbon emissions by 2030 and accelerating investments in the adoption of clean energy technologies. Unlike the greenwashing strategies that worked in the past, today’s businesses (and their investors) specifically require data-backed solutions and proof of progress toward these goals. Furthermore, the grid must undergo a profound modernization to bring renewable energy sources online and unlock their full potential while proving environmental impact. For example, renewable energy generated from solar or wind must be available even if the sun isn’t shining and the wind isn’t blowing. Advanced storage technologies coupled with AI-driven software to properly manage renewable assets will be crucial to transforming the grid from an aging supplier of commodity electricity to an intelligent “system of systems” that produces optimized energy and environmental outcomes.

Stem is a global leader in AI-driven clean energy solutions and services with over a decade of energy experience helping to simplify the deployment and maximize the value of storage, solar, and EV charging assets for owners and operators. Stem’s unified clean energy management platform, Athena®, uses historical and real-time data, advanced data analytics, machine learning (ML), and data-driven applications to offer more accurate modeling, as well as optimally size projects, to help reduce risk for its customers. From the beginning, Athena was built as an open, extensible platform, and it has been getting smarter as it processes more data and continuously learns from the over 32 gigawatts (GW) of solar, energy storage, and EV charging assets under Athena management across 50 countries. This real-world data advantage and making it actionable are central to the value that Stem’s Athena platform delivers to businesses.

Use-case

Today, Stem’s Athena platform processes terabytes of data each day. Its ML algorithms generate multiple forecasts – about weather, prices, solar generation, energy demand, and other factors – and analyze how energy assets can capture value at different times. Then, Athena formulates a strategy for maximizing that value and implements it in various ways: by bidding assets into wholesale markets, charging and discharging batteries, injecting solar energy into the grid, or storing it for later, thereby allowing EVs to charge now or overnight. Moreover, because Athena constantly monitors system performance, it adjusts to changing circumstances in real-time.

Stem energy experts design Athena’s algorithms to learn and adapt to each customer’s priorities and goals, whether it’s an independent power producer’s bidding strategy in a wholesale market, a water district’s need for backup power, or a facility’s goal to reduce greenhouse gas emissions. Athena automates the acquisition and curation of data, forecasting, optimizing, and performing real-time autonomous controls to maximize asset performance and ROI, including interfacing with wholesale markets and utilities, integrating and optimizing various assets and value streams, and complying with incentive policies and asset warranty requirements.

Data is at the heart of Stem’s value to its clients. Therefore, Stem must make sure of the quality of its data, that the data is accessible in real-time, and that the data can be adequately analyzed, in real-time, by Stem’s Athena’s platform.

Solution architecture

Stem’s Athena platform collects data from several thousand devices in the field that continuously stream data every second. This is currently over a terabyte of raw data per day and constantly growing. As Stem’s business grew and the Athena platform amassed greater volumes of data at faster intervals, Stem needed a storage system that could provide the unlimited storage of data for long-term archiving. Furthermore, they knew early on that its AI-based Athena platform would be best served by the elastic big data storage and elastic compute, which are only possible in a cloud-native environment.

Stem chose AWS as the cloud infrastructure provider given its proven track record in providing scalable infrastructure that’s easy to integrate. Over time, the technical stack components that Stem uses have changed, and many AWS offerings have grown and evolved to continue to support Stem’s infrastructure needs.

figure-stem-athena-ai

Figure: Stem’s Athena AI Solution architecture on AWS

Since 2017, Stem standardized the utilization of container technology for consistency and ease when deploying its services. Stem’s build process generates container images, which can then be deployed on developer workstations using Docker Compose and on production compute clusters using Kubernetes. Stem was an early adopter of Kubernetes and at the time had to deploy and manage its own Kubernetes clusters on Amazon Elastic Compute Cloud (Amazon EC2). When AWS started offering Amazon Elastic Kubernetes Service (Amazon EKS), Stem found immediate value in making the switch. Amazon EKS provided pre-packaged Kubernetes integrations into AWS services, such as Elastic Load Balancing, optimized networking drivers, and the ease of staying up-to-date on frequent security and feature releases. Besides the elastic scaling of Amazon EKS clusters on Amazon EC2, Stem is now evaluating AWS Fargate for the serverless elastic scalability of compute capacity on Amazon EKS. Stem uses Kubernetes Jobs to spawn hundreds of containers for AI and ML model training and simulations as needed, and the compute elasticity provided by Amazon EKS and Fargate makes it effortless.

Managing growing volumes of data

Over the years, Stem has amassed petabytes of data and relies on Amazon Simple Storage Service (Amazon S3) for the tiered storage of that data. By leveraging Amazon Kinesis to stream data and Amazon Kinesis Data Firehose to continuously persist it to Amazon S3, Stem has an efficient, no-code, and serverless method for archiving incoming data. Furthermore, this lets Stem focus on building stream processing components for other core value-added data processing tasks.

Stem’s operations team wanted the ability to do ad-hoc queries on raw data for various analyses, often on data for the lifetime of the systems deployed. Stem didn’t want them to bog down its operational databases with the ad-hoc query load. Moreover, operational databases only contained a short time span of high-resolution data. Since Stem archives data using near real-time streaming into Amazon S3, the team turned to Presto on Amazon EMR to support these query needs. This lets Stem use dedicated compute capacity to query data on Amazon S3 without impacting the performance of its operational data stores. Stem did experience issues with right-sizing the Presto cluster to support variable query load and subsequently switched to Amazon Athena (not to be confused with Stem’s Athena platform) for these queries. Stem was among the early adopters of AWS Athena and welcomed the ease of use of serverless queries on AmazonS3 data. The pain points shifted to the manual registration of numerous partitions that Stem had in Amazon S3 data. In later versions, AWS Athena introduced partition projection, which improved performance and eliminated the need for partition management in the Glue metadata store. Over the years, Stem has evaluated several technologies to replace AWS Athena, but none could beat the performance and cost-effectiveness for its needs. Today, in addition to long-term ad-hoc data mining, Stem uses AWS Athena for several near real-time analyses and monitoring data pipelines used by network operations.

In addition to Amazon S3, Stem uses several other data stores including Amazon ElastiCache for Redis for various forms of in-memory storage, Amazon DynamoDB as an operational data store of time series data, Amazon Aurora for relational data, and Amazon OpenSearch Service for logs and dashboard monitoring.

AWS technologies are helping Stem reliably manage big data and big compute for our AI-powered energy management solutions.

Conclusion

In this post, you learned how Amazon services are helping Stem support its data-driven intelligent energy platform, all while achieving the AWS design principles of sustainability. Amazon’s pursuit of serverless technologies enables right-sizing workloads to drive optimal resource utilization and reduce energy demand for the data center. Stem’s usage of these technologies not only aligns with Amazon’s sustainability goals, but also results in cost reductions and environmental impact for Stem.

If you would like to get started building your own solution, contact us at sustainability@amazon.com.

TAGS: , , , , , , , , , , , , ,
Larsh Johnson

Larsh Johnson

Larsh Johnson, Chief Technology Officer, Stem As Chief Technology Officer, Larsh Johnson leads hardware and software engineering to meet the unique needs of Stem’s C&I, utility and energy market customers. Prior to joining Stem, Larsh was Chief Technology Officer at Siemens Digital Grid, where he led technology development teams on products spanning from consumer metering, demand response and analytics to control center software and grid automation. He joined Siemens via the acquisition of eMeter, a Bay Area software company of which he was a co-founder and responsible for innovation and development of meter data management, analytics and advanced smart grid applications. Prior to eMeter, he co-founded CellNet Data Systems, a pioneer in wireless networks for smart metering and distribution automation and now a unit of Landis+Gyr. Larsh was a founding member of the Department of Energy’s Gridwise Architecture Council (GWAC) and remains a member emeritus. He earned his B.S. and an M.S. in mechanical engineering from Stanford University.

Sunit Randhawa

Sunit Randhawa

Sunit is a Senior Solutions Architect at AWS working with digital native customers on their cloud native journey. He is a builder at heart and passionate about leveraging containers and AI/ML technologies to help small to enterprise customers innovate faster. Sunit has previously founded startup in the Loyalty space called PunchPie and has held various senior technical leadership positions in various companies ranging from startups to publicly listed companies.

Prashant Tyagi

Prashant Tyagi

Prashant joined AWS in September 2020, where he now manages the solutions architecture team focused on enabling digital native businesses. Prashant worked previously at ThermoFisher Scientific, and GE Digital where he held roles as Sr. Director for their Digital Transformation initiatives. Prashant has enabled digital transformation for customers in the Life Sciences and other industry verticals. He has experience in IoT, Data Lakes and AI/ML technical domains. He lives in the bay area in California.

Vikas Walia

Vikas Walia

Vikas Walia, VP & Chief Platform Architect, Stem As Chief Platform Architect, Vikas Walia leads product engineering and design at Stem. Prior to Stem he served as Principal Key Expert, Energy Business Platform & Applications, at Siemens, where he was responsible for engineering and design of wide range of solutions for Advanced Metering Infrastructure, Meter Data Management, Billing, Settlement, Demand Response, Energy Management, Market Hubs and Smart Grid Analytics. He joined Siemens via acquisition of eMeter Inc. where he was Chief Architect of the EnergyIP suite of products, which have been consistently ranked by Gartner as the visionary and leader in that space. He earned his B.E. in Electronics and Telecommunications from NSIT, Delhi.