Utopus Reduced Data Processing Time from Weeks to Hours by Going Serverless on AWS
Renewable energy analytics provider Utopus Insights (Utopus) wanted to generate more computing power so that it could improve the performance of Scipher, its industrial Internet of Things (IoT) analytics solution platform that assesses the performance of wind and solar energy assets using data in near real time. When the company initially launched the platform, it ran the solution on premises, which required ongoing investments in hardware to scale its infrastructure to meet increasing demand.
To meet its data processing needs, Utopus explored going serverless and decided to migrate to Amazon Web Services (AWS) in 2018. A year later, Utopus became fully native to the cloud, which has helped it scale quickly, accelerate its time to market, and achieve compute cost optimization. The company has maintained high availability on Scipher and reduced the time that it takes to process data from weeks down to hours. By working on AWS and expanding its data processing capabilities, the company can focus on innovating its digital solutions and facilitating the global transition to clean energy.
Using serverless solutions on AWS means that we don’t have to worry about scalability, manageability, or configuration.”
Technical Director of Architecture, Utopus Insights
Exceeding Data Processing Capacity on Premises
Founded in 2017, Utopus aims to accelerate the integration of renewable energy into the modern grid by optimizing energy production using data analytics and insights. Utopus serves over 235 customers globally and holds over 90 granted and pending patents across its suite of products, ranging from asset management coverage to power forecasting insights. This suite is underpinned by the company’s platform, Scipher, which collects data from wind and solar assets to drive reporting and actionable insights using machine learning (ML). Using this solution, businesses can proactively monitor their assets worldwide and predict maintenance needs or part failures, helping to increase the longevity and performance of their renewable energy assets.
When launching Scipher in 2017, Utopus used an on-premises data center to power its digital solutions and process petabytes of energy data at any given time. As more customers adopted its solution, the company faced challenges in quickly scaling its infrastructure. “The amount of data that we had exceeded our processing capacity,” says Ziad Rida, technical director of architecture at Utopus. “Our main goal was to ingest the data and make it available for data science.” In 2018, Utopus engaged AWS to accelerate its data processing capabilities and optimize its compute resources. To test the viability of working in the cloud, the company developed proofs of concept working alongside the AWS team. Utopus received resources and technical training with subject matter experts, and it kicked off its cloud migration that same year. “AWS helps us scale more rapidly than we could have on our own,” says Michael Wilkinson, chief product officer at Utopus. “At a time when both climate change and energy security are urgent global concerns, making data-driven innovations on AWS is the key to unlocking reliable, smart solutions both superfast and at scale.”
Taking a Serverless-First Approach to Scale to Demand
When migrating to AWS, Utopus opted for fully managed, serverless solutions when possible. “Using serverless solutions on AWS means that we don’t have to worry about scalability, manageability, installation, or configuration,” says Rida. For instance, the company adopted Amazon Kinesis Data Streams, which companies use to easily stream data at virtually any scale. On average, the company streams 7 TB of data through Scipher every day.
After streaming its data, the company processes it using AWS Lambda, a serverless, event-driven compute service that companies use to run code for virtually any type of application or backend service without provisioning or managing servers. “AWS Lambda is highly scalable and very powerful,” says Rida. “We can manage latency, make our processing faster, and decide how much we want to spend.” Using AWS Lambda, the company processes over 200 billion signals from wind and solar sources every day, and in under 2 hours, it can process the same amount of data that would have taken 2 weeks to process on premises. By gaining deeper and timelier insights, its customers can improve the performance of their renewable energy assets and minimize their carbon footprints.
Utopus also uses AWS Fargate, which provides serverless compute for containers, as the basis for its unified scoring pipeline. To containerize its applications, Utopus uses Amazon Elastic Kubernetes Service (Amazon EKS), a managed container service, to run and scale Kubernetes applications in the cloud or on premises. Using these solutions together, the company can provide short-term forecasting by automatically applying the most recent data to its ML models, improving the cost efficiency and timeliness of scoring functions.
To further improve the performance of its ML, Utopus stores its data in the cloud using Amazon Simple Storage Service (Amazon S3), an object storage service built to retrieve any amount of data from anywhere. Utopus can train its ML models using structured and unstructured data that it stores on Amazon S3, driving deeper analytics and insights. The company has quickly grown to store data from over 55,000 wind turbines across more than 65 countries, and it has built a data lake on Amazon S3, which the company has used to store up to 266 TB of data.
Utopus runs some of its workloads on Amazon Elastic Compute Cloud (Amazon EC2), which offers secure and resizable compute capacity for virtually any workload. “We modify our Amazon EC2 instances to gain cost savings,” says Rida. By using different Amazon EC2 instances, Utopus has increased the availability of its applications, and it can scale to support over 50,000 processing unit cores simultaneously. Utopus also implemented Amazon EC2 Auto Scaling, which gives companies the ability to add or remove compute capacity to meet changes in demand. By automating this task, the company has increased agility, accelerated its time to market, and paved the way for innovation. Now, Utopus can focus on developing new products that offer its customers deeper insights on how to maximize their renewable energy resources. “We have the ability to explore new ideas and services,” says Rida. “We can scale as needed and get our product out to market, which is critical.”
Decarbonizing the Electricity System on AWS
Utopus completed its migration to AWS in 2019 and continues to try new services to improve the performance of Scipher. To further reduce its time to market and introduce new capabilities, Utopus is trialing AWS IoT Greengrass, which helps companies build IoT devices faster. The company aims to accelerate the adoption of renewable energy across all industries. “Frankly, we couldn’t be doing the work of trying to decarbonize the electricity system without using AWS,” says Wilkinson.
To future proof its environment, Utopus is performing an AWS Well-Architected review, helping the company to learn, measure, and build using architectural best practices. “It’s a very useful activity that we’re doing,” says Rida. “The kind of collaboration that we get from AWS is invaluable.”
About Utopus Insights
Utopus Insights, a subsidiary of Vestas, is a data-driven energy analytics company that develops renewable energy software products for customers worldwide. Based in New York, with development centers in Bengaluru, India, and Budapest, Hungary, and a presence across North America, Europe, and Asia, it seeks to accelerate an era of reliable, clean, and cost-effective energy worldwide.
Benefits of AWS
- Achieved high availability on Scipher
- Supports the adoption of renewable energy analytics software
- Reduced data processing time from weeks to hours
- Streams an average of 7 TB of data every day
- Processes roughly 200+ billion signals from wind and solar sources daily
- Stores data from 55,000+ wind turbines across 65 countries
- Improved accuracy of ML models by building a data lake
- Accelerated time to market and innovation
- Optimized compute resources
AWS Services Used
Amazon Kinesis Data Streams
Amazon Kinesis Data Streams is a serverless streaming data service that makes it easy to capture, process, and store data streams at any scale.
Amazon Simple Storage Service (Amazon S3)
Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance.
AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers.
Amazon Elastic Compute Cloud (Amazon EC2) Auto Scaling
Amazon EC2 Auto Scaling helps you maintain application availability and allows you to automatically add or remove EC2 instances according to conditions you define.
Organizations of all sizes across all industries are transforming their businesses and delivering on their missions every day using AWS. Contact our experts and start your own AWS journey today.