Utopus Reduced Data Processing Time from Weeks to Hours by Going Serverless on AWS

Founded in 2017, Utopus aims to accelerate the integration of renewable energy into the modern grid by optimizing energy production using data analytics and insights. Utopus serves over 235 customers globally and holds over 90 granted and pending patents across its suite of products, ranging from asset management coverage to power forecasting insights. This suite is underpinned by the company’s platform, Scipher, which collects data from wind and solar assets to drive reporting and actionable insights using machine learning (ML). Using this solution, businesses can proactively monitor their assets worldwide and predict maintenance needs or part failures, helping to increase the longevity and performance of their renewable energy assets. 

When launching Scipher in 2017, Utopus used an on-premises data center to power its digital solutions and process petabytes of energy data at any given time. As more customers adopted its solution, the company faced challenges in quickly scaling its infrastructure. “The amount of data that we had exceeded our processing capacity,” says Ziad Rida, technical director of architecture at Utopus. “Our main goal was to ingest the data and make it available for data science.” In 2018, Utopus engaged AWS to accelerate its data processing capabilities and optimize its compute resources. To test the viability of working in the cloud, the company developed proofs of concept working alongside the AWS team. Utopus received resources and technical training with subject matter experts, and it kicked off its cloud migration that same year. “AWS helps us scale more rapidly than we could have on our own,” says Michael Wilkinson, chief product officer at Utopus. “At a time when both climate change and energy security are urgent global concerns, making data-driven innovations on AWS is the key to unlocking reliable, smart solutions both superfast and at scale.”

When migrating to AWS, Utopus opted for fully managed, serverless solutions when possible. “Using serverless solutions on AWS means that we don’t have to worry about scalability, manageability, installation, or configuration,” says Rida. For instance, the company adopted Amazon Kinesis Data Streams, which companies use to easily stream data at virtually any scale. On average, the company streams 7 TB of data through Scipher every day. 

After streaming its data, the company processes it using AWS Lambda, a serverless, event-driven compute service that companies use to run code for virtually any type of application or backend service without provisioning or managing servers. “AWS Lambda is highly scalable and very powerful,” says Rida. “We can manage latency, make our processing faster, and decide how much we want to spend.” Using AWS Lambda, the company processes over 200 billion signals from wind and solar sources every day, and in under 2 hours, it can process the same amount of data that would have taken 2 weeks to process on premises. By gaining deeper and timelier insights, its customers can improve the performance of their renewable energy assets and minimize their carbon footprints. 

Utopus also uses AWS Fargate, which provides serverless compute for containers, as the basis for its unified scoring pipeline. To containerize its applications, Utopus uses Amazon Elastic Kubernetes Service (Amazon EKS), a managed container service, to run and scale Kubernetes applications in the cloud or on premises. Using these solutions together, the company can provide short-term forecasting by automatically applying the most recent data to its ML models, improving the cost efficiency and timeliness of scoring functions. 

To further improve the performance of its ML, Utopus stores its data in the cloud using Amazon Simple Storage Service (Amazon S3), an object storage service built to retrieve any amount of data from anywhere. Utopus can train its ML models using structured and unstructured data that it stores on Amazon S3, driving deeper analytics and insights. The company has quickly grown to store data from over 55,000 wind turbines across more than 65 countries, and it has built a data lake on Amazon S3, which the company has used to store up to 266 TB of data. 

Utopus runs some of its workloads on Amazon Elastic Compute Cloud (Amazon EC2), which offers secure and resizable compute capacity for virtually any workload. “We modify our Amazon EC2 instances to gain cost savings,” says Rida. By using different Amazon EC2 instances, Utopus has increased the availability of its applications, and it can scale to support over 50,000 processing unit cores simultaneously. Utopus also implemented Amazon EC2 Auto Scaling, which gives companies the ability to add or remove compute capacity to meet changes in demand. By automating this task, the company has increased agility, accelerated its time to market, and paved the way for innovation. Now, Utopus can focus on developing new products that offer its customers deeper insights on how to maximize their renewable energy resources. “We have the ability to explore new ideas and services,” says Rida. “We can scale as needed and get our product out to market, which is critical.”

Utopus completed its migration to AWS in 2019 and continues to try new services to improve the performance of Scipher. To further reduce its time to market and introduce new capabilities, Utopus is trialing AWS IoT Greengrass, which helps companies build IoT devices faster. The company aims to accelerate the adoption of renewable energy across all industries. “Frankly, we couldn’t be doing the work of trying to decarbonize the electricity system without using AWS,” says Wilkinson. 

To future proof its environment, Utopus is performing an AWS Well-Architected review, helping the company to learn, measure, and build using architectural best practices. “It’s a very useful activity that we’re doing,” says Rida. “The kind of collaboration that we get from AWS is invaluable.”