Building scalable DERMS solutions for DER aggregators on AWS

As the energy landscape evolves toward a more distributed model, distributed energy resources (DERs) bring not only challenges, but also opportunities to different players in the energy market (utilities, legislators, aggregators, consumers, and service providers).

We have planned a series of blogs discussing how different stakeholders can use Amazon Web Services (AWS) to make the best use of DERs. In our first blog, we’ll explore how AWS services can help aggregators build a robust distributed energy resource management system (DERMS) that scales as their businesses grow.

The DER aggregator challenge

DER aggregators operate in an increasingly complex environment where they must efficiently manage thousands of distributed energy assets. These assets range from residential solar installations and industrial battery storage systems to widespread electric vehicle charging networks. The challenge extends beyond asset management. Aggregators must optimize their market participation across various energy markets, while ensuring regulatory compliance and maintaining high levels of service reliability.

Success in this environment requires sophisticated solutions capable of handling real-time monitoring and control, as well as integrating with multiple communication protocols. They also need to execute advanced forecasting and optimization algorithms, maintain robust cybersecurity measures, and manage massive amounts of data efficiently. These technical challenges are compounded by the heterogeneous nature of DER portfolios. Aggregators must harmonize different technologies with varying response times, operational constraints, and communication capabilities.

Implementing such solutions demands a robust telecommunication infrastructure and advanced cybersecurity measures to protect against emerging threats to grid security, all while maintaining cost-effectiveness. Furthermore, aggregators must develop complex bidding strategies that account for both system-level constraints and individual DER limitations. They must also address intricate settlement processes and manage financial risks across multiple market products.

The regulatory landscape is also influencing the decision-making process for grid operators on how to solve challenges. In North America for example, the implementation of FERC Order 2222 requires Regional Transmission Organizations (RTOs) and Independent System Operators (ISOs) to enable DER aggregation participation in wholesale markets. The regulatory order mandates meeting minimum size requirements (specifically, aggregation can be as small as 100 kW), establishing dual participation rules between retail and wholesale markets, and coordinating with distribution utilities to ensure reliable grid operations.

Another example is the Paragraph 14a of the Energy Industry Act in Germany, which requires Distribution System Operators (DSOs) to actively manage customer-owned DERs (such as heat pumps, EV chargers, and batteries). While this regulation enables better grid stability through dynamic DER control, it requires aggregators to carefully balance DSO requirements with their market commitments.

These regulatory frameworks exemplify the evolving relationship between grid operators, DER owners, and aggregators. It highlights the need for increasingly sophisticated management and communication systems.

Solution overview

AWS provides a comprehensive suite of services that provide DSOs and Aggregators a way to address the complex challenges of modern DER management. In the following sections we will dive deep into how this can be achieved by walking you through a cloud-native architecture implementation of DERMS on AWS.

This architecture aims to solve critical industry pain points including:

• Near real-time visibility and control of heterogeneous DER assets
• Integration of multiple communication protocols
• Dynamic optimization across distribution constraints
• Market participation coordination
• Seamless scaling from hundreds to millions of connected devices

The architecture brings together edge computing capabilities, robust data streaming, advanced analytics, and enterprise-grade security. AWS empowers utilities to transform from traditional distribution system operators into sophisticated distributed system orchestrators.

Figure 1 – High-level solution architecture

Let us now dive deep in each section of the architecture, why it’s relevant to address the problem and how AWS services are helping to resolve it. Following are the sections we will discuss:

1. DER integration at the edge
2. Data ingestion
3. Data streaming
4. Storage layers
4.1 hot storage – Time-series database
4.2 cold storage – Data lake
5. Real-time analytics
6. Application integrations
7. DERMS application
8. Artificial intelligence and machine learning
9. Security and compliance

1. DER integration at the edge
DERs present unique integration challenges due to their diverse nature and reliance on traditional industrial protocols. Many DER assets communicate using local protocols such as Modbus and DNP3, making direct cloud integration impossible. To bridge this gap, AWS provides AWS IoT Greengrass, an open-source edge runtime and cloud service for building, deploying, and managing device software. As a service that runs at the edge, AWS IoT Greengrass can manage the logic to transform local protocols into cloud-compatible formats. It enables secure and reliable data transmission to AWS using modern, standardized protocols.

AWS IoT Greengrass uses a modular design: some modules are provided by AWS while others can be developed by customers or end user. Furthermore, many AWS Partners directly support or integrate with AWS IoT Greengrass, providing protocol adapters and other capabilities to facilitate device integrations.

Typical integrations are through protocols compatible with:

• Standard IEEE 2030.5 (widely used in the North American utilities)
• Modbus (common in industrial settings)
• DNP3 (prevalent in utility operations)
• IEC-61850 (critical for substation automation)

2. Data ingestion
As DER portfolios grow from hundreds to millions of assets, utilities face increasing complexity in managing device registration, organizing assets by location or type, monitoring operational status, and executing coordinated control actions. AWS IoT Core serves as the central hub for secure device connectivity, handling millions of concurrent connections from DER assets.

AWS IoT Device Management addresses these scalability challenges by providing comprehensive capabilities to register, group, search, monitor, and remotely manage edge devices at scale. This provides a way for utilities to efficiently organize and control their DER fleet. For example, grouping all batteries within a specific city, postal code, or geographic region for coordinated demand response events.

For legacy or existing DER assets that cannot be quickly updated, AWS IoT Greengrass can act as an intermediary management device. This allows older devices to continue using their native protocols while AWS IoT Greengrass handles protocol translation, security, and cloud connectivity. Additionally, a custom protocol adapter can enable integration with proprietary or legacy protocols. This confirms that even non-standard devices can be incorporated into the DERMS ecosystem without requiring a hardware upgrade.

One of the most critical challenges in DER management is maintaining reliable control over distributed assets with intermittent network connectivity. Grid operators need guaranteed command delivery and state synchronization, even when devices temporarily lose connection. The AWS IoT Device Shadow service solves this by maintaining a persistent virtual representation (or shadow) of each device in the cloud.

This shadow service three state views: the desired state (what the solution wants), the reported state (last known device status), and the delta state (difference between desired and reported). When a device goes offline, any control commands are stored in the desired state. Upon reconnection, the device automatically receives and processes these pending commands, verifying no control actions are lost. If conflicts arise between device and solution updates, the shadow service employs last-writer-wins conflict resolution with version tracking to maintain data consistency.

AWS IoT Device Defender protects IoT devices by continuously monitoring for abnormal behavior, detecting security issues, and alerting you to potential threats before they can compromise the IoT fleet.

All the data flowing through AWS IoT Core is routed to other AWS services using AWS IoT Rules. Each IoT Rule selects and sends data to the next functional block, the Data Streaming services.

3. Data streaming
Modern DERMS platforms must handle diverse data processing requirements that mirror the complexity of DER operations. Grid operators treat aggregated DERs as virtual power plants, requiring monitoring and control capabilities similar to traditional generators. The timing requirements span a wide spectrum: critical grid services such as frequency response and voltage support demand sub-second monitoring and control, typically between 100 millisecond to 1 second intervals. Larger DER assets (>1 MW) performing spinning reserve or capacity services require 2 to 4 second monitoring, while behind-the-meter DERs participating in demand response or energy arbitrage typically report at 5 to 15 minute intervals. Additionally, long-term planning and settlement functions need months or years of historical data for analysis and compliance.

AWS provides different streaming services to address these needs:

• Amazon Kinesis Data Streams is a serverless service and is ideal for scaling up and down DER deployments quickly without guessing capacity. Kinesis Data Streams integrates natively with other AWS services and is recommended for use in fully AWS-native architecture with dynamic numbers of DERs connecting and disconnecting with the DERMS solution.
• Amazon Managed Streaming for Apache Kafka (Amazon MSK) is suited for more stable DERMS deployments with expected volumes of messages. Amazon MSK can keep the processed data for longer times and act as storage. This feature can be used to perform reprocessing directly from Kinesis Data Streams without using external storage. Amazon MSK is recommended to be used in more cases of predictable numbers of DERs to be handled with non-AWS native architectures.

For less time-sensitive data flows, such as archiving telemetry data for regulatory compliance or long-term planning, Amazon Kinesis Firehose offers a code-free solution. It can directly receive data from AWS IoT Core and write to Amazon Simple Storage Service (Amazon S3), with built-in options for data batching and partitioning.

4. Storage layers
As described previously, DER operations span diverse timescales, from millisecond-level grid stability responses to hourly market participation.

This necessitates a dual-storage approach:

• A time-series database (4.1) for low-latency, real-time operations
• A data lake (4.2) for higher-latency historical analysis and planning

The solution architecture optimizes performance and cost, while meeting the varied timing demands of modern DER management.

Time-series database (4.1 hot storage) and data lake (4.2 cold storage)
The hot storage layer, optimized for low-latency access, is crucial for real-time DER operations where response time directly impacts grid stability. Time-critical functions (such as frequency regulation) require data access in milliseconds, while demand response programs typically need response times within seconds.

Amazon Timestream for InfluxDB serves these needs so utilities can focus on operations rather than database administration. For more complex queries involving both time-series and relational data, Amazon OpenSearch Service provides flexible indexing and search capabilities. It is particularly valuable for fault analysis and performance monitoring that require correlating DER behavior with grid events.

The cold storage layer, implemented using Amazon S3 and AWS Lake Formation, handles historical data analysis and regulatory compliance needs. This layer stores comprehensive operational data, from individual device telemetry to aggregate performance metrics, typically accessed within minutes rather than seconds. While the response time is slower, the cost for each terabyte is significantly lower. This is critical for utilities managing years of DER operational data for regulatory reporting and long-term planning. AWS Lake Formation centralizes permission management and streamlines secure data sharing across organizational boundaries, essential for coordinating with regulatory bodies and market operators.

An AWS Glue Data Catalog serves as a central metadata repository, making it quicker to discover and manage data assets across the organization. This catalog maintains a comprehensive inventory of data attributes, schemas, and locations, so teams can quickly find and access relevant historical data for analysis. AWS Glue complements this by automating data preparation and transformation tasks, making it faster to analyze historical trends and performance patterns.

The decision to move data from hot storage to cold storage is driven by balancing cost savings against data access requirements. Hot storage data that is no longer used for mission critical, real-time DER operations (for instances, after a few hours or a day), may be considered for moving to cold storage. This older data can be pre-defined by a time threshold for moving into cold storage for cost saving and performance purposes.

5. Real-time analytics
The true value of DER operations emerges through three critical dimensions. First, grid operators rely on DERs as powerful tools for managing peak loads and maintaining grid stability. Second, consumers are increasingly focused on maximizing their financial returns through strategic market participation. Third, utilities find significant value in using DERs to defer costly infrastructure upgrades.

To support these objectives, DERMS platforms must process and analyze an array of real-time data streams. These include everything from basic voltage and current measurements to sophisticated market signals and weather forecasts. Each data point plays a crucial role in making informed dispatch decisions that balance grid stability with market opportunities.

For implementing these near real-time analytics capabilities, AWS offers two distinct services that address different use cases:

• Amazon Managed Service for Apache Flink excels in scenarios such as coordinating thousands of DERs for frequency response, where stateful operations and exactly-once processing are crucial. The ability of Flink to handle event time processing makes it ideal for applications such as real-time market bidding, where precise timing and order of events matter.
• AWS Lambda complements this architecture by handling discrete, event-driven processing needs. It’s well-suited for tasks such as individual device telemetry validation, alarm processing, or updating device status in response to specific events.

These services typically work together in a comprehensive DERMS solution. Flink handles the continuous, complex stream processing requirements, while Lambda manages discrete, event-driven tasks.

6. Application integrations
DERMS must seamlessly integrate with a utility’s existing enterprise systems to function effectively.

Critical enterprise integrations begin with:

• Customer relationship management (CRM) systems provide essential data for DER program enrollment, billing, and service management
• Advanced metering infrastructure (AMI) delivers real-time energy consumption and production data from smart meters
• Geographic information system (GIS) mapping enables spatial awareness of DER locations relative to grid assets
• Supervisory control and data acquisition (SCADA) systems provide real-time grid state information and control capabilities
• Weather forecasting systems help predict renewable energy generation and load patterns
• Asset management systems track DER maintenance schedules and operational status
• Distribution management systems (DMS) ensure coordination with bulk power system operations

These integrations are critical for comprehensive decision-making. For example, combining CRM data with AMI readings allows for personalized demand response programs. While integrating GIS and SCADA enables location-aware DER dispatch for voltage support. Many utilities still host these systems on-premises, necessitating a flexible integration approach that can adapt as applications migrate to the cloud.

AWS offers tailored solutions for various integration scenarios. Amazon AppFlow excels in software as a service (SaaS)-to-AWS integration. It is particularly useful for modern CRM systems such as Salesforce or cloud-based weather services. For instance, it can automatically synchronize customer DER program enrollment data from a SaaS CRM for analysis.

AWS AppSync, leveraging GraphQL, is ideal for building real-time data APIs that aggregate information from multiple sources. In a DERMS context, it could create a unified API that combines live SCADA data, AMI readings, and weather forecasts, enabling near real-time DER optimization algorithms.

For legacy on-premises systems, AWS Direct Connect provides secure, high-bandwidth connectivity, while AWS Storage Gateway can create a bridge between on-premises data and cloud storage. Amazon API Gateway, combined with AWS Lambda, can create serverless API layers to decouple legacy applications.

7. DERMS application
The DERMS application layer represents the core business logic that provides aggregators a way to effectively manage their DER fleets.

Whether using a third-party solution or developing a custom application, this layer typically needs to support three fundamental capabilities:

• Asset management and control orchestrating logic for the registration, configuration, and operation of DER assets. It interacts with the digital twin of each asset, changing their operational parameters according to the grid signals. This component interfaces with the AWS IoT Device Shadow service to facilitate reliable command and control operations, even with intermittent connectivity.
• Market operations handle the participation in various energy markets by processing market signals, managing bids and offers, and coordinating asset dispatch. This component interacts with the streaming and analytics layers to make informed decisions about resource allocation and market participation strategies.
• Grid services management verifies the delivery of contracted grid services, such as frequency regulation, voltage support, and demand response. It processes grid operator signals and translates them into coordinated control actions across the DER fleet, while monitoring compliance with service requirements and grid constraints.

These core capabilities can run on Amazon Elastic Compute Cloud (Amazon EC2) or using containerized workloads on Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Kubernetes Service (Amazon EKS). In case of containers AWS Fargate can also be used to delegate the management of any server to AWS. Although most of the core logic is long running, some of the event-driven functions can be hosted on AWS Lambda for quicker scaling and management. Finally, metadata and configuration should be stored in a managed database through Amazon Relational Database Service (Amazon RDS).

Note, this application layer should be designed to leverage the rest of the architectural components for data management, real-time processing, and integration with external systems.

8. Artificial intelligence and machine learning
Artificial intelligence and machine learning (AI/ML) capabilities are essential for modern DERMS solutions, enabling better forecasting, optimization, and anomaly detection. AWS provides several services that can be leveraged to implement these critical functions without having to build complex ML infrastructure. Amazon SageMaker can serve as the foundation for developing, training, and deploying ML models at scale, while Amazon Bedrock provides straightforward access to leading foundation models through a single API for building generative AI applications.

For DER aggregators, key ML applications include:

• Load and generation forecasting predicts DER behavior by combining historical performance data with external factors, such as weather forecasts, seasonality, and local events. These predictions are critical for market participation and grid service delivery. Using the built-in forecasting algorithms or custom models through Amazon SageMaker, aggregators can generate accurate predictions at different time scales, from real-time to day-ahead forecasts.
• Equipment health monitoring utilizes ML models to detect anomalies in DER performance and predict potential failures before they occur. By processing real-time telemetry data through Amazon SageMaker near real-time inference endpoints. The solution can identify unusual patterns and trigger preventive maintenance actions, improve asset reliability and reduce operational costs.
• Price and market behavior analysis helps optimize market participation strategies. ML models can analyze historical market data, weather patterns, and grid conditions to predict price movements and identify optimal bidding strategies. This capability can be implemented using the built-in algorithms of Amazon SageMaker for time series analysis and reinforcement learning.

9. Security and compliance
Security is paramount for DERMS solutions that help manage critical energy infrastructure. Given that DERMS platforms manage grid-critical DER assets and handle sensitive operational data, security must be embedded at every layer of the architecture, while meeting stringent industry regulations such as NERC CIP and IEC 62351.

The DERMS security architecture addresses three critical protection domains:

• Device and communication security confirm that DER assets and their control systems are protected from unauthorized access and tampering. AWS Certificate Manager handles digital certificate lifecycle management for device authentication while AWS Key Management (AWS KMS) creates and controls encryption keys. The device authentication and authorization mechanisms of AWS IoT Core verify that only registered, verified devices can connect to the DERMS platform. All communication channels are encrypted using industry-standard TLS protocols, protecting control signals and telemetry data from interception or manipulation. The monitoring of the AWS resources in the account is handled by Amazon GuardDuty.
• Operational security protects the core DERMS decision-making and control functions. AWS Identity and Access Management (IAM) implements role-based access control and the principle of least privilege. It verifies that operators can only execute commands within their authorized scope. All operations and activities are also logged in Amazon CloudWatch for troubleshooting and reporting, while AWS Security Hub provides a comprehensive view of your security alerts and compliance status across multiple AWS accounts.
• Data security safeguards both real-time operational data and historical records required for compliance and analysis. AWS KMS provides encryption for data at rest and in transit, while AWS CloudTrail creates immutable audit logs of all system activities. This is essential for both security monitoring and regulatory compliance.

Looking Ahead

The transition to distributed energy resources represents both a challenge and an opportunity for energy aggregators. The provided architecture demonstrates how AWS services can be combined to create a robust, scalable DERMS solution that meets the complex demands of modern electricity markets.

By leveraging the managed services of AWS, aggregators can focus on their core business while benefiting from a comprehensive set of services and capabilities. The architecture enables reliable and secure device connectivity at scale, paired with near real-time data processing and analytics capabilities. The flexible storage services accommodate both real-time and historical data needs, while advanced AI/ML capabilities empower sophisticated forecasting and optimization. All of this is protected by comprehensive security and compliance controls designed for critical infrastructure.

The modular nature of this architecture provides aggregators a way to start with basic functionality and incrementally add capabilities as their business grows. Whether implementing a custom solution or integrating third-party DERMS software, AWS provides the foundational infrastructure to support the next generation of DERMS. As the energy transition accelerates, this architecture can evolve to support new market opportunities, grid services, and DER technologies. It can help aggregators stay competitive in an increasingly dynamic distributed energy landscape.

Contact an AWS Representative for more information about how we can help accelerate your business.

Further reading

• Concerto Optimize: securely manage and optimize the integration of behind- and front-of-meter distributed energy resources (DERs) in the electricity grid
• How to control distributed energy resources using AWS IoT
• DERMS on AWS
• Solutions for Energy and Utilities
• Security & Compliance for Energy & Utilities