AWS for Industries
How to manage and bill structured energy contracts on AWS
Energy companies are still using architectures based on monolithic systems when managing and billing energy service contracts. Architectures where the billing processes have been built using stored procedures inside a database are common. These architectures limit the capabilities of the business units to create, manage, and invoice contracts based on new products and services, especially when a higher flexibility in contract management and billing is needed.
To reduce the time to market of new products and services, increase the agility of the technical teams when implementing new business requirements, and reduce the interdependency of the different actors who participate in the business processes, a domain-driven architecture is more adequate. A domain-driven architecture uses specialized architecture patterns and software modules for each functional domain rather than having all the data and programmatic functions laying within the same monolith.
Figure 1 shows all the main functional blocks required to build a system that is able to contract; bill; invoice the contracts; manage the risks associated with the energy traded with third parties; manage the transactions interchanged with the transmission system operator (TSO), the distribution system operator (DSO), and the markets; and properly maintain the relations with the clients through a purpose-fit customer-relations-management system (CRM).
Figure 1. Domain-driven architecture for an energy company.
In this blog, we are not exploring all the components and processes of an energy company, but we will focus on one of the core processes: billing and calculating the economic value of contracts, the profit part of profit and loss (P&L), using a suite of services from Amazon Web Services (AWS).
Structured energy contracts
Structured energy contracts are gas volumes or electrical energy that utility companies commit to deliver to their customers at a certain price. The utility plays a central role in this type of agreement; it buys the energy either from a producer through a bilateral contract or on the energy markets and then agrees to deliver that bought energy to the final client in a certain time frame. Structured energy contracts are highly tailored contracts where the utility and the customer agree on personalized prices and energy volumes that are specific for each contract.
Electro-intensive businesses run by gas or power sign structured contracts to protect themselves from the variability of the energy prices of the markets, to properly plan their costs, and optimize the price at which they buy the energy. On the other side, energy producers are interested to sell the production ahead of time to account for long-term operation.
Take or pay contracts are structured contracts formed by a baseline of energy that is agreed to be delivered at a fixed price, and if the customer’s final consumption exceeds the baseline, the additional energy is priced according to an energy-index market. On the contrary, if the final client consumption is below the baseline, the customer needs to pay the agreed-upon volume at the contracted price. Contract structure is represented in the form of time series bands (see figure 2 for an example).
Figure 2. A typical contract structure—the customer and the utility are agreeing on 18 MWh at a price of €30. The next 7 MWh will be priced at €35, and any additional energy delivered at the service point will be priced according to a reference energy price index, such as the Nord Pool index.
Technical representation of contracts and object modeling
Structured energy contracts are not based on products coming from a product or service catalogue. Each contract needs to solve a custom case, in the extreme, being able to configure a different price and a different delivered energy volume for each hour and with different levels of energy aggregation, such as aggregation by the number of participating service points and/or the number of customer subaccounts. These contracts are difficult to fit into a SQL model because they are highly tailored.
An alternative to SQL is a JavaScript object notation (JSON), which is an open-data interchange format that is both human and machine readable. You can build a JSON file that defines a structured energy contract and is understood or even built by a business user. With rules agreed upon by the business and technical teams, the business team can define a JSON document that represents the contract and is signed by the customer as shown below.
In this example, the JSON document array servicepoints represents a list of meters that are participating in the agreement. It could be an array of only one service point, so all the energy is delivered to that service point, or it can be an array that includes several service points. In such a case, measurements at each service point need to be added up.
The energypricevector array represents, in short notation, the time series set (price, energy, hour) that has been agreed on. It refers to the band structure that needs to be created in a time series format. For example, the following array [[“01/01/2024”,”31/12/2024”, “30”, “18”], [“01/01/2024”,”31/12/2024”, “35”, “7”], [“01/01/2024”,”31/12/2024”, “NORDPOOL”] contains instructions for a programmed function to generate time series records attached to the contract with hourly resolution in the following way:
Information about the tariff that applies to each service point and other technical data needs to be stored in an additional service point entity that is not discussed in this blog. Technical data of a service point includes information like the maximum admissible power, average annual consumption, electrical tension, tariff, and address, among other technical information. Once the contracts have been registered and before being able to bill them, measurements at the service point must be loaded. The recommendation is to initially load a forecasted measurement for the life of each contract and then correct that forecasted measurement with the real data as the information comes in. With that, it becomes straightforward to calculate the initial contract value versus the real contract value on a daily basis.
Contract forecasted value will be computed by an add function: contract value = Σ1n servicepoint Σ1n hour Σin bandp (price band or forecasted price index) * q (forecasted)
For each service point in the contract, for each hour, and for each band, you multiply the price of the forecasted delivered-energy volume times the forecasted price.
Contract real value will be computed exactly in the same way but substituting forecasted values with actual values. Every day you can compute the forecasted and actual values of all the contracts and visualize the result in a curve that shows how the portfolio performs.
Figure 3 shows a diagram with the data object model that is required to store the information of the contract. The entities are contract definition, contract time series representation of the contract definition, service points, forecasted measurement per service point, real measurement per service point, and time series of the markets’ price indexes.
Figure 3. Main objects required to store the information that will be needed to perform the billing and contract-value calculations.
Solution architecture
Contract definition
Figure 4 shows the architecture to manage the data preparation and manipulation stages where data contract is inserted, updated, and queried based on user requests.
Figure 4. Architecture for contract management.
There will be an API that consumers will use to store contract information and update the JSON objects when required. The API will need to implement logic for data validation and will also implement and exploit AWS Lambda — a serverless, event-driven compute service—that transforms the contract definition into a time series representation. Every time a JSON is updated in the Amazon DocumentDB (with MongoDB capability), a fully managed native JSON document database, by the contract management AWS Lambda, the contract exploits function is called to ensure the contract is properly represented in its time series format.
Price and energy time series management
Figure 5 depicts the architecture to process the files that are coming with the index prices, load curves, and metering. Depending on the size of the portfolio, these files might contain a significant volume of data and come in large numbers; therefore, it is important to plan ahead and implement autoscaling. For an example, see https://aws.amazon.com/blogs/containers/amazon-elastic-container-service-ecs-auto-scaling-using-custom-metrics/. With AWS Fargate—a serverless, pay-as-you-go compute engine—you use a serverless architecture so that you can focus on deploying and managing applications, not infrastructure. AWS Fargate removes the operational overhead of scaling, patching, securing, and managing servers.
In this case, the use of Amazon Elastic Container Service (Amazon ECS), a fully managed container orchestration service, tasks running on AWS Fargate are more appropriate than the use of AWS Lambda because AWS Lambda functions have a limitation on execution time. File processing can exceed the maximum time allowed by AWS Lambda.
In Figure 5, there is another element of interest, the presence of an endpoint in Amazon SageMaker, which is used to build, train, and deploy machine learning (ML) models for nearly any use case. This endpoint runs a forecasting ML model to inference measurements and prices for the dates without real measurements or prices.
Figure 6 shows this scenario. Real measurements coming from the files are loaded into the database tables in Amazon Timestream, a fast, scalable, and serverless time series database service. For the dates where measurements do not exist, you can inference them by a forecasting ML model, and the predictions can be stored in the Amazon Timestream database. Having measurements for the whole life of the contract will allow you to calculate the total economic value of the contract. When the contract starts, you will only have a prediction or reference load curve. All the energy values will be forecasted. As time passes by, you will compute real economic value for real measurements and forecasted contract value for forecasted measurements. You can visualize forecasted versus realized contract value and have high-quality information about the portfolio performance—what you expected to earn versus what you are really earning.
Figure 5. Architecture for index price and measurements time series storage and query.
Figure 6. Real measurements are loaded by the file import processes. Forecasted measurements are loaded by Amazon SageMaker inference endpoint.
Billing contracts
Finally, once all the time series records are properly stored, the calculation of the contract value and the value of the portfolio can be computed every day by a batch process. The batch calculates daily the actual and forecasted price-quantity view.
Figure 7 shows the final module of the architecture. The computation engine is a batch process. It calculates daily the value of all the contracts and stores the results in Amazon Simple Storage Service (Amazon S3), an object storage service. The information stored in Amazon S3 can be directly exploited using the capabilities of Amazon QuickSight, which powers data-driven organizations with unified business intelligence at hyperscale, through Amazon Athena, a serverless, interactive analytics service.
The jobs in the batch processor request the information from the APIs to retrieve prices, energy quantities, and contract information. The outcome of the calculations is stored in Amazon S3, where users can exploit the calculations according to their needs.
Figure 7. Architecture for the calculation engine.
Conclusion
In this blog, an open, scalable architecture based on microservices is presented to manage structured energy contracts. The architecture is valid for gas and power. For gas contracts, time series resolution is daily, while power contracts have an hourly or even 15-minute resolution. The architecture serves equally for new enterprises that need to build or enhance their current enterprise, as well as consolidated enterprises that are running their billing engine as SQL-stored procedures inside the core operational databases or as complex monolithic batch jobs inside a mainframe.
This architecture will allow you to start decoupling all-in-one transactional systems with no risk and at speed. You can select a number of contracts from your portfolio, move them to this architecture, pilot the new solution, and then move the rest of the portfolio. It gives enterprises a common approach to manage nearly any type of contract decoupling energy management, price management, contract management, calculations, and storage of the final calculations.