Enhance price capture in energy and commodity trading with AWS machine learning

In the energy and commodity trading industry, traders negotiate transactions with brokers and counterparties directly. These negotiated transactions are called over-the-counter (OTC) deals. Price discovery for OTC deals is unlike price discovery for common equity, options, and other financial instruments, where pricing information is generally market-efficient and available in near real time. In OTC trades, the specific terms of traders’ agreements drive the price discovery process. For example, energy and commodity traders use the Interconnectional Exchange (ICE) trading solution’s chat program to discuss transactions and discover fair commodity prices. In this blog, we describe how global integrated energy company bp, an AWS for Energy trading customer, uses AWS machine learning (ML) and serverless technologies to enhance its price capture process. We share its AWS solution architecture and explain why it’s important to bp’s energy and commodity trading business.

In the OTC deal negotiation process, price quotes from traders’ chat messages are the source of intraday transactional pricing information. Once transaction terms are agreed upon, traders submit instructions in bp’s trading systems to execute a trade. At that point, pricing information gets recorded. The rest of the negotiation details and prospective transaction prices are not recorded. However, that information is still valuable because other personas in a trading organization—such as analysts, strategists, and risk managers—can use it to develop future trade strategies, marketing strategies, risk management, and additional market data sources.

bp trading wanted to improve trade strategies and profit and loss (P&L). They achieved this objective by building the application discussed in this blog. This application allows bp trading to capture all transactional data discussed in traders’ chats. Once data is captured, bp trading can provide strategists with detailed transactional data to develop trading strategies, generate P&L through new quantitative strategies, improve execution of existing trade strategies through value findings and alert systems, and improve end-of-day (EOD) marking process accuracy. bp trading analyzes its liquidity and manages its risk positions through this process. Actual transactional pricing throughout the day leads to more accurate EOD marks.

In the following sections, we will go over the business and data workflow and the AWS reference architecture of the solution.

Business and data workflow

Below is an image of a sample trader chat message, which shows that there are multiple entities in a quote. For example, it includes contract period, location, time, volume, bid, ask, and other pieces of information. Our task in this phase of the business and data workflow is to extract all these entities, including quotes and pricing information, and store them individually while discarding any personal messages.

Diagram 1. Sample chat message exchange between a trader and broker

The high-level business workflow begins nightly by pulling traders’ chat messages from the ICE chat platform. Raw data is cleansed and processed in chunks, and messages are examined for price quotes. Once a piece of information is classified as a price quote, the quote is processed for entity extraction. Various elements from the quote are labeled, and the data is stored in a persistent data store.

Diagram 2. Data and ML workflow

Solution architecture

The solution architecture includes six steps, detailed below.

1. Ingestion

a. Amazon EventBridge, a serverless event bus, provides a scheduler that triggers a job each night, invoking a function with AWS Lambda, a serverless, event-driven compute service. This AWS Lambda function pulls down the chat messages in XML format from the ICE chat Platform and stores them in Amazon Simple Storage Service (Amazon S3) bucket.

2. Message buffering

a. The Amazon S3 event notification triggers a Lambda function, which chunks the data and writes it to an Amazon Kinesis Data Streams, a serverless streaming data service. Amazon Kinesis Data Streams is used as the buffer to accommodate the near-real-time message processing planned for a later phase. In this near-real-time setup, we use the ICE APIs to pull chat messages and write them to an Amazon Kinesis Data Stream.

b. An AWS Lambda function with event source mapping to the Amazon Kinesis Data Stream polls the stream for new messages and processes them as soon as they are written to the stream. This Lambda function cleanses the messages, breaks them into chunks, and invokes a workflow in AWS Step Functions, a serverless orchestration service.

3. Message processing

a. A workflow in AWS Step Functions orchestrates Lambda functions that call the model endpoints to handle message classification and entity extraction.

b. If a message is classified as a quote by the classification AWS Lambda function, it further calls the entity extraction AWS Lambda function, which separates out individual entities.

4. Machine learning workflow

a. Building the models

i. Classifier: A scikit-learn Random Forest classifier was selected to probabilistically classify the line of text as a quote or non-quote. This model was selected for predictive and computational performance as well as for explainability. It was trained on the previous 7 years’ worth of data with reasonable class balance, resulting in mean F1 scores of 0.97 and no obvious bias on unseen data.

ii. Entity Extraction: Named entity extraction is performed using a custom trained spaCy (v2) EntityRecognizer, trained from one year—approximately 10 thousand quotes—of expertly labeled chat data, using its convolutional neural network architecture. At the time of development, an alternate approach considered was to use simple lexers/parses (ANTLR) to take advantage of a priori knowledge of the quote structure. Upon consideration of variation and volatility of quote formats and the respective combinatorial explosion, this approach was ruled out.

iii. Post-processing: Post processing is employed to perform three simple actions on the quote entities after extraction:

• • • Standardize synonymous terms, such as December and dec, to a consistent term.
    • Infer the year where implicit from the quote context. For example, a quote from July 2023 citing March implies March 2024; however, a quote citing December is presumably citing December 2023.
    • Categorize the type of quote from which named entities were extracted, such as a single month quote, a monthly put/call spread, or locational spreads.

b. Amazon SageMaker Studio, a fully integrated development environment (IDE) for ML, offers functionality to train and build models. The models are trained on the previous 7 years’ worth of data. Once the model is built, it is tested against a test dataset. If the results are satisfactory, the model is deployed with auto-scaling in multiple AWS Availability Zones.

c. Messages that are classified as non-quotes are further examined with a human in the loop, and Amazon SageMaker Ground Truth—which allows users to apply human feedback across the ML Lifecycle—and retraining are triggered if we realize that we are getting too many quotes in a different format.

5. Quote storage

a. The extracted quote, respective entities, and any failures are written to Amazon DynamoDB, a fully managed NoSQL database service, which can handle millions of requests per second. It can also handle changes in the structure of the underlying data, so in case we start receiving additional or different entities associated in a quote message, DynamoDB is able to accommodate the new entities.

6. Enriching the data

a. The quote and pricing data is further stored in bp’s data lake, where it gets enriched with weather data, historical trade information, market risk data, and other information. bp’s data scientists use this to train models to build new business strategies.

Diagram 3. Reference architecture

Conclusion

In this blog, we demonstrated how bp is able to ingest data from daily chat messages between traders and brokers, identify the quotes or price information from those messages, label the individual entities within a quote, categorize the type of quote, and store the information in a database. The solution is built using AWS serverless and event-driven services for orchestration, data processing, and storage. These services scale up and down based on usage, whether batch or near-real-time message processing. Furthermore, Amazon SageMaker offers an easy way to build, train, and deploy ML models complete with web-based IDE.

This adds business value in the following ways:

1. The data is further used to enrich the existing datasets in bp’s Trading data hub to build new trade and marketing strategies.

2. It serves as an additional source of market data.

3. It allows efficient and reliable price marking.

4. It provides more accurate risk metrics.

Future areas of growth identified include:

1. Automatic identification of statistical outliers and flagging for expert review

2. Expansion to other commodities and markets necessitating significant retraining

The solution can be expanded and also applied to use cases from other industries using chat programs for communication –

1. Airlines – Create ticket booking based on the travel details shared by customers with agents or airline ticket booking agencies.

2. Insurance – Create insurance policy template based on the insurance plan specifics, existing conditions and other preferences described by the insured.

For additional energy trading customer use cases, check out energy trading on AWS. If you have other ideas or feedback, please reach out to us or leave a comment. To learn more about how you can transform the core and build the future of your energy business, see AWS for Energy.