Contribution: Introduction of a carbon credit evaluation system using Generative AI by Osaka Gas Co., Ltd. (First half)

This article is a contribution from Mr. Kazuya Okada, the leader of the Carbon Credit Development Unit in the Future Value Development Department of Osaka Gas Co., Ltd., regarding the company’s efforts to use generative AI for a carbon credit evaluation system. It will be introduced in two parts, this being the first part.

In the first half, we will explain the background and results of using generative AI to evaluate carbon credits. In the second half, we’ll introduce the continued development on AWS and the business implications.

The second half of the article has also been published, so please be sure to check it out.
“Contribution: Introduction of a carbon credit evaluation system using Generative AI by Osaka Gas Co., Ltd. (Second half)”

1. Introduction

Osaka Gas aims to achieve carbon neutrality by 2050. Toward this goal, the company has set a plan to contribute to a 10 million tons reduction in CO2 emissions by fiscal year 2030. Osaka Gas is working towards the realization of a sustainable society through various initiatives, including expanding the use of renewable energy, improving energy efficiency, and introducing new technologies. In addition, the utilization of carbon credits is positioned as an important option. Carbon credits are attracting attention as a means of offsetting (offsetting) CO2 emissions that cannot be fully reduced by renewable energy (hereafter, renewable energy) and energy-saving efforts. The company is promoting the acquisition and utilization of carbon credits through investments in domestic and international projects.

Accordingly, this is strengthening the company’s efforts towards the establishment of a low-carbon society and the realization of a sustainable future.

Figure 1: Our Carbon-Neutral Vision (Details: https://www.osakagas.co.jp/en/whatsnew/pdf/230309_4_1.pdf)

2. What are Carbon Credits?

Carbon credit is a mechanism to certify the amount of greenhouse gas emissions reduction in a tradable form. Companies and organizations implement greenhouse gas emission reduction projects and issue credits for the achieved reductions. These credits can be purchased by companies that are unable to meet their reduction targets, to offset their emissions. Some carbon credits are legally recognized (*1), and are widely utilized under the Paris Agreement. It is officially operated in many countries and regions, such as the J-credit Scheme in Japan. A global trading market has been formed to support international emissions reduction efforts. Accordingly, this allows companies to flexibly achieve their own emissions reduction targets and contribute to global warming countermeasures.
*1: For details, see How to Set Carbon Offsets in Japan (Guidelines) 4th Edition (Japanese)

Figure 2: What is carbon offsetting (Source: How to Set carbon offsets should be done in Japan (Guidelines) 4th edition – Japanese)

3. Quality of Carbon Credits

There are important issues regarding the quality of carbon credits, such as “additionality,” “permanence,” “leakage,” and “impact on local communities.”

• “Additionality” means that the project achieves additional emissions reductions that would not have occurred without the project, under existing regulations or normal economic activities. For reliability and environmental effectiveness, proof of reduction using new technologies or methods is required.
• “Permanence” refers to the reductions being maintained not just temporarily, but over the long term, requiring continuous monitoring and risk management.
• “Leakage” is the problem where emissions reductions in a specific region lead to increased emissions in other regions, highlighting the need for a broader, cross-regional perspective and collaboration.
• “Impact on local communities” means minimizing the project’s impact on the local ecosystem and residents’ living environment, requiring consultation with and incorporation of local opinions. Benefit sharing and environmental education are also important.

However, confirming the assurance of these factors requires a high level of expertise.

4. Issues with Low quality carbon credits

When utilizing carbon credits, a serious issue arises if the quality of the credits does not meet a certain standard, it becomes unclear whether actual greenhouse gas emissions reductions have been achieved. If the credit quality is low, there is a risk that the reduction was not actually achieved or the reduction effect is only temporary.

Furthermore, intentionally or unconsciously offsetting with low-quality carbon credits can be considered “greenwash,” where a company claims to be taking environmental action while not actually implementing effective measures. This can severely damage the company’s credibility and reputation.

5. Carbon Credits Evaluation System Utilizing Generative AI

To address the issues regarding carbon credit quality, our company has developed a carbon credit evaluation system using generative AI on AWS. This system analyzes carbon credit project plans using generative AI and evaluates the quality of carbon credits. Specifically, generative AI thoroughly examines the publicly available carbon credit project plans and assesses the risks based on important evaluation criteria such as “additionality,” “permanence,” “leakage,” and “impact on local communities”.

Figure 3: Carbon Credit Evaluation System Using Generative AI

This system was developed with the aim of greatly improving the transparency and reliability of the carbon credit market. By utilizing this system, companies and investors can select high-quality credit and achieve effective emission reductions. It can also avoid greenwash risks and improve a company’s environmental performance. Ultimately, we expect that sustainable global warming countermeasures will progress through the introduction of such generative AI technology.

6. System architecture overview

The project plan documents for carbon credits are published in PDF format on the internet. We perform optical character recognition (OCR) on the published project plan documents, and store the extracted text data in Amazon S3. The text data stored in Amazon S3 is then used to perform risk assessment using two types of generative AI (*2). The risk evaluation process semantically judges how well each project plan document meets the evaluation criteria, taking into account the surrounding context and nuance. At that time, in order to confirm that no halting has occurred, we confirmed evidence on which text information was used to make judgments, and obtained validation. The raw data, such as risk evaluation results, is stored in Amazon S3, and difference information when there is a change in the project plan is stored in Amazon Dynamo DB so that differences can be checked.

*2: Utilize two types: GPT-4o by Open AI and Claude 3.5 Sonnet by Anthropic, which can also be used in the Tokyo region from August 2024.

As of September 2024, Anthropic’s Claude 3.5 Sonnet, OpenAI’s GPT-4o, Google’s Gemini 1.5 Pro, etc. have become widely adopted in the market. Generative AI is evolving rapidly, and it is anticipated that high-quality underlying models will continue to be released in the future. In this situation, it is important to design the system in a loosely coupled manner, where the base model and the risk assessment processing can be readily updated to the latest models. AWS services can be architected in a flexible manner and have a wealth of managed services. This allows us to focus on application development, so we think it’s a very good platform.

Figure 4: AWS Architecture Overview

7. Discussion on the Evaluation Results of this System

In order to evaluate the accuracy of this system, we had our randomly selected carbon projects evaluated by a specialized rating agency.

Assuming that the evaluation results were positive, we carried out verification (*3) with Claude 3.5 Sonnet and GPT-4o to what extent the prediction results of this system match the ratings company’s evaluations.

The evaluation method was to use the rating agency’s evaluation as the ground truth, and verify how well each generated AI model matched the rating agency’s evaluation. The randomly selected project plan documents contained 200 risk evaluation criteria.

*3: In this verification, prompt tuning is the result of performing prompt tuning so that optimal results are produced for each basic model.

Below are the risk assessment results. (The numbers to the right of the “+” indicate the number of criteria partially matched.)

1. Rating Agency
1. Criteria met: 112 (108+4)
2. Criteria not met: 88 (1+87)
2. Claude 3.5 Sonnet
1. Criteria met: 109 (108+1)
2. Criteria not met: 91 (4+87)
3. Accuracy: 97.5%, Recall: 96.4%, Precision: 99.1%
3. GPT-4o
1. Criteria met: 112 (107+22)
2. Criteria not met: 88 (5+66)
3. Accuracy: 86.5%, Recall: 95.5%, Precision: 82.9%

Taking Claude Sonnet as an example:
Accuracy: (108+87) ÷ (108+4+1+87) = 97.5%
Recall (proportion of actual positives correctly identified): 108 / (108+4) = 96.4% Precision (proportion of predicted positives that are correct): 108 / (108+1) = 99.1%
Below are risk assessment results using each generated AI.

Figure 5: Risk Assessment Verification Results for Each Foundation Model

Despite the generally low output accuracy of generative AI, by leveraging domain knowledge in the carbon credit domain and high expertise in generative AI, we were able to achieve highly accurate results in this case.

8. Conclusion

We plan to further advance the quality evaluation of carbon credits and improve reliability and transparency by continuing the development of this system in the future. We believe that this will enable companies and investors to select carbon credits based on more accurate information, leading to avoiding greenwash risks. Additionally, by utilizing the technology of this system as expertise and know-how for carbon credit trading and evaluation, we aim to support market participants in rapidly and accurately assessing and trading appropriate carbon credits.

In this article, we introduced our generative AI-based carbon credit evaluation system. Please refer to the latter half of “Introduction of a carbon credit evaluation system using AI generated by Osaka Gas Co., Ltd. (Second half).”

Author

Kazuya Okada (AWS Solution Architect Associate)
Leader, Carbon Credit Development Unit, Future Value Development Department, Osaka Gas Co., Ltd.

At Shionogi Pharmaceutical, he planned and developed new businesses. He established AdvanSentinel, a joint venture with Shimadzu Corporation, and developed and launched a service for predicting the spread of COVID-19 and other outbreaks, winning the Special Award from the Japan Open Innovation Awards Selection Committee and the Minister of Health, Labour and Welfare Award.
He then worked as a consultant in the healthcare sector at Deloitte Touche Tohmatsu Consulting.
He joined Osaka Gas in 2022 and has been leading the planning and development of new environmental businesses, primarily focused on carbon credits. In June 2024, he was a panelist at a press conference held by the Ministry of Agriculture, Forestry and Fisheries on private-sector Joint Crediting Mechanism (JCM).

This article was translated by AWS Professional Services Riho Matsui, and Solutions Architect Satoshi Aoyama.