The primary review of program code utilizes the OSS AI tool which is "PR-Agent." The DBRE group, an organization across KTC that supports solving database-related issues, adopted PR-Agent to automate the primary review of pull requests (PRs) with the goal of reducing the time developers spend on code reviews.
Motoaki Hoshino of DBRE Group DBRE Engineer said, “We adopted PR-Agent after evaluating the fact that it is OSS, that it is compatible with multiple language models, and how easy it is to implement. We evaluated large language models (LLMs) for their integration with the AWS services used at KTC and their ability to be deployed with speed. As a result, we adopted Claude 3.5 Sonnet available on Amazon Bedrock."
When PR-Agent was adopted, prompt tuning was repeated so that the intended output can be obtained in response to the input. By managing prompts on GitHub Wiki, we make it easier for team members to modify prompts.
“With Amazon Bedrock's managed service, we were able to concentrate on developing prompts without the workload of connecting models, etc.” (Mr. Hoshino)
With the introduction of PR-Agent, it has been possible to reduce the load of PR creation and improve review accuracy.
“Creating a PR title and description, which used to take about 10 minutes, is now completed instantly. Since parts that are often overlooked in the review process are pointed out, the accuracy of code reviews has improved” (Mr. Hoshino)
PR-Agent, which specializes in code reviews, is also used by the company for tech blogs. Masaki Hirose of DBRE Group Principal DBRE Engineer said, “By introducing it to tech blogs, the review load has been reduced. In addition to checking typographical errors and grammar, it is different from conventional static analysis review tools that it is possible to review even what content should be included in order to increase the appeal of sentences and what should be done to make an article easy to understand, and we have heard from users that LLM feedback was helpful.”
Currently, based on the technology developed with PR-Agent, the company is working on a full-scratch schema review using generative AI. In the future, they plan to apply this to multiple projects to provide an even better developer experience.
The other use case, simplified data analysis, was initiated because there were no services suitable for small-scale cases. Generic Artificial Intelligence (AI) web applications lack necessary features, while specialized Artificial Intelligence (AI) applications with search-augmented generation (RAG) are highly functional but expensive. Therefore, we decided to build an analysis platform that is serverless and simple and does not require RAG. Takuya Uehira, from the Data Analysis Group, which supports data analysis across the entire company, said, "Initially, we thought that generative AI could be used to create SQL." In addition, there was a need to summarize questionnaire results etc. from the field, so we started building a minimal generative AI utilization platform. “Minimal” here means the minimum necessary architecture without waste. Even if it is minimal, it has performance that can handle batch input of about 140,000 characters,” he says.
At the time of development, Amazon Bedrock and AWS Lambda were adopted to create a serverless configuration to reduce maintenance costs and maintenance man-hours. The prompt generation process is an add-in, making it easy for users who are not familiar with programming to use it.
"The deciding factor for adopting AWS was that all the necessary features for building AI applications are serverless. Prototype development was completed in about 2 weeks. The feature is that users can arbitrarily change the AI mode, and since the number of modes increases simply by creating a prompt and placing it on Amazon Simple Storage Service (Amazon S3), new functions can always be used.” (Mr. Uehira)
The minimal generative AI utilization platform is currently being used for tasks such as searching through more than 100 data analysis dashboards to find those that align with the user's objectives, and analyzing the profiles of users who follow X accounts of various services and products.
“In the X follower analysis project, we were able to reduce the man-hours by around 50%. Since reference data and system prompts can be provided in advance, some users have mentioned that it has become easier to get better results with prompts.” (Mr. Uehira)
In the future, it is planned to expand applications and apply it to automatic generation of SQL and automatic data extraction in Japanese. Hideki Tomonaga, Principal Data Scientist and Assistant Manager, said, “We plan is to expand the user base to all employees. Therefore, it is necessary to raise awareness, and we will continue to disseminate information through tech blogs, etc.”
