RareJob Technologies Uses Amazon SageMaker for English Speaking Tests to Speed Up ML Model Development by More Than 10x

The RareJob Group is developing human resource education services for English, such as Smart Method, which is English conversation coaching for corporations, and RareJob English Conversation, which is an English conversation service that has 1 million members, 60 million
lessons, and 6,000 instructors.

In June 2020, the company developed the English speaking-ability measurement system PROGOS, which achieves automatic scoring using AI, and began providing the RareJob Speaking Test Powered by PROGOS (speaking test) service for individuals and corporations using PROGOS.

The speaking test measures the examinee’s ability to speak English. Examinees take the test online, and it takes about 20 minutes to answer the questions. Examinees are notified of the results in as little as 2–3 minutes, and they know their English proficiency based on international assessment standards. “The speaking test is a service released so that test-takers can grasp their own abilities and use them as a reference for their study plans,” says Kentaro Haneda, executive officer and chief technology officer at RareJob Technologies. “Scoring with AI provides a solution for evaluation blurring due to an individual rater’s experiences and senses, helping objective results to be announced quickly.”

The RareJob Group migrated its RareJob English Conversation service from another cloud provider to AWS in 2016. Since then, most of its services have operated on AWS.

“We decided to migrate to AWS because we judged that the bottleneck of our infrastructure would be a hindrance to our business,” says Haneda. “Since most of the lecturers in RareJob English Conversation are in the Philippines, we also evaluated AWS’s region in Singapore as well as its excellent support, technological development, and amount of technical information.”

PROGOS, which began development in 2019, also uses AWS. PROGOS consists of two services: the application side, which asks examinees questions and accepts answers, and the scoring side, which stores the voice data from examinees and scores it with AI. Both services use containers and serverless, managed services such as Amazon Elastic Container Service (Amazon ECS), AWS Fargate,  Amazon DynamoDB and AWS Lambda for the runtime environment. To loosely-coupling services on the application side and the scoring side, PROGOS uses Amazon Simple Queue Service (Amazon SQS).

“Since PROGOS is used for various services, we designed an architecture with scalability in mind,” says Haneda. “Also, we use Amazon SQS to achieve preprocessing launched at the same time that the user starts taking the test. Amazon SQS then quickly transitions to scoring after confirming the completion of the test.”

RareJob Technologies uses Amazon SageMaker to develop ML models for AI scoring, and AWS Glue and Amazon Athena to acquire data from examinees for model development. “At first, we were developing ML models on a local PC, but only one model could be launched at a time per developer, which created a bottleneck in development,” says Takuya Yamashiro, ML engineer in the EdTechLab at RareJob Technologies. Yamashiro and his team once tried Amazon EC2 and Amazon ECS, and they needed larger instances with a lower cost. “So, we focused on model training jobs using Amazon SageMaker. We decided to adopt it after seeing that instances with more GPUs and larger memory could be selected relatively freely and that ML could be trained at a lower cost by using things like managed spot training.”

The adoption of Amazon SageMaker significantly improved the efficiency of developing ML models. “We were able to reduce learning time by up to a quarter compared to model development using a local PC,” says Yamashiro. “It is now possible for one developer to simultaneously perform multiple learnings up to the service limit, and learning efficiency has improved by more than 10 times.”

The RareJob Group is also accelerating its use of AWS in terms of business. Employees transitioned to remote work due to the COVID-19 pandemic, and when the capacity of existing VPNs was insufficient, the company adopted AWS Client VPN in a short period of time to create a remote work environment for approximately 150 employees. The company also began using Amazon WorkSpaces for department personnel that handle sensitive management information and personal information. For the routine work of retrieving necessary data from cloud services and uploading files to document management tools, the company developed a system that automatically acquires data using AWS Lambda, reducing the workload by approximately 100 hours per month. “The mission of RareJob Technologies is to increase the value of the RareJob Group, and AWS has established a style that leads to solutions for issues while actively uncovering potential needs,” says Katsuaki Hirakawa, IT solution manager at RareJob Technologies.

The company also set up a department specializing in DevOps to improve development productivity as well as built a continuous integration and delivery environment to support daily infrastructure operations.

It has been about 7 years since the RareJobs Group began using AWS in 2016. Now, AWS is an essential foundation for business.

“In 7 years, the RareJob Group’s sales have almost tripled. In response, internal data use and systems integration have also accelerated, and the size of the engineering department has increased,” says Haneda. “Thanks to AWS, we were able to support a three-fold growth in sales.”

The number of examinees taking the speaking test using PROGOS is also rising steadily, and the number of examinees who choose automatic scoring using AI rather than manual scoring by graders is increasing.

“In recent years, usage by major client companies has been expanding, such as being used to determine the English proficiency of employees and for training new graduates,” says Haneda.

PROGOS is continually evolving while receiving feedback from examinees. Going forward, RareJob Technologies plans to improve the service to be easier to use while introducing a fraud detection system to prevent proxy test-taking. The company is also working on cost reduction using Amazon SageMaker Serverless Inference, which charges only a time inferences run..

The RareJob Group as a whole will promote the efficiency of back-office operations and implement measures to increase the value of products.

“We will actively provide new services while using customer behavior data and lesson data that we have accumulated so far in order to become an AI assessment company,” says Haneda.