Recruit provides services that connect customers and clients such as "Rikunabi", "Jalan", and "SUUMO”. To operate services at different scale, data analysis for advertising data, log data and demographic data is indispensable. Therefore, the company's Data Management & Planning Office developed "Crois" as a data processing platform that can be used across internal organizations and internal projects.

'We initially developed Version 1 in 2017 as a prototype of our data processing infrastructure for ad technology. And then Version 2 in its current form was released in 2019 after continuous evolutions. In 2021, Recruit merged its operating and functional companies into a new organization. After examining the data analysis environments of each company, we found similar processing and requirements there, so we began operating this product as an infrastructure that can be used throughout the company" says Mr. Naoyuki Abe who is the Division Officer of the Data Technology Unit, Data Management & Planning Office, Product Development Management Office, Product Management Division.

“Crois" is equipped with a workflow function that does jobs analysis as a data processing infrastructure utilizing machine learning and other methods. The processes of data processing are catalogued as reusable "modules". Highly versatile modules can be used in across the organization regardless of the business unit. Users can also register their own modules for common use.

“The concept we’ve developed is a "data solution kitchen”. We regarded data as ingredients, modules as recipe and that the business organization cook a dish called a data solution based on the recipe. This solution eliminates the burden on data scientists and data engineers in the business organization with creating a separate analysis infrastructure, and at the same time allows scientists and planners, who do not have engineering skills, to focus on data analysis without the help of engineers.” (Mr. Abe)

Currently, "Crois" provides functions including module execution infrastructure, a container image catalog, a workflow engine, and an authorization management system, all of which can be manipulated via API or Web UI.

AWS Batch and Amazon ECS on AWS Fargate are used for the container execution infrastructure. AWS Batch is used for tasks which consume huge amounts of CPU, memory and disk resources, such as machine learning (ML) and for tasks that requires GPU instances. Amazon ECS on AWS Fargate is used for lightweight tasks such as querying a data warehouse that can scale quickly.

AWS Step Functions is used as workflow engine. By transferring all control of job execution to Step Functions and by separating it from “Crois”, Recruit was able to ensure scalability and fault tolerance at the same time.

“Since there were only a few development members at the beginning, the key point was to use managed services that could reduce the operational load. Considering the scalability of running hundreds of instances simultaneously, AWS Batch and Amazon ECS on AWS Fargate were the way to go, as the containers are simple to scale and do not impose an operational burden." Said Mr. Toshiaki Uno from Data Management & Planning Office, Product Development Management Office, Product Management Division, explaining the reason for choosing these AWS Services.

The authorization management system utilizes AWS Identity and Access Management (IAM) and AWS Key Management Service (AWS KMS) to ensure high level of security by issuing KMS keys that are paired with IAM roles on a root project basis.

“If data is stored without encryption, developers with access privileges can view the contents. Therefore, we took security and privacy into consideration by encrypting all intermediate files and access keys with AWS KMS so that even developers cannot see related information" said Mr.Uno.

The development and operation of "Crois" has been conducted entirely in-house by the company's development team since Version 1. The development team, which started with a few people, has grown to more than a dozen as the number of job execution has increased.

“During our development, we always discuss among team members what are the best practices out of the abundant AWS services to use and how to develop efficiently utilizing these tools. Whenever we have a problem, we consult with AWS solution architects". said Mr. Uno.

Launched in 2019, "Crois" has immediately been used in internal projects. Its performance has been gradually recognized and the number of jobs executed per day has now reached approximately 1,200. The utilization rate has been at 99.9%. “Since merging our operating companies in 2021, when the Data Management & Planning Office became a matrix-type organization that crosses the domain-specific unit for data strategy/planning in each business domain and the specialized unit to provide cross-disciplinary support over the domains, the number of requests to use “Crois” in their own businesses has increased as a result of its reputation spreading by word of mouth across the business organizations", said Mr. Masafumi Tsurutani from Data Management & Planning Office, Product Development Management Office, Product Management Division.

On "Crois," multiple large-scale processes can be executed simultaneously, supporting up to 30,000 CPU cores as of December 2022.

“Within Recruit, there are many departments that run large-scale Machine Learning (ML) workloads, with requirements for 1,000 or 1,500 parallel processing."Crois” is a scalable platform that can easily meet these requirements," said Mr. Uno.

Currently, "Crois" is being used in about 50 projects within Recruit. A typical use case for large-scale processing is the recommendation system for "SUUMO”, their real estate information website.

The "SUUMO" recommendation system had been using an in-house batch workflow for many years, which had accumulated technical debt and reduced development and operational efficiency. The introduction of "Crois" has increased development efficiency by using general-purpose modules and reusing modules created by users themselves, reducing lead time in development by 50%. We can also leave the management and operation of the compute infrastructure to AWS's managed services, which is expected to save up to approximately 10 person-months of operation per year," says Mr. Tsurutani.

Expectation on “Crois” is increasing more and more, as Mr. Abe says, “For business organizations, the biggest advantage is that they can concentrate on efforts to improve matching accuracy, which they should originally be committed to, without having to develop data processing infrastructure on their own".

“As the number of projects in use increases, it is predicted that more computing resources and a faster execution infrastructure will be required in the future. We will continue to work with AWS to evolve into a more scalable infrastructure," said Mr Abe.