The High Energy Accelerator Research Organization in Japan Creates HPC Environment and Reduces TCO with AWS

The High Energy Accelerator Research Organization (KEK) in Japan leveraged Amazon Web Services (AWS) to construct a high performance computing (HPC) environment. This HPC environment provides services to analyze data obtained from synchrotron radiation beamlines and cryogenic electron microscopes for universities and private businesses. By automating the operational schedule of the infrastructure, KEK reduces their total cost of ownership (TCO) and scales on demand using AWS.

KEK is an inter-university research institute corporation conducting research into the laws of nature and the basic structure of matter. KEK uses particle accelerators to speed particles to near light speed and create high-energy states. While using this process to conduct their own experiments, researchers across Japan can also access this shared environment for accelerators.

By using quantum beams, like synchrotron radiation, positrons generated by particle accelerators, neutrons, and muons produced by proton accelerators, the Institute of Materials Structure Science (IMSS) can conduct comprehensive research into the structure and function of diverse materials. These diverse materials range from the atomic level to the macromolecular, bimolecular, and even individual levels. "In recent years, cryogenic electron microscopy has made major contributions in accelerating this research field and has given us new methods for the observation and structural analysis of proteins and other fragile bio macromolecules, in particular high molecular weight protein complexes," explained assistant professor Yūsuke Yamada from the Structural Biology Research Center, SR Science Division ll in IMSS.

KEK introduced a cryogenic electron microscope in March 2018. Since then, it has been working to provide an environment so researchers from universities and private businesses can share findings regarding the structural analysis of proteins. KEK then considered how it could provide the necessary computing resources to analyze large amounts of data. "Creating a computing environment for fast processing and analysis of data from measuring instruments such as microscopes is a significant hurdle in terms of budget and operation for individual researchers and research organizations. To this end, a suite of computing resources and applications for analysis has been created by KEK and we would like this to be used as a service by external researchers," explained Dr. Yamada.

KEK started to study cloud environments in November 2018, and selected AWS as their cloud service provider. "AWS is a world leader in cloud infrastructure, offering a wide variety of high performance computing infrastructure, including GPUs, as well as a suite of services such as clustering for the HPC environments that are essential to analysis processing. We felt that it could rapidly respond to future advancements in system infrastructure," explains Yamada.

Data output from synchrotron radiation beamlines and cryogenic electron microscopes is uploaded from KEK's on-premises system to the cloud. Once the data is in the cloud, researchers can access and experiment with the provided information. This process requires the cloud to store and manage data, ensuring that each user's data is isolated. KEK was able to achieve this and put in place the necessary security measures to meet these requirements with AWS.

KEK continued to develop their infrastructure and analysis applications. This led to KEK providing analysis services in April 2019. "We first carried out a proof of concept (PoC) using the cloud for the automation and optimization of analysis. The service itself is still strictly in the trial stage, and we are moving forward with improving the system architecture and processing flow," said Takuya Shinkawa from Fusic Co., Ltd., an AWS Partner.

KEK created a menu for various data analysis tasks on the cloud. This allowed each researcher to customize their needs by selecting the appropriate analysis task and accessing their own data. While researchers run these tasks, the required computing resources are all being allocated as needed in the background. These resources use a range of applications that are linked and automated to create a seamless user experience.

"We are implementing a program that incorporates APIs necessary for infrastructure development and control, and this is progressing smoothly because we are actively utilizing AWS managed services to make up for any deficiencies in requirements," said Shinkawa.

KEK’s hybrid cloud environment is connected by the Science Information NETwork (SINET). SINET is an information sharing network built and operated by the National Institute of Informatics (NII) as an academic information platform for universities and researchers throughout Japan. Because AWS had experience with SINET, the system was created smoothly and seamlessly connected overseas networks.

Even as improvements to the environment are made, the impact of these services is visible in various scenarios. This impact is seen in the flexible allocation and release of resources that is only possible in the cloud. At KEK, usage of the synchrotron radiation facilities is split into three periods per year, with operations lasting two months each period. Analysis and processing data is only required for about six months of the year, so the AWS service can be turned on and only used during that time. This means there are no fees or maintenance work when the service is off, which reduces the TCO.

Researchers are also using calculations that were previously unavailable remotely. Yamada said, "KEK holds training sessions on structural analysis as needed. Formerly, the venue required that enough PCs are provided for everyone. Now, by using a virtual desktop infrastructure (VDI) and a specific template to set up enough virtual desktops, participants can participate remotely."

KEK believes they have achieved their vision of a data analysis service used for experiments in structural biology. As the service moves towards full-scale operation, further improvements in analysis performance will be required. Currently, KEK uses Amazon Simple Storage Service (Amazon S3) to store the entire environment, and the organization plans to utilize Amazon S3 with their large datasets. Looking ahead, KEK is considering tuning Amazon Elastic Compute Cloud (Amazon EC2) instances and including parallel computation through AWS ParallelCluster and Amazon FSx for Lustre.

KEK aims to begin full-scale operations as soon as possible while resolving existing, individual problems. "Our initial aim was to help individual researchers in analysis of their own data, but looking forward, we hope to become an information platform that can share results from researchers both in Japan and abroad," said Yamada, as evidence of KEK's increased hopes with AWS.