Crypto Trader Kronos Research Gains Speed and Scale by Running Workflows on AWS
By migrating its research workflow to AWS, Kronos Research is saving 4–5 hours on machine learning (ML) processing each day and is able to shorten its time-to-market. Kronos Research is a proprietary trading firm that uses ML and quantitative research capabilities to generate stable returns for crypto investors. The firm is using Amazon FSx for Lustre for high-performance file storage, Amazon ParallelCluster to manage high-performance computing clusters, and Amazon EC2 Spot Instances for low-cost compute power.
Moving our research workflow to AWS played a key role in making sure we could scale up to keep pace with market activity.”
Chief Technology Officer, Kronos Research
Using Machine Learning (ML) for Financial Trading Research
Many experts believe machine learning (ML) will become a determining factor for success in institutional trading. The ability to retrieve and process high volumes of data is key to capitalizing on the best market opportunities. Kronos Research is a trading firm with robust research infrastructure dedicated to building, testing, and deploying advanced ML models trained on large volumes of proprietary market data.
The founders at Kronos Research are Wall Street veterans. Their team of seasoned professionals applies quantitative trading techniques honed in the traditional finance industry to bring a more rigorous approach to crypto trading. The firm, based in Taiwan, offers investors stable returns in the volatile cryptocurrency market.
Staying Close to Crypto Exchanges on AWS
Kronos Research uses a mix of on-premises and cloud-based servers to meet its production requirements. The firm began using Amazon Web Services (AWS) in 2018, choosing the platform because of compatibility with crypto exchanges built on AWS. By leveraging AWS Regions and Availability Zones that are physically close to the exchanges, Kronos Research maintains the low latency required for high-frequency trading.
Crypto trading volumes hit new heights in 2020 and it was a breakout year for Kronos Research. However, server space and power requirements became constraints that hindered rapid expansion. The company began moving most of its research processes—the core of the business—to AWS in late 2020. “Moving our research workflow to AWS played a key role in making sure we could scale up to keep pace with market activity,” says Hank Huang, chief technology officer of Kronos Research.
Finding Optimal Services for High-Performance Computing
Since 2018, Kronos Research has relied on Amazon Simple Storage Service (Amazon S3) to store time-sensitive, granular market data that feeds its ML research models. After conferring with AWS to help solve its scaling challenge, Kronos Research began using Amazon FSx for Lustre for high-performance, low-latency file storage linked to Amazon S3 buckets. It also implemented AWS ParallelCluster to simplify management of high-performance computing (HPC) clusters and autoscale the resources needed to run ML research applications.
The combined use of AWS ParallelCluster with Amazon FSx for Lustre delivered the perfect balance of performance and elasticity to accelerate data downloads and ML processing. Previously, the firm’s research team required 3 hours to prepare data dumps from Amazon S3 onto their compute nodes. With the new AWS service, they’ve reduced that time to just 15 minutes.
Improving Time-to-Market with Shorter Research Modelling
Kronos Research has improved its time-to-market for ML modelling by switching to research processing on AWS. Formerly, internal teams had to compete for computing resources to run their research jobs, with wait times extending to 2–3 weeks. On AWS, they can run their jobs in parallel without waiting, which is saving at least 4–5 hours per day. The total time required to run computations, including any wait time incurred, has reduced by up to eightfold on AWS.
This has led to a significant increase in the number of ML models in use. Before switching to AWS for research processing, the firm was only able to update 100 models per week. Now, researchers are updating more than 800 models per week to enhance trading research. “We’re getting through the research faster, plus we’re able to update our existing models and deploy new ones a lot more frequently,” Huang explains.
Feeding the Innovation Cycle
Innovation has likewise become more efficient. Teams can now iterate and run experiments freely, without concern that the research cluster will be backed up. “Our teams can try something new within a couple of hours of coming up with an idea, and that’s been significant in keeping up with our momentum of innovation,” adds Huang. Productivity and job satisfaction are up, he reports, and company revenues have increased significantly in 2021.
Trading volume has also increased, and Kronos Research has become a top-five player on most major crypto exchanges. At the start of 2020, the firm recorded a peak day-trading volume of $1 billion. But as of mid-2021, the peak daily volume had already reached $22 billion. “Even our 30-day trailing average is now about $7.5 billion per day, which is a more than sevenfold increase over 2020,” Huang shares. In addition to sheer volume, gaining a foothold across numerous exchanges is critical for the firm’s growth. This is because the highly-scattered crypto market is unlike traditional financial markets, which traditionally revolve around a handful of big exchanges.
Controlling Costs while Expanding
Another advantage of doing HPC research on AWS is the flexibility of available resources to optimize performance while controlling costs. Kronos Research uses a variety of Amazon Elastic Compute Cloud (Amazon EC2) instance types including M5 for fast processing of mixed workloads, C5 for compute-intensive jobs, and R5 for memory-heavy workloads.
The business is also using Savings Plans to reduce costs and Amazon EC2 Spot Instances to add low-cost compute capacity as needed to research-related ML models. “We could increase the number of cores for our research without costs hitting the roof. This has been very helpful as we aggressively grow the business,” Huang explains.
Nevertheless, while cost is certainly a consideration for the business, it isn’t the overarching concern. “Research is our major investment and differentiator, and we view our AWS usage as an investment rather than a cost center. Basically, every dollar we pay AWS allows us to generate more. This has been a real change in mindset as our company expands,” adds Huang.
Migrating New Workloads for High Availability
Kronos Research is looking to add more AWS Regions to support growing business demand, and is also planning to migrate more production workloads onto AWS in the second half of 2021. Taiwan has recently experienced a couple of power blackouts that affected on-premises servers, and the firm has experienced firsthand the high availability and reliability it can expect on AWS after migration.
“We look forward to growing our business with AWS. The ease of launching instances without any premeditation takes a lot of the considerations away when we’re looking at how best to expand,” Huang concludes.
About Kronos Research
Kronos Research is a proprietary trading firm that delivers superior trading performance for crypto investors through its deep quantitative research capabilities. Research and machine learning are at the core of the business strategy.
Benefits of AWS
- Saves 4–5 hours per day for ML processing and updates
- Loads data in 15 minutes instead of 3 hours
- Accelerates time-to-market and internal innovation
- Scales operations 7x in less than 1 year
- Increases ML model throughput 8x
- Controls costs as business rapidly expands
- Optimizes instance types to suit workload requirements
AWS Services Used
Amazon FSx for Lustre
Amazon FSx for Lustre is a fully managed service that provides cost-effective, high-performance, scalable storage for compute workloads. FSx for Lustre file systems can also be linked to Amazon S3 buckets, allowing you to access and process data concurrently from both a high-performance file system and from the S3 API.
AWS ParallelCluster is an AWS-supported open source cluster management tool that makes it easy for you to deploy and manage High Performance Computing (HPC) clusters on AWS. ParallelCluster uses a simple text file to model and provision all the resources needed for your HPC applications in an automated and secure manner.
Amazon EC2 Spot Instances
Amazon EC2 Spot Instances let you take advantage of unused EC2 capacity in the AWS cloud. Spot Instances are available at up to a 90% discount compared to On-Demand prices. You can use Spot Instances for various stateless, fault-tolerant, or flexible applications such as big data, containerized workloads, CI/CD, web servers, high-performance computing (HPC), and test & development workloads.
Amazon Simple Storage Service
Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. Amazon S3 is designed for 99.999999999% (11 9's) of durability, and stores data for millions of applications for companies all around the world.
Companies of all sizes across all industries are transforming their businesses every day using AWS. Contact our experts and start your own AWS Cloud journey today.