Digital Asset Research Ingests 1 Billion Crypto Trades Daily and Protects Data with SingleStoreDB Cloud and AWS PrivateLink

Digital Asset Research (DAR) implemented AWS PrivateLink with SingleStoreDB Cloud to speed up analytics to ingest 1 billion crypto trades per day, automate manual processes, and better protect its customers’ data. PrivateLink with SingleStoreDB Cloud helps DAR quickly and securely connect private services in its own VPC to SingleStoreDB Cloud.

As cryptocurrency continues to grow in popularity, companies like Digital Asset Research (DAR) are becoming increasingly important to businesses seeking to make crypto investments. DAR, an independent provider of research and analysis on the factors that impact crypto investing, delivers trustworthy digital asset pricing, market data, research, and guidance to financial institutions across the globe. “Getting involved in trading crypto can be complicated, and our methodologies simplify it by, for example, using pricing engines that filter out market-manipulating crypto trades and vet crypto exchanges,” says Michael Zimberg, chief technology officer at Digital Asset Research. DAR offers solutions such as the DAR Exchange Vetting Methodology, which gives crypto market participants a transparent view of the process used to evaluate the largest digital asset exchanges by reported volume.

Because of the high volume of trading, DAR needs to process and scale data as quickly as possible. “On an average day, we need to ingest and normalize about 200 million trades and calculate pricing at many intervals including 400ms,” Zimberg says. However, DAR could only process around 10 million trades daily using its existing MySQL relational database management system on. “We were having performance issues during periods of high-volume trades, and we would sometimes lose trades,” Zimberg says. “If we don’t have trades, we can’t create prices, which means we fail to deliver for our clients.” Additionally, as data volumes increased, it became more important for DAR to protect sensitive client data from cyberattacks. “Data protection is critical in the financial services industry, and we needed to ensure that our customers’ data couldn’t be manipulated by outside users,” says Zimberg.

To increase performance and scalability and enhance security, DAR turned to SingleStore, an AWS Data & Analytics Competency Partner that provides an all-in-one database for operational analytics and applications requiring fast data ingestion and high-performance queries. SingleStore offers SingleStoreDB Cloud, a distributed relational database that runs on Amazon Web Services (AWS). “By running our solutions on AWS, we get elasticity and scalability at a lower cost, along with tiered storage in the cloud,” says Manish Kumar, product manager at SingleStore. “SingleStoreDB Cloud also enables real-time analytics by ingesting high volumes of data quickly and making it available for querying.”

Recently, SingleStore began offering AWS PrivateLink with SingleStoreDB Cloud, a solution that integrates natively with AWS PrivateLink. AWS PrivateLink offers private connectivity between providers of SaaS applications on AWS and consumers, virtual private clouds (VPCs), or on-premises networks without exposing traffic to the public internet. Using PrivateLink with SingleStoreDB Cloud, customers can quickly and securely connect private services in their own VPC to SingleStoreDB Cloud. “Security services such as AWS PrivateLink should be available to everyone, and our solution automates the setup of AWS PrivateLink and connects it to applications running their VPCs in minutes,” Kumar says. “This simplifies the experience of connecting applications to PrivateLink.”

AWS PrivateLink with SingleStoreDB Cloud also provides high availability for global SingleStore customers through AWS Availability Zones. “Resiliency and high availability are critical for financial services companies, and AWS offers that,” says Dushyant Jha, technology partnerships manager at SingleStore.

By implementing SingleStoreDB Cloud, DAR can ingest larger volumes of trading data, speeding up time to analytics for customers. “Using SingleStoreDB Cloud on AWS, we were able to massively scale our pricing engines, so we can now run 1,000 ETL pipelines per second,” says Zimberg. “On a high-volume day, we can easily ingest 1 billion trades, and the solution just works.”

Because it can analyze more trades now, DAR is delivering crypto pricing data such as reference data and raw blockchain data to customers faster. As a result, customers can more quickly find data on crypto investments. Additionally, DAR spends less time on administrative tasks because SingleStoreDB Cloud centralizes online analytical processing (OLAP), online transaction processing (OLTP), cache DB, and extract, transform, and load (ETL) processes for integrating data. “We have more flexibility in terms of querying trade data because OLAP, OLTP, and ETL processes are in the same environment,” says Zimberg. “That removes a lot of manual work I had to do myself. I don’t think we could have the applications we have without SingleStoreDB Cloud, especially because we have a pretty small team.”

DAR integrated quickly and securely with PrivateLink through PrivateLink private connectivity with SingleStoreDB Cloud, which helps the company protect sensitive financial services data for customers. “With PrivateLink connectivity with SingleStoreDB Cloud, we can easily maintain our list of which crypto exchanges are vetted versus our watch list while remaining confident that our data can’t be manipulated by someone,” says Zimberg. “This is becoming more critical for us as our data becomes used more often in financial services products.”

“From a security perspective, it’s important for companies like DAR to provide end-to-end protection for data,” Kumar says. “We already use encryption at rest by default for our customer workloads, but network security is also key. That’s where PrivateLink automation comes in. It gives DAR confidence that it’s running protected data workloads, now and in the future.”