Reducing Latency by Over 98% Using Amazon Aurora and Zero-ETL Integration with Pionex US

Pionex US is one of the world’s first cryptocurrency exchanges to have built-in automated trading bots. To support these advanced capabilities, the company relies on a robust data architecture that is built around two data streams.

The first data stream runs from Amazon Aurora, a relational database management system that stores user account data and trading strategy parameters, to Amazon Redshift, a price-performance cloud data warehouse. Pionex US uses this stream to analyze user behavior, optimize trading strategies, and inform data-driven decisions to enhance the trader experience.

The second data stream runs from Amazon DynamoDB, a serverless, NoSQL database service that stores order books and trading records, to Amazon Redshift. This stream is where Pionex US analyzes transaction data, monitors market trends, and manages risk. By combining the two data streams with the analytics capabilities of Amazon Redshift, Pionex US gains valuable insights into user preferences, market dynamics, and system performance.

Pionex US wanted to use these near real-time analytics to offer its users up-to-date insights into market trends, empowering them to make more informed decisions. However, the company often saw data delays of up to 30 minutes, which made it challenging to track sudden price fluctuations or shifts in trading volume. To solve this challenge, Pionex US began exploring new integration options on AWS.

After a careful evaluation, Pionex US chose Amazon Aurora MySQL zero-ETL integration with Amazon Redshift. This integration seamlessly makes data available in Amazon Redshift, removing the need to build and manage complex data pipelines that perform ETL operations. Pionex US began to implement the integration into the first data stream in September 2023, followed by the second data stream in October 2023.

For the first data stream, Pionex US is facilitating near real-time analytics by using the integration to continuously replicate data from Amazon Aurora to Amazon Redshift. Because the data syncs in near real time without the need for traditional ETL processes, the company can query data and gain insights at a much faster pace.

“Amazon Aurora MySQL zero-ETL integration with Amazon Redshift facilitates data updates in seconds,” says Liao Wan Chun, senior director of data and analytics at Pionex US. “The query performance of Amazon Redshift makes it simple to query large datasets and support complex queries across databases and tables.”

The second data stream uses Amazon DynamoDB and Amazon Kinesis Data Streams, a serverless streaming data service, to manage and process transaction data in near real time. Using Amazon DynamoDB and the Amazon DynamoDB Streams feature, Pionex US can capture and emit transaction changes, which are then ingested by Amazon Kinesis Data Streams. From there, the data is streamed to Amazon Redshift for analytics and processing.

With the new data architecture in place, Pionex US has enhanced its data-handling capabilities and response speeds. Latency has decreased by over 98 percent, from 30 minutes to less than 30 seconds. “We can now query the latest data in near real time using Amazon Redshift, which was not possible before,” says Liao. “The improved efficiency and speed of integration have empowered us to respond more quickly to market demands and expand our data analysis and business decision capabilities.”

Pionex US can also integrate new data streams from Aurora faster and more efficiently. The zero-ETL integration has reduced the workload for creating pipelines by more than 80 percent, from half a day for two people to 30 minutes for one person. This has accelerated the onboarding speed for new data streams and improved access to up-to-date data throughout Pionex US’s systems.

In addition to performance improvements, Pionex US has realized substantial cost efficiencies. Deployment and operational costs have been reduced by more than 80 percent, and overall operational costs have decreased by 66 percent.

Pionex US is providing a more responsive experience for its users using the zero-ETL integration. With access to near real-time data, the company can effectively manage risk and quickly adapt to changing market conditions. This lets Pionex US maintain a stable, efficient, and secure trading environment for its users. By optimizing both data streams using AWS services, Pionex US can obtain query results and make decisions at a much faster pace. The second data stream processes and analyzes transaction data in near real time.

Pionex US plans to incorporate new features to enhance the second data stream even further, such as Amazon DynamoDB zero-ETL integration with Amazon Redshift. With this progressive approach, the company can keep its service responsive, relevant, and resilient.

“On AWS, we have successfully built a high-performance, low-latency, near real-time data analytics solution that supports our innovative development,” says Liao. “We are very satisfied with the results of this project and plan to continue expanding our use of AWS services and features.”