Ticks by SIX platform on AWS provides high-precision historical tick data

Since 2008, SIX has been amassing a rich history of tick data across more than 360 trading venues. The volume of tick data—such as trades, order cancellations, or price changes—accumulates at a rate of about 10 TB per month and can triple in a day under volatile market conditions. Data exists in binary code, which had been stored on disks readable only by machine. Previously, a SIX engineer had to fulfill client requests manually by selecting files from the server and having the customer parse them to create usable outputs, a process that took days.

“We have to store enormous volumes of data in a clever, efficient way, and then be able to make it accessible for fast retrieval,” says Henk D’Hoore, head of product development, financial information, at SIX. “To do this with homegrown technology is a very steep and costly learning curve.”

SIX enlisted the help of RoZetta, which integrated best practices from its decades of experience with other financial customers. To develop a ready-made solution for SIX, RoZetta ingested and normalized hundreds of terabytes of tick data, including 10 years of history and more than 30 million instruments. Working alongside AWS, RoZetta developed the Ticks by SIX pilot in 3 months, including data transformation and enhancement pipelines, data governance controls, and a secure, high-performance framework. “We knew that speed to market was very important to SIX, and our DataHex solution was well placed to make this happen,” says Gary Mah, chief operating officer at RoZetta. “AWS was also able to provide financial support, which minimized the upfront investment. This meant SIX could integrate a greater volume of data into the platform, thereby enhancing the market appeal to SIX’s global customer base.”

To meet client requests, SIX can pull large, customized historical datasets in hours, a process that used to involve engineers and could take days. Sales teams can generate sample files for prospective clients within seconds. “Flexibility is one of the key value propositions for SIX,” says Mah. “Customers can select data down to the field level or listing level by market and extract the exact information that they want. That minimizes the amount of data wrangling from their end.” For example, a bank or hedge fund can obtain listings only for the American markets or for certain sectors like oil and gas and limited only to the attributes that interest them.

Clients use data through Ticks by SIX for a variety of use cases. They can run transaction cost analysis to meet regulatory requirements, quickly pulling a full record of market liquidity, quotes, and trades over a particular period and matching the information to the timestamps of orders. Clients can also use Ticks by SIX for backtesting, using tick histories to simulate trades and strategize for future investments.

To power Ticks by SIX, DataHex captures multiple petabytes of tick data and billions of orders daily as raw, near real-time market feeds. It stores the data at scale in Amazon Simple Storage Service (Amazon S3), an object storage service offering industry-leading scalability, data availability, security, and performance. SIX incurs costs only from the storage it uses and significantly reduces the cost of egress for customers who need to transfer or share data in the cloud.

In response to client requests, Ticks by SIX extracts and transforms the data into a user-friendly CSV format. For flexibility in the development and deployment of the application later, RoZetta uses Amazon Elastic Kubernetes Service (Amazon EKS), which automates Kubernetes cluster infrastructure management. RoZetta offloaded the operational burden of managing database capacity through the use of Amazon Aurora, a relational database management system built for the cloud. “Using AWS, we can scale easily up and down on storage and compute with durability and performance,” says Mah. “We are also able to dynamically respond to the evolving business requirements, reducing processing extraction times for the huge datasets we’re dealing with.”

Using Ticks by SIX, customers can receive data within 2 hours of market close, subject to exchange embargo times. Additionally, they have access to precalculated analytics such as time bars, which aggregate tick-level data into a time-based summary of all market activity within 1 minute or other defined timeframe.

SIX plans to offer additional advanced analytics capabilities to help clients better understand risk or to measure, for example, the sensitivity of an option’s price to market fluctuations. The company is also considering a self-service offering in which customers could build their own services using tick data by SIX.

“Overall, we have properly built a ready-to-use core capability as part of our market data strategy,” says D’Hoore. “The speed at which the service works and lets us store and retrieve data when we want to, or when the customer wants to, is unparalleled. We wouldn’t have been able to do it in the time that we have without working alongside RoZetta and AWS in the background.”