Scaling Historical Trading Data Storage Using Amazon S3 with BMLL

BMLL provides high-quality and granular historical data for financial markets. Customers use the data for pre-trade analysis, generating trading strategies, testing ideas, running market simulations, and more. Traders also depend on the data for post-trade analytics to help gauge effectiveness and identify ways to lower transaction costs.

BMLL’s dataset grew to more than 20 PB and is projected to reach 50 PB by 2030. Because of that growth and customer demand for more granular access, the company wanted to optimize how it stored, processed, and delivered data. “The demand for getting this right was incredibly high because huge investment decisions are based on that data,” says Elliot Banks, chief product officer at BMLL.

To sustain its growth while optimizing costs, BMLL uses the Amazon S3 Intelligent-Tiering (S3 Intelligent-Tiering) storage class, which automates storage cost savings by moving data when access patterns change. By using a more efficient data storage tier with almost no manual overhead, the company improves scalability and cost efficiency as its volume of data grows.

Instead of hosting servers on premises and increasing staff and resources to manage the data, the company can rely on Amazon S3 as a fully managed service. “We’re not only packaging the data and shipping it,” says Banks. “We’re making it accessible and available in different products, and our customers trust us with it. Amazon S3 underpins that as the storage layer for all our data across all our products.”

Using AWS to store data, BMLL can scale seamlessly and provide rich insights to customers to help them understand the markets and make the right decisions. The company can also efficiently process data and make it available in a clean, ready-to-use format. Customers don’t need to prepare the data for analysis, and they can access all the historical data quickly. “High-quality historical data is our number one differentiator,” says Leonie Alsop, head of marketing at BMLL. “Our end users can rely on our data, so they have much less work to do in terms of cleaning up issues or data formatting.”

Customers can also use BMLL’s vast dataset to train customized models. As a result, customers save the months it would ordinarily take to curate a high-quality dataset and improve the quality of investment decisions. BMLL can support all this without constantly spinning up on-premises servers to respond to changes in demand. “If we had 10 PB of data and two on-premises data centers, we’d need to send a whole team there every week to manage them,” says David Robinson, chief technology officer of BMLL. “It would be a full-time job.”

The company copies its backup data by using Amazon S3 Replication, which replicates objects between buckets across different AWS Regions. BMLL stores these backups in Amazon S3 Glacier (S3 Glacier)—long-term, secure, durable storage classes for data archiving. “Given the scalability of AWS, we can store more and more data, with almost no need to do anything manually,” says Robinson. “It just works.”

BMLL saves 500,000 dollars per year by using S3 Intelligent-Tiering. And it saves an additional 3 million dollars yearly by using S3 Glacier for long-term archiving and backup of infrequently accessed data.

The company orchestrates its data pipeline with Apache Airflow, creating a series of processes that retrieve data and pass it to the next step automatically. BMLL used to run Apache Airflow itself. Now, it automates those processes by using Amazon Managed Workflows for Apache Airflow (Amazon MWAA), secure and highly available managed workflow orchestration for Apache Airflow.

This gives its operations team detailed visibility and off-loads most of the work so that engineers can focus their time on more important work. “We don’t need to get involved in the underlying details of how Apache Airflow orchestration works,” says Nigel Kettlewell, lead technical architect at BMLL. “It just runs every day, and the operations team can see it and interact with it.”

Reliability is crucial to maintaining the trust of customers, and BMLL is delivering reliable services with the built-in resilience of AWS. The company uses Amazon S3 Access Points, which businesses can use to easily manage access for shared datasets on Amazon S3. Now, it can help a virtually unlimited number of customers access data.

BMLL fortifies security and limits permissions by using policies for buckets and identity and access management. The company also helps mitigate cybersecurity risks by using Amazon S3 Object Lock, data protection from ransomware events with object-level immutability to protect objects from accidental or malicious deletions and overwrites.

On AWS, BMLL has built a foundation that empowers it to scale efficiently while delivering historical financial data of excellent quality to its customers. By adopting new and innovative features from Amazon S3, BMLL can continue optimization as it scales. To support its ongoing growth and provide deeper insights to customers, the company plans to explore Amazon S3 Tables, which optimize query performance and cost as an organization’s data lake scales.

“Our customers have confidence in our products, and we trust in AWS as the foundation for those products,” says Robinson. “Using AWS exceeds almost anything we could do internally.”