National Bank of Canada Case Study

National Bank of Canada is one of Canada’s leading financial services organizations. Together with its subsidiaries, the company has more than CAD$219 billion in assets. The bank’s Global Equity Derivatives Group (GED) is a leader in providing stock-trading solutions that manage exchange-traded securities such as stocks, funds, futures, and options. GED offers trade-facilitation services to a growing group of global organizations.

National Bank of Canada’s GED collects and processes a fast-growing volume of stock-market financial data, such as trade and quote history information. However, the organization was having difficulty scaling its data-analysis platform using its on-premises IT environment. “Our traditional hardware environment and relational databases couldn’t keep up with our data growth,” says Pascal Bergeron, director of algorithmic trading for GED. “We needed a more scalable environment to help us analyze the data more efficiently so we could provide meaningful insight to the bank.”

GED also sought a more effective way of processing financial data. “We need to process and analyze both unstructured and structured data,” Bergeron says. “For example, we have a large number of log files that need to be analyzed against updated market data.”

Additionally, GED wanted a better performing analytical solution. “The application we were using wasn’t effective. We were only able to answer 10 percent of the questions we wanted to answer,” Bergeron says. “We also couldn’t process historical data, which we needed to do in order to get more context.”

The Global Equity Derivatives Group decided to use Cloudera, a comprehensive distribution of the Apache Hadoop open-source, big-data processing framework. “We selected Hadoop because it is scalable and supports both structured and unstructured data,” says Bergeron. To support its Cloudera and Hadoop solution, GED wanted to implement cloud technologies. “We knew it would be very easy for us to do a pilot in the cloud, spinning up virtual machines and shutting them down if the pilot wasn’t working,” says Bergeron. After evaluating leading cloud technologies, GED chose Amazon Web Services (AWS). “AWS was much more straightforward in terms of deployment, and it offered the most services,” says Bergeron. “AWS also supports Cloudera, which we had selected for our Hadoop distribution.”

GED then chose to work with TickSmith, a Canadian software provider specializing in big-data management and analysis technologies for financial data. “TickSmith has deep experience with the AWS Cloud and Cloudera,” says Bergeron. GED implemented the TickSmith TickVault application, which is based on Hadoop and accumulates and rebuilds all Canadian stock-trading activity and uses it for post-trade analysis and back-testing.

When installed on AWS, TickVault takes advantage of dozens of Amazon Elastic Cloud Compute (Amazon EC2) instances. The application also uses the Amazon Relational Database Service (Amazon RDS) for operating and scaling the GED database. In addition, TickVault relies on Amazon Simple Storage Service (Amazon S3) for data backup. GED can store up to 500 terabytes of financial data in TickVault, which consolidates trading-activity data in a central repository and allows GED employees to access data using visualization and API tools.

The GED was able to get up and running on TickVault in fewer than two weeks. “Building a Hadoop cluster in-house would have taken us several months, but we were using this solution much faster because of the AWS Cloud,” says Bergeron. “That was key for us, and it validated our decision to move to the cloud.”

GED can easily scale TickVault to consume and analyze financial tick data. “Using TickVault on the AWS Cloud, we can easily process and analyze hundreds of terabytes of trade data and historical quote data,” says Bergeron. “We would not have been able to do that without the scalability of AWS. And we can now look at data from 10 years ago if we need to, because we don’t have to rely on our on-premises solution anymore.”

The organization’s business analysts can now conduct post-trade analysis much faster than before. “The speed and performance of AWS are impressive. Data manipulation processes that took days are now down to one minute,” says Bergeron. “Post-trade analysis used to take weeks, so we wouldn’t do it that often. But now, relying on TickVault and AWS, we can look at both current and historical data in just a few hours.”

Now, GED can better serve its customers. “We have faster and better post-trade analysis capabilities using TickVault and AWS,” says Bergeron. “As a result, we can improve and optimize our trading operations and generate more revenue for National Bank of Canada. We can also use the data to see how we can trade at better prices for our customers.”

GED will be extending its use of TickVault to include compliance and risk data management. “We see this as just the beginning of our use of the AWS Cloud and TickVault,” Bergeron says. “We know we will be able to do much more with our financial data as our platform continues its fast growth.”