Nasdaq Migrates Its Growing Data Warehouse to a More Modern Data Lake Architecture

While most people know Nasdaq for its equities exchange, Nasdaq is also a technology and services provider to more than 120 different exchanges, regulators, and post-trade entities in more than 50 countries worldwide. Every evening, Nasdaq receives up to 60 billion records that must be loaded for billing, reporting, and other processes before the markets open the next morning. As that record volume increased over time, Nasdaq opted to migrate to a data lake solution in order to scale the ingest process and query data in a more parallel manner. Nasdaq moved its entire ingest process to Amazon Web Services (AWS), writing the data to Amazon Simple Storage Service (Amazon S3).

From there, Nasdaq chose Amazon Redshift Spectrum to enable its queries to run faster than in its existing data warehouse.  

According to Robert Hunt, vice president of software engineering for global surrounding systems at Nasdaq, “We have one billing process that went from taking about 40 minutes to now running at 4 minutes—a 90 percent improvement in that one process. But across all of the processes, we’re seeing 60–70 percent improvement across the board.” 

View the video below to learn more about how Nasdaq leveraged the AWS Data Lab for expert advice and recommendations to achieve its migration goals.

Watch Nasdaq’s session from re:Invent 2019 and learn more about AWS for Financial Services.