How Dow Jones modernized their data storage

Listen now Voiced by Amazon Polly

Twenty years into their market data platform, Dow Jones, one of the world’s largest business and financial news companies, needed to modernize their database storage. The Dow Jones Market Data team was tasked with eliminating expensive on-premise SQL licensing costs and, if possible, shutter on-premises database servers and move to the cloud. 

Dow Jones wanted to move quicker than the estimated two years they expected it would take to complete a migration, with the staffing levels available on the Market Data team.

While the task of migrating market data across WSJ.com, Factiva.com, MarketWatch.com, Barrons.com seemed daunting, by partnering with AWS, Dow Jones was able to migrate their market data platform from an on-premise SQL solution to Amazon Aurora nearly two weeks ahead of schedule, while cutting costs by over 50%. Let’s see how they did it.

Racing the clock

The Dow Jones team initially tried to tackle modernization as an incremental effort. There were many other priorities competing for attention, and the modernization project was deemed “important” but not yet critical. However, the company was keen to lower their market data pricing. To achieve this, they would have to dramatically cut database licensing costs, either by changing to a self-managed, on-premises open source database, or by offloading the burden of database operations to AWS. 

This decision was complicated by time. From prior experience, they assumed they’d need to double the size of their team to manage the migration, and even then would need at least two years to complete the task. However, due to contractual changes, the market data team had just one year to migrate from on-premise SQL servers to their new, target database. 

To help determine the best course of action, the Market Data engineering team established five success criteria:

1. On-time delivery.
2. Minimal disruption to customers.
3. Accuracy of the data (i.e., news and data accuracy is the cornerstone of Dow Jones). 
4. Cost optimization (i.e., they didn’t want to significantly expand the team, because the whole premise of the exercise was to reduce spend, and in any case they wouldn’t have been able to get new staff up to speed fast enough).
5. Optimum resource utilization (i.e., they had a limited number of engineers and wanted to make efficient use of them).

Given these five criteria, the team decided that the optimum solution was to embrace the cloud with AWS. AWS offered Dow Jones a way to migrate all sensitive data from on-premise SQL servers to Amazon Aurora MySQL without disrupting existing applications, while ensuring the company could shave costs and make more efficient use of existing headcount.

How much money did they save? By moving to Amazon Aurora, costs were chopped by more than half when considering hardware, licensing, maintenance, rack space and power costs. Nor was that all they saved: “By moving much of the operational overhead to AWS, and completely eliminating the need to manage storage, Amazon Aurora frees up funds for innovation,” said Ramin Beheshti, Chief Product and Technology Officer, Dow Jones.

But how did they get there?

The process of modernization

According to Luke Sawatsky, Software Engineering Manager at Dow Jones, migrating from on-premise SQL server to Amazon Aurora was an “easy” process. First, Dow Jones worked with AWS on a Well Architected Review (WAR) to reduce complexity in their overall infrastructure.  

Dow Jones had been pulling and processing tens of thousands of messages per second from five different data providers. During a big market event, this traffic could spike 3-4x. Prior to modernization, the company relied on on-premises setup comprised of 16 database servers, with four masters mirrored across datacenters while the rest were split into distributors and subscribers, to process symbols for each company, prices, etc. All in, the Dow Jones entire market data platform ran on roughly 200 servers, all on-premise and split between two datacenters. 

Dow Jones consolidated some of these data feeds to simplify their architecture, but more was needed. By using the AWS Schema Conversion Tool and AWS Database Migration Service to migrate data continuously from on-premise SQL servers  to Amazon Aurora MySQL, as well as positioning NGINX servers between their data clients and MarketData services, Dow Jones was able to migrate all client traffic to their cloud APIs, reducing the time needed for the AWS Aurora MySQL migration to eight hours on a single Saturday, rather than months. The NGINX servers ensured uninterrupted service for clients by passing client requests through the legacy local servers to AWS. In the background, Dow Jones moved to a significantly simplified database architecture featuring a global cluster with one writer, five readers in Virginia and six readers in Ohio – closer to their data providers.  All with the capability to failover writers to opposite regions.

The architecture improvements didn’t end there. Post-migration, Dow Jones was able to use Amazon Cloudwatch, an observability service, to re-evaluate their infrastructure. They soon realized that the instances they were running on were too large for their needs: they were using just 3% of the CPU and 40% of the memory. They accordingly downsized their instances and/or reduced the number of instances, “which was a fairly simple exercise that Amazon helped a lot with,” said Sawatsky. After redeploying, they reduced their weekly spend more than 50%. They also started using more AWS services, including Amazon Elasticache, “deploying more things in the cloud, but reducing our costs.” 

The result? 

“To our leadership’s amazement, our team was able to deliver this project 12 days ahead of the schedule. There was absolutely no disruption of service to customers—we were able to continue to serve the market data accurately on all our websites and [print] publishing. We had no issues relating to quality as well: the data was all accurate throughout. There was a huge savings in the on-premises infrastructure that we are now repurposing in other parts of Dow Jones,” says Mona Soni, VP, Engineering.

More to come

To see how your organization can also save money while improving performance, please continue to join me as I highlight companies on their modernization journeys on to AWS. As you do, I hope you’ll also ask the question, what’s your plan for moving off Microsoft SQL Server? Or off Oracle? Or whatever old-guard technology keeps you from modernizing to better care for your customers?

As Dow Jones did, let AWS help you assess how your company can get the most out of cloud. Join all the AWS customers who trust us to run their most important applications in the best cloud. To have us create an assessment for your Microsoft SQL Server applications or all your applications, email us at WhatsYourModernizationPlan@amazon.com, and please consider joining the conversation using the #WhatsYourModernizationPlan hashtag. 

To learn more on modernizing Windows Server or SQL Server, visit Windows on AWS.