Cost Savings of 20% and 8 Hours of Data Processing Saved across 500 Spark Jobs Using AWS Graviton2 Processors with Wealthfront

Wealthfront integrates smart saving and investing products to help the next generation of investors build long-term wealth. Founded in Palo Alto, California, in 2008, the startup has grown to manage more than $30 billion in assets for over 500,000 clients. It has been using AWS from the beginning. Now, Wealthfront manages over 500 data pipelines, running some of its preinvesting jobs. The large financial data processing workloads are on a combination of transient and persistent clusters that run continuously using Amazon EMR, an industry-leading cloud big data solution for petabyte-scale processing, interactive analytics, and machine learning. By using Amazon EMR to support its compute workloads, Wealthfront generates derived datasets for marketing needs, clickstream data, client financial data, and tax-related data.

Wealthfront has been improving its Amazon EMR infrastructure every year and wanted to take these improvements a step further by using AWS Graviton processors, which are designed to deliver the best price performance for cloud workloads running on Amazon EC2. Saving time is critical for Wealthfront because its customers depend on fast and efficient data pipelines to make financial investment trades. On busy trading days, all the available financial data needs to be processed before the following day’s trading can be computed. To better support this workload and accelerate data processing on Amazon EMR, Wealthfront migrated to AWS Graviton2 processors.

As part of the upgrade, the company also did some prerequisite work that included Scala and Spark version upgrades compatible with Amazon EMR 6.2. From 2019, Wealthfront made several upgrades before migrating to Amazon EMR 6.2 in February 2022. Once EMR 6.2 was implemented, the implementation of AWS Graviton2 processors took less than a month, and the rollout was completed in March 2022. “Because of the way the code is structured to launch Amazon EMR infrastructure, the upgrade went smoothly,” says Nithin Bandaru, data infrastructure engineer at Wealthfront. “We needed to make sure critical pipelines were functional and do some runtime analysis, and the entire upgrade went well.” Each year during re:Invent, an AWS conference for the global cloud community, the company produces innovative ideas to help improve infrastructure and efficiency and to further reduce costs. “AWS is awesome,” says Bandaru. “It has been a really nice experience working on AWS.”

“Using AWS Graviton2 processors, our pipelines run faster and cheaper, providing us with important benefits,” says Bandaru. “Running our data workloads faster means downstream jobs run faster. And because Amazon EMR is one of our main expenses, the profitability of the service was important to us.” Saving runtime was the main motivator for Wealthfront in using AWS Graviton2 processors. Each of the company’s 500 data ingestion pipelines ingests data every day. Across all pipelines, the company has saved 8 hours of data processing a day, amounting to a reduction of 5 percent.

Using AWS Graviton processors, the data ingestion pipelines run automatically in the background while engineers work on other tasks. When these pipelines run faster, the output for the day is improved and the whole operation is completed more quickly. “Any saved time increases our ability to start trading at the right moment,” says Arup Ray, head of data engineering at Wealthfront. “Accelerating our data processing is critical from a business perspective.” This faster runtime translates to more time for our automated investing algorithms to better manage our clients’ investments, and with the same instances running for a shorter duration, lower power consumption translates to a lower carbon footprint for the company.

Another major benefit of upgrading to AWS Graviton2 processors is the cost savings. “Using AWS Graviton2 processors provides, at a minimum, a 20 percent discount for the same jobs in the same amount of time compared with the old system,” says Bandaru. The company has seen performance reports of higher discounts as well. Each month, the company saves 20 percent by using AWS Graviton2 processors. Implementing the service on more pipelines will offer even more savings. “The main impact of using AWS Graviton2 processors is the cost savings,” says Bandaru. “As the underlying architecture of the processors changes, we will reap more benefits.”

Wealthfront currently runs around 95 percent of its data workloads using AWS Graviton2 processors. The company serves more than 500,000 clients, and this solution can scale to support over a million clients while still producing faster runtime. “We are able to serve more clients without incurring large additional data processing costs,” says Ray. “Using AWS, we’ve optimized our infrastructure to scale along with our company’s growth. And running AWS Graviton2 processors is a cost-efficient way of improving our elasticity.”

For more information about Wealthfront, including full disclosures, visit here.