Experian Cuts Time to Market for Analytics Products from a Year to Under 90 Days on AWS

Experian, a leading credit bureau and household name in credit monitoring, recently turned its focus toward giving business customers cutting-edge tools for understanding and managing financial data. Because Experian’s customers need to test, analyze, and manipulate huge quantities of financial data—3 PB—the company started out by creating a sandbox to provide the security layer necessary to handle sensitive financial data. But Experian went on to develop that idea even further, building a high-performance, award-winning suite of analytical tools called Ascend. Ascend gives 15 top financial institutions—such as OneMain Financial, a provider of personal installment loans—the ability to use big data, artificial intelligence, and machine learning in a secure hybrid-cloud environment to create and test new initiatives. To design this system, Experian needed a powerful computing platform with high-performance storage that could scale to hundreds of thousands of users while delivering an exceptional customer experience.

The credit bureau selected Amazon Web Services (AWS) as its computing solution for Ascend, incorporating Amazon Elastic Block Store (Amazon EBS), an easy-to-use, high-performance block storage service, to back up its critical customer data; Amazon EMR, the industry-leading cloud big data solution for processing vast amounts of data; and an array of other AWS services to manage big data, provide top-level security, and deliver computing power that adjusts to Experian’s needs. On AWS, Experian not only was able to launch Ascend faster than it would have by building its own solutions but also continues to expand the capabilities of Ascend: since Ascend launched on the AWS Cloud, it has seen exponential growth both in revenue—from $50 million in 2018 to more than $270 million in the first quarter of 2020—and in its customer base.

Experian’s credit reporting service captures over a billion events a day. With such a huge trove of consumer data, Experian is able to support 2,300 financial institutions in modeling new products and services and analyzing the future viability of everything from mortgages and auto financing to insurance policies. Even as Experian worked to provide its customers with flexible, powerful tools, the credit bureau was hemmed in by its legacy tech infrastructure. “Before Ascend, it took months to a year for us to roll out a product,” says Moied Wahid, vice president of Ascend platform engineering at Experian.

To deliver the tools its clients needed to develop their own financial products, Experian came up with the idea for Ascend, which the company determined had to be built in the cloud. For one, creating an on-premises system with the deep functionality of Ascend would involve building dozens of solutions from scratch—solutions that already existed on AWS. Using AWS would help Experian launch Ascend much faster. “We are in the business of creating value for our clients,” says Moied. “We are not in the business of creating solutions from the ground up.” What’s more, Moied firmly believes in five pillars for all Experian products: security, standardization, availability, resiliency, and elasticity. The cloud in general and AWS in particular offer a robust environment to help Ascend meet these criteria. All told, Experian would come to use 92 AWS tools in its Ascend architecture, and of these services, Amazon EBS would be the most consequential.

Experian uses Amazon EBS for various tasks. Some of the services Experian uses include Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon Redshift, and Amazon Relational Database Service (Amazon RDS). “And these are backed by Amazon EBS,” says Moied. All the front-end analytics applications supported by Ascend—such as SAS, RStudio, JupyterHub, H2O, and Tableau—run on Amazon Elastic Compute Cloud (Amazon EC2), backed by Amazon EBS general-purpose SSD (gp2) volumes, which give the platform the best balance of price and performance. Experian was also able to use Amazon EBS throughput-optimized HDD (st1) volumes, which cut the EBS cost in half. Just one Ascend product uses 2 PB of data, and Amazon EBS makes managing it easy. Backups using Amazon EBS snapshots were also a selling point, Moied notes: “We take a snapshot of customer data so that if something happens, we can restore that data faster.” Experian can restore its fully provisioned Amazon EBS volumes in minutes using the Amazon EBS Fast Snapshot Restore (FSR) capability. The feature also lets the company automate and standardize how it launches instances by taking snapshots for each virtual machine and replicating it, using that image to create multiple instances of the node.

Security, a crucial aspect of financial data, was an important factor in Experian’s decision to use Amazon EBS. Experian needed to be sure its customers—in industries ranging from banking and insurance to automotive—felt secure in the decision to store data in the cloud as opposed to in a data center. This is another area where Amazon EBS delivers, as its encryption enables both data-in-flight and data-at-rest security by encrypting data volumes, boot volumes, and snapshots using customer-managed keys via the AWS Key Management Service (AWS KMS).

Experian first delivered Ascend in 2018 and has grown the platform from one to six tools since that time, with more on the way. “The capabilities that we get from AWS enable us to build products quickly,” says Moied. “Our cycle time for new products has gone from years to less than 90 days.” All its customers, including business analysts, data scientists, bankers, and top financial institutions, can safely integrate their own sensitive data with Ascend. “Amazon EBS enables us to add more data and quickly expand the volume of the data,” says Moied. “That is a huge game changer for us in how we react to market conditions.”

For instance, the COVID-19 pandemic and the ensuing economic downturn presented Experian with an opportunity to put its AWS-backed architecture to the test by quickly creating a special product on the Ascend platform to help its clients understand how the pandemic was affecting the markets. “We wanted to create state-of-the-art dashboards for the COVID-19 pandemic initiative, and we launched it in less than 30 days,” says Moied. “That’s a great example of how we are using AWS capabilities and building business value for our customers.”

Handling all that consumer data and making it available to customers in a timely and useful manner are what Experian excels at—and that wouldn’t be possible without Amazon EMR. “We use Amazon EMR for cleansing the data, enriching the data, transforming the data,” Moied says, noting that Amazon Simple Storage Service (Amazon S3)—an object storage service that offers industry-leading scalability, data availability, security, and performance—was also critical. “We load data in Amazon S3, and our data scientists build dashboards that our customers use to see what’s happening in the industry and build their own consumer products.”

A key benefit of using AWS tools was the flexibility of separating computing and storage. “That way, we’re not running hot all the time,” says Moied. “In Hadoop Distributed File System [HDFS], we have to run the compute and storage together. On Amazon EMR, we use the same storage for different engineering workloads, different products. The same storage can be used for product one, two, three, or four. We have the ability to flex up and flex down. It’s a pretty huge benefit.” That flexibility gets passed on from Amazon EMR to Experian’s Ascend customers, offering them computing power when they need it; then, when processes are done, that power is scaled back to a minimum.

On AWS, Experian built a multifaceted platform, scaled its computing power, and decreased time to market for new products—and the financial services and technology industry is taking notice: the FinTech Breakthrough Awards named Experian’s Ascend Analytical Sandbox the “Best Overall Analytics Platform” in 2019. “Ascend is currently live in six countries around the world with further launches targeted for fiscal year 2021,” says Moied, referring to expansions planned in nine countries, including India and Singapore. Using a highly sophisticated and interconnected solution comprising dozens of AWS services, Experian’s sandbox simplifies the process of analyzing, testing, and collating big data to create innovative new services and insights, even in uncertain times, putting the power of financial data into the hands of its customers.