Migrating to an Event-Driven, Serverless Architecture
In 2013, CreditorWatch realized it had outgrown its single on-premises server. It needed a stable, scalable solution to support its rapidly growing client base and the increasing volume of data it was processing. The company decided to migrate its online platform to Amazon Web Services (AWS) Cloud for scalability and stability.
The next phase of expansion came in 2017, when major corporations came knocking. Enterprises, including Australia’s Big Four banks, were looking to leverage CreditorWatch’s rich dataset and granular analytics capabilities. To drive efficiency and boost agility, the company then pivoted to an event-driven serverless architecture, splitting its single-tier platform into 22 microservices.
The ease and affordability of setting up a multi-tenant architecture on AWS allows CreditorWatch to continue scaling the business without limits. “AWS enables a small player like us to tender with big organizations without driving up overhead due to the shared-service nature of the platform,” says Joseph Vartuli, chief technology officer at CreditorWatch.
Processes 400 Billion Data Points in 8 Hours
CreditorWatch uses
Amazon Kinesis to collect real-time streaming data and more than 250
AWS Lambda functions to execute the entire event-driven process without manual intervention.
AWS Database Migration Service is deployed to extract data from 26 sources. Data is then populated in a compressed format to an
Amazon Simple Storage Service (Amazon S3) data lake. The data processing magic happens in the
Amazon EMR big data platform, and reports are generated to assign businesses a risk score.
The scores take events into account as they happen, with real-time updates including factors that could impact a partner’s ability to repay debt. The volume of data that CreditorWatch processes—some 400 billion data points—means their analytics engines can evaluate businesses across multiple geographic and demographic spaces. It can then output an informed assessment that shows how a business fares compared to peers in its industry.
Data dumps for all 400 billion data points are completed in just 8 hours once a month. Smaller, live updates are streamed and automatically performed as changes occur to the data following that, until the next major data migration 30 days later. CreditorWatch spins up huge Amazon EMR clusters for this monthly data transformation, then shuts down the clusters after the scheduled job. It only pays for the resources used, in line with AWS’s pay-as-you-go pricing model. In an on-premises setup, costs would be 30 times higher, as high-capacity servers would need to run for the entire month.
Machine Learning Models Enrich Payment Predictions
In 2019, CreditorWatch began running machine learning (ML) algorithms in Amazon EMR to dig deeper and find additional factors that could lead to insolvency in the next 12 months. The ML models were then backtested against actual customer data over the past 5 years to improve accuracy and generate payment predictions.
The business employs just four people on its data analytics team to serve its more than 55,000 customers. Vartuli says, “If we didn’t have tools like Amazon S3 or Amazon EMR clusters at our fingertips, there’s no way we could carry out these ML interactions with such a lean team. AWS provides the tooling and code-based infrastructure that allows us to further monetize our product.”
Proactive Support from AWS Experts and Community
Assistance from
AWS Business Support specialists has been critical in helping CreditorWatch build and optimize its data pipeline. Developers consult frequently with AWS solutions architects throughout the product lifecycle, proactively discussing concepts before implementation and troubleshooting when issues arise. The AWS team participated in determining the right amount of RAM, processing power, and storage required to run CreditorWatch’s analytics engine at scale, yet control costs during off-peak processing times.
CreditorWatch has also connected with other AWS customers that have undergone similar data-driven paths. This allows the business to understand common pitfalls and forge the best path from the start. Several developers have also completed AWS certifications to enhance their understanding of efficient cloud architecting.
Vartuli comments, “It would have taken us at least a year and a half to get our ML processes right if we had to tweak settings on a hard drive in a server room. But with AWS Business Support and Amazon EMR, completion took a matter of weeks.”
Obtaining ISO Certifications and Creating Custom Data Sets
CreditorWatch leads with security by design and is using
Amazon GuardDuty for intelligent threat protection along with
AWS Web Application Firewall (AWS WAF). To instill further confidence in its solution among the rising number of enterprise customers it’s now attracting, the business is seeking ISO 27001 accreditation. Using AWS solutions that are already ISO-certified has streamlined this process.
As a next step, the company has begun working more closely with enterprises and the Big Four banks to provide custom risk scores. CreditorWatch sends its data scientists into an enterprise customer’s office to collaborate with the customer’s internal teams and mutually uncover new value-added applications for their combined data sets.
One recent ongoing opportunity—with the potential for further work—is a project initiated by the Australian Treasury to provide insights into industry payment times. “For a bureau of our size, the cost to integrate a high volume of data and spin up machines would be in the hundreds, if not millions, of dollars on premises,” Vartuli says. “We’re able to spin up and perform these actions affordably in almost no time at all on AWS.”
Learn More
About CreditorWatch
CreditorWatch has been helping businesses assess their partners’ creditworthiness since 2011. It currently enables its 55,000-plus customers to perform a credit check on any Australian business.
Benefits of AWS
- Processes 400 billion data points in 8 hours
- Designs ML framework in a few weeks instead of 1.5 years
- Controls overhead with multi-tenant, event-driven architecture
- Serves 55,000+ customers with a data analytics team of 4
- Secures infrastructure against threats while preparing for ISO certification
- Receives proactive support from solutions architects
AWS Services Used
Amazon Kinesis
Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information.
Learn more »
Amazon EMR
Amazon EMR is the industry-leading cloud big data platform for processing vast amounts of data using open source tools such as Apache Spark, Apache Hive, Apache HBase, Apache Flink, Apache Hudi, and Presto.
Learn more »
AWS Database Migration Service
AWS Database Migration Service (AWS DMS) helps you migrate databases to AWS quickly and securely.
Learn more »
Amazon S3
Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.
Learn more »
Get Started
Organizations of all sizes across all industries are transforming their businesses and delivering on their missions every day using AWS. Contact our experts and start your own AWS journey today.
/Creditorwatch_logo%402x.6e5d0da490c143b4b882a503f3887e4367d4767f.png)
