Bendigo Reduces Compute Costs by 60% Using Amazon EC2 Spot Instances and Amazon EMR

Financial service provider Bendigo and Adelaide Bank (Bendigo) started its digital transformation journey on Amazon Web Services (AWS) so it could comply with Australia’s open-banking mandates. In March 2020, Bendigo quickly migrated 30 non-material workloads to the cloud, underpinned by Amazon Elastic Compute Cloud (Amazon EC2), which provides secure and resizable compute capacity for virtually any workload.

Since completing its initial cloud migration, Bendigo focused on optimizing its architectures, so that it can increase cost savings and improve the performance and availability of its digital offerings. To balance performance with cost savings, Bendigo started running its nonproduction workloads on Amazon EC2 Spot Instances, which let customers take advantage of unused Amazon EC2 capacity in the cloud. The bank also achieved greater resiliency and scalability by implementing other AWS features for fault-tolerant workloads. With this new infrastructure, Bendigo has reduced its compute costs by approximately 60 percent, increased the resiliency of its workloads by nearly 30 percent, and improved the performance of its banking system by roughly 20 percent.

Bendigo’s vision is to become Australia’s bank of choice by providing its 2.1 million customers with data transparency and simple banking solutions. The bank’s purpose is to feed into the prosperity of its customers and communities—not off of them. Bendigo holds over $83.4 billion in deposits, and it serves over 110,000 shareholders while employing over 7,000 people across 317 branches.

In May 2018, the Australian government passed the Consumer Data Right legislation, which required all major financial institutions to comply with open-banking practices by July 2020. This legislation also meant that financial institutions must provide their customers with access to and control over their personal data. Facing these industry changes, Bendigo expected an increase in demand on its on-premises infrastructure. To prepare, the bank decided to undergo a digital transformation using AWS.

Bendigo chose AWS as a cloud service provider because of the maturity of its services, and in March 2020, it kicked off its cloud migration using AWS. The bank also began running its workloads using Amazon EMR, a cloud big data solution for running large-scale distributed data processing jobs, interactive SQL queries, and machine learning applications using open-source analytics frameworks. To meet open-banking mandates, the company configured clusters that would help it maintain a high availability across its banking systems. After completing this project, Bendigo wanted to improve its cloud architecture and reduce compute costs by running its clusters using Amazon EC2 instance fleets. "Cost savings wasn't something that we were thinking about initially," says Adam Hobbs, development infrastructure engineer for Bendigo. "But we realised that we wanted to improve the performance of our architecture." In 2021, the Bendigo team entered the second phase of its digital transformation journey, and the team began experimenting with new solutions and features to meet the bank's computing needs.

Bendigo engaged with the AWS team to test Amazon EC2 instances that would maintain a high availability while optimizing compute costs. During this time, the Bendigo team trialed Amazon EC2 Spot Instances, which deliver Amazon EC2 instances at a lower price compared to Amazon EC2 On-Demand Instances. Because Bendigo’s developers use its nonproduction environment for iterating features for its digital offerings, the bank identified this as an area where it could reduce compute costs. Now, the bank runs its nonproduction workloads entirely on Spot Instances. “Amazon EC2 Spot Instances have been an effective way for us to save on compute costs while maintaining an elastic workload,” says Leandro Silva, data lead for the open banking project at Bendigo. By making these changes, Bendigo has reduced its compute costs by approximately 60 percent.

The bank also tested different Amazon EC2 instances for running its Amazon EMR clusters, and it found the best performance using diversified Amazon EC2 instance types, which include varying combinations of CPU, memory, storage, and networking capacities. By diversifying its instance types, the bank has reduced the likelihood of service interruptions and downtime and made its workloads more fault-tolerant. “We went through stages of configuring different instance types,” says Hobbs. “We spent some time with the AWS team identifying which instance types have the highest availability, which has delivered a much better experience.” Under this new model, the bank has improved the resiliency of its near-real-time workloads by roughly 30 percent.

Bendigo’s Amazon EMR clusters consist of different node types, which use Amazon EC2 instances for cluster capacity planning. “We have been setting up core nodes,” says Hobbs. “We went from a 5-node cluster to a 30-node cluster.” To facilitate effective cluster scaling, the bank implemented task nodes, which it can spin up or spin down to increase capacity as needed and save on compute costs. Since then, the company has grown to use 60 core nodes for its production environment. With multiple task nodes, Bendigo’s Amazon EMR clusters can use other available task nodes if one fails, minimizing service interruptions and downtime. By increasing the elasticity of its infrastructure, the bank has increased the performance of its system by approximately 20 percent while driving down compute costs.

Bendigo also started using Managed Scaling for Amazon EMR, which automatically resizes its clusters for best performance at the lowest possible cost. Using Managed Scaling, the company can manage its compute costs more effectively by setting up minimum and maximum limits for its compute capacity. "We gain deeper insights about our workloads, which helps us monitor our usage and true costs," says Ash Austin, platforms practice lead at Bendigo. "Working on the cloud helps us enhance our risk and vulnerability management in a highly secure, robust, and controlled environment," says Austin. "AWS has been great at helping our organization find the right tools for the job and navigate the regulatory landscape. It brings us the global expertise that we need to accelerate our digital transformation journey.”