Scaling for performance with AWS Mainframe Modernization and Micro Focus

Mainframe applications often execute business critical functions which have to be resilient, scalable, and cost efficient. When migrating and modernizing these applications with AWS, it is important to design an application and infrastructure architecture meeting the performance requirements while optimizing the cost and satisfying availability and scalability needs. This applies to both online and batch applications. For example, performance requirements can specify response time, transactions per second, or batch duration. This blog post describes the performance benchmark of a mainframe online application running on AWS, showcases vertical scaling and horizontal scaling with Micro Focus engine included in AWS Mainframe Modernization service, and highlights some best practices for performance.

Performance benchmark architecture

The Ordering application used for the benchmark is a COBOL/CICS suite of programs that is designed to be deployed in different configurations. This application manages orders and payments. The Ordering application is made up of a set of transactions performing read and write accesses to the data stored in Amazon Aurora PostgreSQL database using VSAM data file and Micro Focus Database File Handler (MFDBFH). More than 85% of the transactions are “write” transactions.
The distribution of the transactions is the following:

• Make a New Order: This transaction is inserting/updating some data into the database. It represents about more than 40% of the total number of transactions
• Make a Payment: This transaction is inserting some data into the database. It represents about more than 40% of the total number of transactions
• Check the Order Status: This transaction is reading some data from the database. It represents about 5% of the total number of transactions
• Check the Delivery: This transaction is reading some data from the database. It represents about 5% of the total number of transactions
• Check the Stock Level: This transaction is reading some data from the database. It represents about 5% of the total number of transactions

For the performance benchmark tests, we have designed, configured and tested two topologies.

The first topology is a standalone environment designed to showcase scale up also known as vertical scalability. In this scale-up topology, we will increase both the size of the EC2 instance with a single Micro Focus Enterprise Server region and the size of the EC2 instance but also increasing the number of discrete Micro Focus Enterprise Server regions.

Figure 1. Standalone environment for vertical scaling

The second topology is a highly available environment designed to showcase scale out also known as horizontal scalability.

Figure 2. Highly available environment for horizontal scaling

For both topologies, the user workload is generated through traditional TN3270 connections using some Silk Performer test automation scripts. The injection scripts used for the benchmark simulate multiple virtual users executing the transactions as described above. A think time has been introduced between the transactions within the Silk script in order to make the benchmark tests more realistic.

Vertical scaling

Test scenario

For vertical scaling, the objective is to showcase the scalability with a single Enterprise Server instance. We do this by adding some workload, meaning increasing the number of virtual users executing the business transaction, and observing the throughput (tasks per seconds inside Micro Focus Enterprise Server), response times, as well as the CPU consumption.
The Amazon EC2 instance selected for the vertical scaling are c5 instances which are optimized for compute-intensive workloads.

The 3 following instance types have been used during the vertical scaling tests:

Figure 3. Instance types for vertical scaling

The following test scenario have been executed for each instance type:

Figure 4. Scenarios for vertical scaling

For the C5.2xlarge and C5.4xlarge instances, 2 types of configuration have been tested:

• A single Enterprise Server collocated in 1 EC2 instance
• Multiple Enterprise Server collocated in 1 EC2 instance

Test result highlights

The performance tests showed that a c5.large instance running a single Micro Focus Enterprise Server can handle up to more than 400 tasks per second, simulating 3,000 virtual users. In such configuration, the scalability limiting factor is the CPU consumption, which reaches almost 100%.

Figure 5. Test results with c5.large vertical scaling

The performance tests showed that the number of tasks per sec with 3 Micro Focus Enterprise Server collocated in the EC2 instance is increasing up to 1,600 tasks per second, simulating up to 12,000 virtual users. It demonstrates that a c5.2xlarge instance, which is 4 times larger than the c5.large instance, is able to process 4 times more tasks per second (1600 versus 400) than the c5.large.

Figure 6. Test results with c5.2xlarge vertical scaling

The performance tests showed that the number of tasks per sec with 6 Micro Focus Enterprise Server collocated in the EC2 instance is increasing up to 3200 tasks per second, simulating 24,000 virtual users. It demonstrates that a c5.4xlarge instance, which is 8 times larger than the c5.large instance, is able to process 8 times more tasks per second (3200 versus 400) than the c5.large.

Figure 7. Test results with c5.4xlarge vertical scaling

The tests results demonstrate linear vertical scaling of transactions per second throughput and stable response times by adding more CPU resources. The implication with larger EC2 instances is that the vertical linear scalability is reached by increasing the number of Micro Focus Enterprise Server regions.

Horizontal scaling

Test scenario

For horizontal scaling, the objective is to showcase the scalability with multiple single Enterprise Server each running on different EC2 instances.
The horizontal scalability is shown by adding some workload, meaning increasing the number of virtual users executing the business transaction as well as by adding some Micro Focus Enterprise Server EC2 instances and observing throughput (tasks per seconds inside Micro Focus Enterprise Server) and response times.
The Amazon EC2 instance selected for the horizontal scaling is a C5 instance. This instance is optimized for compute-intensive workloads.

The following instance type has been used during the horizontal scaling tests:

Figure 8. Instance type powering Enterprise Server

For the configuration of the Micro Focus Performance and Availability Cluster (PAC), the Amazon Elasticache for Redis was used. The model used for the test is a r6g.4xlarge with the following characteristics:

Figure 9. Intance type powering ElastiCache

The following test scenario have been executed on this instance:

• 3000 virtual users
• 6000 virtual users
• 9000 virtual users
• 12000 virtual users
• 15000 virtual users
• 18000 virtual users
• 21000 virtual users
• 24000 virtual users
• 27000 virtual users

Test result highlights

The following diagram shows the tasks rate accumulated across all Micro Focus Enterprise Server instances correlated with the total tasks rate and the number of Micro Focus Enterprise Server instances when increasing the number of virtual users from 3,000 to 27,000.

Figure 10. Test results with horizontal scaling

The tasks rate within each Micro Focus Enterprise Server instances is stable for the 9 instances when increasing the number of virtual users. This rate is around 450 tasks per second, which is the same threshold identified during the scale up tests.
The total tasks rate accumulated across all Micro Focus Enterprise Server instances is increasing linearly with the number of Micro Focus instances when increasing the number of virtual users.
The following diagram shows the tasks rate accumulated across all the Micro Focus Enterprise Server instances correlated with the rate of the business transactions from an end user perspective when increasing the number of virtual users from 3,000 to 27,000.

Figure 11. Transaction rates with horizontal scaling

The tasks rate accumulated across all Micro Focus Enterprise Server instances is increasing linearly from 400 tasks per second to almost 4000 tasks per seconds when increasing the number of virtual users.

This figure shows that with 9 c5.large instances running Micro Focus Enterprise Server with a total capacity of 18 vCPU, the scale out configuration can handle up to 27,000 virtual users and process more than 4,000 task per second.
During these scalability tests, the task response time stays stable below 30 milliseconds. It is measured from the start of the task in a Micro Focus Enterprise Server instance, including the database access, until the task completes in the Enterprise Server instance.

Performance best practices

Scale horizontally – The results of the performance tests show that a scale out topology running relatively small EC2 instances hosting a single Micro Focus Enterprise Server instance should be preferred for online applications versus a scale up topology with larger instances. It allows better repeatable and consistent performance by reusing and reproducing the same compute and performance tuning parameters for an increasing number of end users. It allows better availability and more stable performance with a smaller blast radius. In case of an instance failure, the amount of impacted requests is smaller, providing a better user experience, reducing the amount of fail-over infrastructure processing, and spreading the fail-over load to a larger number of compute instances. It provides better cost efficiency by starting and stopping instances adapting to the load and only paying for what you need and use.

Although the horizontal scaling is preferred for online workloads, it does not necessarily apply to batch workloads. Batch applications do not have the same availability requirements and performance profiles, and some batch workloads can benefit from vertical scaling.

Optimize instance type – Amazon EC2 provides a wide selection of instance types optimized to fit different use cases. Instance types comprise varying combinations of CPU, memory, storage, and networking capacity and give you the flexibility to choose the appropriate mix of resources at the right price point for your applications. Each instance type includes one or more instance sizes, allowing you to scale your resources to the performance requirements of your mainframe workload. M6i instances provide a balance of compute, memory, and network resources, and is a good choice for many applications. R6i instances are an ideal fit for memory-intensive workloads. If your application requires fast CPU processing with high clock speed, M5zn instances deliver fast Intel Xeon Scalable processors, with an all-core turbo frequency up to 4.5 GHz. If you need to combine fast CPU and very large amount of memory x2iezn family starts at 256 GiB of memory.
Choosing instance types also applies to other AWS services including Amazon Relational Database Service, AWS Mainframe Modernization, and Amazon AppStream 2.0.

Pool connections – Micro Focus Enterprise Server instances can have a large number of open connections to the database server. It’s important to monitor the database connections from the Enterprise Server instances. You should consider using Amazon RDS Proxy if the open connections reach such numbers of open connections. Amazon RDS Proxy allows applications to pool and share connections established with the database, improving database efficiency and application scalability.

Tune the database – The data store performance is critical to support the overall application performance. When using Amazon Aurora fully-managed service, the scaling is facilitated with automated storage scaling. Nonetheless there are best practices you should consider in Managing performance and scaling for Aurora DB clusters. When using Amazon Relational Database Service, we recommend following Best practices for Amazon RDS.

File-system performance – When modernizing mainframe applications and files with AWS, it is important to choose the right file-system for the right application data profile. File systems have to meet stringent functional and non-functional requirements balancing multiple dimensions such as performance, availability, and cost. The Selecting File-Systems for AWS Mainframe Modernization blog post describes typical requirements for mainframe applications, AWS file-system options, and provides guidance for choosing an appropriate AWS file-system.

AWS fully-managed services – They free your teams from time-consuming administrative tasks like server provisioning, software maintenance, and complex deployment configuration. For performance, fully-managed services include and promote performance best practices with proven designs and vetted tuning parameters. These services also include native capabilities to monitor and optimize the performance of your applications. For example, the use of AWS Mainframe Modernization fully-managed runtime and the use of Amazon Aurora simplify and optimize the deployment of highly available, elastic, and high performance environments.

Go global – With AWS global infrastructure, you can easily deploy your applications in multiple regions around the world with just a few clicks. This means you can provide lower latency and a better experience for your customers at minimal cost. This is especially relevant for online applications with users in multiple geographic areas. For example, some mainframe applications have heavy read-only traffic. For such applications, the application logic can be deployed in multiple regions nearby end-users while the application data can be synchronized across regions for lower latency.

Experiment often – With virtual and automatable resources, you can quickly carry out comparative testing using different types of services, compute, storage, database, network, or configurations. This is even more relevant when using fully-managed services accelerating deployment activities and allowing the test of more configurations quickly. For example, when using AWS Mainframe Modernization fully-managed runtime, you can quickly change and test various compute instance type, or you can update with a few clicks the number of parallel instances processing the workload.

Load-test and benchmark – Using AWS services, you can run production-scale environments to test your architecture aggressively. Since you only pay for the test environment when it is needed, you can carry out full-scale testing at a fraction of the cost of using an on-premises environment. Take advantage of the AWS Cloud to test your workload to see where it fails to scale, or scales in a non-linear way. Depending on whether your mainframe application has low latency requirements, or a large number of end-users, or a high transaction per second, such performance test should be done early to gather learnings and help drive architecture decisions as well as resource selection.

Application and architecture optimizations – Although there are many infrastructure-level tuning options and parameters, the performance of a system is mostly driven by the application logic, data manipulations, and interactions with other systems over the network. If infrastructure tuning does not allow meeting performance requirements, you should consider more substantial changes and optimization of the overall architecture or application logic or data model. For example, some applications benefit from the addition of data caching layers to accelerate data access latency. Some applications benefit from logic code changes to parallelize processing as opposed to serialize processing. Some applications benefit from schema and data access changes to reduce locking and bottlenecks.

AWS Well Architected Performance Efficiency pillar – The AWS Well-Architected Framework helps you understand the pros and cons of decisions you make while building workloads on AWS. Using the Framework helps you learn architectural best practices for designing and operating reliable, secure, efficient, and cost-effective workloads in the cloud. The Framework provides a way for you to consistently measure your architectures against best practices and identify areas for improvement. Performance Efficiency is one pillar of this Framework where you will find extensive and detailed guidance to further improve the performance of applications running on AWS Cloud.

AWS scalability examples

When evaluating performance and scalability, it’s valuable to analyze the scalability of infrastructure components for multiple workload types. For example, Temenos Banking Cloud solution ran a benchmark of 100 million customers and 200 million deposit accounts with 100,000 transactions per-second and up to 61 transactions per-second, per-core on AWS. Moreover, the AWS STAC-M3 benchmark shows vertical and horizontal scaling for financial services high performance workloads. It uses Amazon EC2 X2iedn instances with 128 vCPU and 4 TB of memory supporting up to 100 Gbps of network throughput and 80 Gbps throughput to Amazon EBS io2 Block Express volumes. Principled Technologies ran a TPC-C-like OLTP workload on Amazon EC2 processing about 1,300,000 New Orders Per Minute (NOPM).

Go scale

This performance benchmark performed for a typical order management core-business application showcases AWS linear scalability for both vertical and horizontal scaling of Micro Focus engine included in AWS Mainframe Modernization service. AWS Mainframe Modernization service managed runtime embeds performance best practices for easier configuration and management.

Such scalability is essential for resilient and highly available applications with varying load and users. For any application and system with high throughput requirements, we recommend you verify linear scalability and elasticity for your specific application early preferably by leveraging horizontal scalability. This performance verification can be done at production peak scale with load tests or performance benchmarks by allocating temporary AWS environments and resources on a pay-as-you-go basis. Using infrastructure as code allows rapidly and safely testing and evolving your architecture for better performance while relying on monitoring data to make fact-based decisions.

If you need assistance in designing or optimizing for performance, feel free to contact an AWS specialist.

About the author: