Accelerating mainframe modernization testing with AWS Transform

Mainframe modernization is necessary for enterprises to increase agility, reduce costs, innovate, and mitigate business risks. Yet organizations face a significant hurdle during modernization: testing is critically important to validate modernized applications behavior, but it typically consumes more than half of project timelines and resources. As enterprises race to modernize their legacy systems, this testing bottleneck threatens to derail modernization initiatives and inflate costs. AWS Transform introduces AI-powered testing capabilities designed specifically to streamline and accelerate the mainframe modernization testing process with a focus on functional testing. In this blog post, we describe AWS Transform new testing capabilities, alongside their benefits and how to get started.

Testing necessity for modernization success

Testing provides the level of assurance and mitigates the risks of mainframe modernization projects. It provides key insights for decision-makers to approve transitioning from legacy to modernized applications. Typically, it includes functional tests, integration tests (scheduler, security, 3rd party), non-functional tests (performance, availability, disaster recovery), and user acceptance tests.

The testing challenge is multifaceted and deeply rooted in both technical complexity and resource availability. Organizations are grappling with a diminishing pool of mainframe subject matter experts, creating significant bottlenecks as experienced professionals retire. This scarcity of expertise means that testing activities must compete for limited resources who understand both legacy mainframe systems and modern architectures.

The technical complexity of these projects further compounds the challenge. Mainframe applications typically process millions of records, making comprehensive testing data intensive. The process is further complicated by the need to translate between mainframe data formats and modern formats. Additionally, these applications often have numerous interconnected components and external system dependencies. For projects needing functional equivalence, such as Refactor and Replatform projects, testing requires meticulous bit-by-bit comparison of outputs.

The stakes are particularly high from a business perspective. Even minor deviations from mainframe results can have serious business consequences. As a result, comprehensive testing remains the primary mechanism for building customer confidence in modernized systems.

These challenges create compounding delays because testing occurs late in the modernization lifecycle, after significant investment has already been made. The sequential nature of test-fix-retest cycles creates a cascade effect, with projects and testing phases frequently extending beyond planned timelines by several months.

Introducing AWS Transform for mainframe testing capabilities

AWS Transform is launching a suite of AI-powered testing agents and capabilities designed to automate and accelerate the testing process for mainframe modernization.

AWS Transform starts with features prioritizing functional testing. Functional testing is the critical phase necessary to validate comprehensive modernized application business workflows and user scenarios. In other words, functional testing provides the mechanism for AWS Transform users to validate and trust that the application code modernized by AWS Transform meets the user business functional requirements. Functional testing is required for Refactor, , and Replatform projects. Refactor and Replatform projects require functional equivalence testing. Reimagine projects can blend a combination of functionally equivalent and non-functionally equivalent testing. When non-functionally equivalent, tests can be derived from application new specifications as opposed to the legacy application behavior.

Functional tests, along with specific subsets of test cases, can be reused to drive subsequent modernization testing phases, including integration and non-functional testing. Once the modernization project is complete, these functional tests also serve as a foundation for regression testing.

For their first general availability release, these AWS Transform testing agentic features focus on batch workloads where the majority of testing troubleshooting time is typically spent. These new capabilities include planning test cases,

These testing capabilities are designed to be used independently or in a sequence. This provides more flexibility to incorporate as part existing testing workflows and integrate them with external tools.

Plan test cases

The Test Plan feature in AWS Transform is designed to automatically generate and prioritize comprehensive functional test cases in natural language. This plan is for modernization project managers, application subject matter experts, and business stakeholders to align on a prioritized list of relevant functional test cases. The plan is the overarching document driving what data will be used, which test cases will be automated and executed in subsequent testing activities.

This feature examines the complex web of dependencies between programs, data, jobs, and scheduler plans to create meaningful and extensive functional test cases. For batch workloads specifically, it creates test cases that follow the expected sequencing of scheduler plan jobs.

The capability intelligently adjusts the test case scope based on three key factors: the complexity of the applications, the number of dependencies involved, and the total lines of code. To make the test plans more manageable and meaningful, it organizes test cases by business functions and domains. The system also includes a prioritization mechanism that considers both business logic information and technical metrics to determine which test cases should be executed first.

Figure 1: Test cases aligned with scheduler flow

The testing and the expected outcomes, making it easier for testing teams to setup data stores and validate results.

The workflow offers flexibility, enabling users to configure test plan inputs, define testing scope, and refine test plans as needed. Refinements include updates, splits, merges, additions, and deletions based on business requirements, technical considerations, metrics, and prioritization needs.

Figure 2: Test case guidance generated from extracted business logic

Generate test data collection scripts

The Test Data Collection Scripts Generation feature in AWS Transform automates the creation of Job Control Language (JCL) scripts to gather test data. This addresses one of the most time-consuming and error-prone aspects of mainframe testing.

These scripts are designed for mainframe to collect both input and output data from various mainframe data stores, supporting various data sources including sequential files (PS, GDG, PDS), indexed files (VSAM), and DB2 tables.

For mainframe data collection, the workflow begins with test plan ingestion, test case selection, script configuration, and script generation in AWS Transform. Then the generated scripts are reviewed by mainframe system programmers and transferred to the mainframe environment. Once on the mainframe, the scripts are executed to collect the required test data, which is then transferred back to AWS. For such data sanitization, the templates can be modified to include batch job steps using the customer data sanitization utilities.

Figure 3: Generated test data collection scripts 

The feature is built with flexibility in mind, offering customizable templates that allow mainframe experts to modify the scripts according to their specific mainframe utilities, mainframe standards, and best practices. The scripts are generated based on the test data requirements identified in the test plan, ensuring alignment with overall testing objectives.

While the feature generates collection scripts, it does not manage actual mainframe job submission or data transfer processes. These activities remain under mainframe system programmer control for security and governance purposes.

Figure 4: Test data collection script generated by AWS Transform agents

Generate test automation scripts

The Test Automation Scripts Generation capability in AWS Transform provides the automation necessary for functional equivalence testing at scale in the modernized AWS environment. Tailored for refactor projects, it generates for modernization specialists comprehensive functional test scripts based on the established test plan. This automation is key for enabling consistent, repeatable test execution and validation throughout modernization projects, and for non-regression testing once the modernized applications are in production.

The generated scripts incorporate the three key phases of functional testing logic. First, they include the logic needed to establish and provide test inputs for initial conditions. Second, they contain the necessary commands to execute the test case logic itself. Third, they incorporate comparison logic to validate results against expected outcomes, ensuring thorough verification of the modernized applications. The generated testing scripts integrate with the AWS Transform refactor testing landing zone making it easier to convert the data and compare results.

Figure 5: Test automation scripts generated by AWS Transform agents

Tools for testing environments

Beyond the automation required to run test cases at scale, AWS Transform provides tools to support specific testing tasks in mainframe modernization environments.

Data Migrator tool is designed to facilitate the migration of database schemas and data from legacy systems (such as Db2 z/OS and IMS DB) to modern database platforms (including PostgreSQL, Db2 LUW, and Oracle), supporting a variety of source and target systems. It provides a reliable way to migrate mainframe data to modern database platforms, including both schema conversion and data migration. It allows users to define multiple steps, such as creating the schema, tables, load data, and create indexes, and execute them in the proper order to complete the migration.

Compare tool automates the verification that a modernized application produces the same results as reference data. For example, in the context of projects requiring functional equivalence, it allows comparing the legacy expected data with the modernized obtained results to make sure that they match bit-by-bit. It supports comparing different data types including flat files, pdf, text files, binary files, and databases (Db2, MySQL, Oracle, PostgreSQL, and SQL Server).

Terminals tool provides the capabilities to connect to legacy mainframe and midrange screen interfaces to capture scenario scripts and videos, in the context of capturing online test cases.

Conclusion

AWS Transform for mainframe AI-powered testing capabilities work together as an integrated solution to accelerate mainframe modernization projects by automating time-consuming aspects of testing, which include test case planning, test data collection, and test script generation. These features can reduce overall project timelines and cut testing phases that typically extend several months beyond planned schedules.

The Test Plan feature provides the strategic foundation by automatically generating comprehensive test cases. The Test Data Collection Scripts ensure you have the right data from your mainframe systems. The Test Automation Scripts execute functional equivalence testing in your modernized AWS environment.

This end-to-end automation addresses the critical bottleneck where testing traditionally consumes over half of modernization project resources.

AWS Transform for mainframe testing capabilities are generally available and can be accessed from the AWS Transform for mainframe user interface. To learn more about how AWS Transform can accelerate your modernization initiatives, visit the AWS Transform documentation or contact your AWS representative today.