Mainframe Data Migration to AWS with Infosys

By Karthik Kumar Guruprasad, Sr. Technology Architect, Mainframe Modernization at Infosys
By Vishnu Sankar Gengaraj, Sr. Consultant, Mainframe Modernization at Infosys
By Abhishek Ashish, Consultant (Cloud), Mainframe Modernization at Infosys
By Vikrama Adethyaa, Sr. Partner Solutions Architect at AWS

Mainframe applications have been running the most mission-critical workloads in many large enterprises for decades. It’s common for them to store, compute, and transfer terabytes of data in day-to-day processing.

Such applications are difficult to maintain and expensive to operate, which is why mainframes are now being migrated successfully to other platforms by either replatforming (middleware emulation) or automated refactoring approaches.

In either approach, modernizing the data store and data access plays a critical factor in the success of the transformation project.

Modernization projects can fail or become subtly complex in execution for a variety of reasons:

• Migration of decades old active data; mainframe-native data formats, organization, and access methods.
• Poor change data capture (CDC) strategy during transition state of application migration.
• Limited capability of exposing mainframe data to other channels and analytics.
• Expensive and complex data retention strategy for regulatory mandates.

With tools and accelerators from Infosys, an AWS Premier Consulting Partner with the AWS Migration Competency, data migration from mainframe systems has already overcome many technical challenges, including z/OS file management system, EBCDIC encodings, hierarchical/network databases, mainframe’s native computational data types, and data access methods.

Infosys, having gained experience on various mainframe modernization projects, has developed an innovative ART Framework and other tools and accelerators. In this post, we will show you how Infosys helps customers overcome data migration challenges and eliminate migration risks.

Migration of Decades Old Active Data

In most industries, mainframe applications are decades old and hold petabytes of transactional data. Characteristics of data that are innate to mainframe systems add complexity to data migration to cloud.

Examples of this include EBCDIC encoding, fixed or variable file formats, key-indexed, random or entry-indexed data access methods, hierarchical or network databases, and computational datatypes.

Amazon Web Services (AWS) offers a wide spectrum of storage services with industry-leading scalability, security, availability, and performance.

The storage services in mainframes, such as sequential files and VSAMs, can be migrated to AWS storage services such as Amazon FSx, Amazon Simple Storage Service (Amazon S3), Amazon Elastic File System (Amazon EFS), and Amazon Elastic Block Store (Amazon EBS).

The data from hierarchical or relational databases—IMS and DB2–can be migrated to Amazon Relational Database Service (Amazon RDS) or any x86-64 platform database engine hosted on AWS. The x86-64 platform database engine could be any of Amazon Aurora PostgreSQL, SQL Server, Oracle, DB2 LUW, or MariaDB.

Data migration is a critical activity in any migration project with its own set of challenges and risks. The Infosys Mainframe Modernization Platform suite of tools like EBCDIC-ASCII convertor, iBREX, and iLEAD help in each phase of the data migration: discovery, migration, and post-migration.

Discovery

During the discovery phase, the data sources along with applications are assessed for compatibility with target architecture, and the integration points are assessed for smooth migration.

Qualitative and quantitative analysis are performed on source data to choose the right set of tools and migration activities. Assessment tools like Micro Focus Enterprise Analyzer, along with custom-built scripts, is used to assess the mainframe data.

Migration

Migration involves schema and data migration to target systems. Before production cutover, it will be tested repeatedly in test environments to avoid any data loss during cut over.

The whole migration process can be largely automated with a one-button process with no human intervention to the production data with backup and fall back procedures. Infosys Data Services Suite (iDSS) accelerates data migration tasks such as EBCDIC-ASCII conversion and schema conversion.

AWS Snowball is a petabyte-scale data transport solution that uses devices designed to be secure to transfer large amounts of data into and out of AWS.

Post-Migration

Post migration, the migrated data will be validated and reconciliated against the source system for data consistency. Functional and performance testing can be carried out to further ensure data migration meets the expected goals of the transformation project.

Infosys Data Testing Workbench (iDTW) helps accelerate the testing process and reduce risks.

Figure 1 – Mainframe application replatforming using a middleware emulator hosted on AWS.

Data Replication for Coexistence and Hybrid Scenarios

Many modernization programs avoid big-bang approaches to avoid risks and are executed in a phased approach. During and post migration, mainframe systems need to coexist with new target systems. This helps to perform system parallel testing and safely cut off mainframe systems without any business disruptions.

During this hybrid scenario, any incremental changes to the data in the source databases will be replicated in target databases in near real-time, and an eventual consistency will be maintained.

Change data capture (CDC) tools like Qlik Replicate can be used to replicate the data across heterogeneous systems like mainframe, AWS, Hadoop, relational database management system (RDBMS), and files.

After migration, CDC also helps to refresh data in prod-like test regions to keep the data in sync with production. This accelerates parallel testing in replatforming or automated refactoring projects.

Figure 2 – Data replication for database migration and data refresh in test environments.

Retaining Historical/Archival Data Post Migration

Tape storage in mainframe systems offer bulk storage at a cheaper price and holds terabytes of historical data in the application. This is governed by the data retention policy of the company, as mandated by regulatory bodies.

Amazon S3 Glacier offers cheap storage for data to be archived, and S3 Glacier Deep Archival for cold data archival requirements. Custom-built Infosys scripts help to retrieve archived data from mainframe storage, convert EBCDIC to ASCII and load into temporary data staging storage, and then transfer to S3 Glacier using direct APIs, SDKs, or CLI.

Any frequently accessed data stored on AWS that exceeds the retention period would be moved to Amazon S3 Glacier and S3 Glacier Deep Archival, according to data retention rules, using custom defined scripts or as an activity via data pipeline setup.

Amazon S3 Glacier also provides various data retrieval options: Standard, Bulk, and Expedited to restore the data per the application requirements.

Data Migration Tools

Various products can be used to streamline data migration depending on the use case. The following list outlines AWS first-party and Infosys proprietary and third-party products that help accelerate data migration use cases:

• AWS Snowball: Petabyte-scale data transport solution for secure data transfer into and out of the AWS Cloud.
• Micro Focus Data Tools: Host Compatibility Option for SQL Server (HCOSS) helps to migrate mainframe data to SQL Server, and DFConv tool helps to convert EBCDIC data to ASCII during migration.
• Infosys Rehosting Suite: Migration tools help VSAMs and generation datasets and variable block files migration.
• Infosys ASWA: Data validation and reconciliation tool for RDBMS data migration.
• Infosys Data Services Suite (iDSS): Provides an automated migration and reconciliation solution with customized migration approach to AWS.
• Infosys Data Testing Workbench (iDTW): Helps perform validations of data integration processes in an automated fashion.
• Qlik Replicate (Attunity): Data replication across a wide range of heterogeneous systems like mainframe, AS400 systems, Hadoop, data warehouse, RDBMS, files, etc.

Customer Success Story

Kmart Australia wanted Infosys to migrate Kmart’s legacy mainframe systems to a cloud-based landscape, thereby reducing the TCO by 54 percent for the legacy merchandise portfolio.

Infosys, using the ART Framework, successfully executed a phased approach mainframe modernization, re-hosting Kmart’s core applications onto Micro Focus Emulation hosted on AWS in the first phase/ This was followed by a re-engineering.

Technology Landscape

• Source System (Mainframe): COBOL, JCL, DB2, Assembler, REXX, CLIST, CICS, QMF, ACF2.
• Target System (Micro Focus Emulation): Micro Focus Enterprise Server, Enterprise Developer, AWS Cloud, Jenkins, ServiceNow, SNS, PagerDuty, GitHub, LRS, Control-M, Microsoft AD, Visual Studio.
• AWS Services: Amazon EC2, Amazon FSx for Windows File Server, Amazon RDS for SQL Server, Amazon Route 53, AWS CloudFormation, Amazon CloudWatch, AWS IAM, AWS Lambda.

The Infosys Way

Infosys has seamlessly migrated the data (1.4TB of DB2 tables and 5TB of data files) from the source environment to the target.

Infosys, in its own way, helped Kmart to mitigate the migration risks:

• Infosys used Micro Focus Host Compatibility Option for SQL Server (HCOSS) for seamless data migration.
• Designed interim architecture keeping in mind the upcoming optimization phase.
• Built automated data reconciliation using Infosys Aswa tool.
• Developed custom-built scripts for automated test environment provisioning with data refresh.
• Amazon RDS is configured to have Always-On availability groups with data replication.
• Amazon FSx for Windows File Server is leveraged to archive old data.

Conclusion

This post addressed the data migration use cases for modernizing mainframe applications using AWS, and why Infosys is an implementation partner that can assist you with the migration.

When a one-solution-fits-all does not work for a big transformation project, the Infosys ART Framework for mainframe modernization—coupled with Infosys’s strategic partnership with AWS—presents a proven approach customized to customers’ technology and business requirements.

If you are considering migration to the cloud, contact Infosys for more information: LegacyMod@infosys.com.

.

.

Infosys – AWS Partner Spotlight

Infosys is an AWS Premier Consulting Partner that helps enterprises transform through strategic consulting, operational leadership, and co-creation of solutions in mobility, sustainability, big data, and cloud computing.

Contact Infosys | Partner Overview

*Already worked with Infosys? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.