AWS Cloud Operations Blog

MagellanRx Modernizes Claims Adjudication Application from AIX Oracle platform to Amazon Linux 2 and Amazon Oracle RDS

Customers desiring to modernize enterprise workloads from on-premises IBM AIX System p environments into AWS face portability concerns. AIX workloads do not lend themselves to a lift and shift type of migration pattern and require re-platforming or refactoring. This is due to operating systems and processor architectures in Power AIX systems and x86 Linux systems that differ. In addition, processor differences between the System p family and x86 architectures, as discussed in the blog post Estimating Amazon EC2 instance needed when migrating ERP from IBM Power Systems must be considered to ensure application performance expectations are met.

In this blog post we’ll review how MagellanRx migrated its strategic  claims adjudication application written in C/C++ leveraging an Oracle OLTP database from on-premises AIX to Amazon Linux 2 and Amazon RDS for Oracle and sustained application performance and RPO/RTO objectives. In addition, we’ll review the automation solution that was introduced to support continuous integration and continuous deployment requirements to reduce the application downtime for maintenance and security patching.

MagellanRx Management a Prime Therapeutics LLC Company, is focused on claims processing and adjudication. MagellanRx is a pioneer in specialty and medical drug management and a leader in serving public sector state government programs. MagellanRx is also an AWS Enterprise Support customer.

Business Objectives

MagellanRx desired to close on-premises data centers, remove CapEx costs and decrease reliance on proprietary hardware platforms by modernizing onto AWS and thereby taking advantage of AWS’s global infrastructure.  In addition, MagellanRx wanted to reduce reliance on hard-to-find specialist knowledge for maintaining the application and database workloads on AIX. MagellanRx’s claim adjudication application is written in C/C++ which introduced a variety of challenges including justifying the costs code refactoring and library dependencies from AIX to Linux as well as ensuring that AWS architecture supported MagellanRx’s high availability and disaster recovery objectives.

Historically, enterprise customers have relied on System p hardware and the AIX operating system to support strategic workloads due to high hardware reliability and computing power and performance of the RISC CPU architecture.  As customers modernize  these strategic workloads from on-premises data centers to AWS.   You may have questions around reliability and performance. The AWS global infrastructure offers computing resources across our availability zones and regions that can satisfy even the most aggressive RTO/RPO requirements.  In addition, AWS offers fit for purpose environments for modernizing high performance workloads with a wide range of EC2 instances like AWS’s 3rd Gen AMD EPYC with frequencies up to 3.5 Ghz.

For years, customers have been successfully modernizing  Java and Oracle-based applications to AWS using AWS’s guidance, pattern libraries and best practices.  Using this material, MagellanRx established their own reference architecture and repeatable patterns to modernize the remainder of their AIX workloads into AWS. The reference architecture supports cross-region high availability and disaster recovery requirements necessary to the business.

MagellanRx, with the support of AWS, completed the modernization of  their strategic workload for Claims Adjudication from AIX to AWS. This is discussed in more detail in this article.

AWS Solution Architecture

MagellanRx was seeking to develop an architectural blueprint for modernizing all installations of firstRx, a proprietary application which consists of application code in C/C++, Java, and Perl. Post application refactoring, MagellanRx chose to use Amazon Linux 2 based instances for deploying the enterprise workload and Amazon RDS for Oracle for the OLTP database needs. Also, due to the large number of installations of FirstRx application, MagellanRx configured a robust CI/CD pipeline using AWS Code Star services for performing migration of the applications, release deployment, and maintenance fixes. To manage traffic distribution from on-premises sources, and later, directly from pharmacy PoS terminals, traffic was directed to AWS Network Load balancers.

FirstRx Code Base and Data Migration

Due to differences between AIX-RISC and x86 CPU architectures, most of the application code had to be re-analyzed, recompiled, and even re-factored entirely. MagellanRx utilized different strategies to bring the entire codebase up to modern standards depending on the language of the original code. Some examples of this modernization effort are explained below, separated by the specific coding languages used in the project.

Java: MagellanRx identified specific library packages with support for x86 architecture, then they were able to port their Java application and services by swapping out those AIX-RISC libraries and recompiling the code for Linux-x86 architecture. Once the application and supporting services had been ported, a period of extensive testing, validation, and performance evaluation were able to be completed successfully. Java provides a robust ecosystem for running applications in both AIX-RISC and Linux-x86 architectures.

Perl: Unlike Java, Perl libraries have lower portability between AIX-RISC and Linux-x86 architectures. Hence, more time was required to refactor Perl code up to modern standards for the Linux-x86 platform. Effort went into ensuring Perl code was not ported to another scripting language. This would have taken time and focus away from other initiatives. While some compatibility was resolved by identifying comparable and suitable libraries for the Linux-x86 architecture, some Perl scripts needed to be re-factored using all new libraries. Next, MagellanRx moved on to their normal and edge case validation to ensure the changed dependencies could handle all of the use cases the team could think of.

C/C++: The majority of the application’s migration was focused around analyzing, and porting 64-bit C/C++ code which was re-compiled from AIX to Linux. First and foremost, MagellanRx started identifying the most performant and compatible gcc, g++, and ld configuration options and then began resolving compilation-related errors. Specific attention was paid to warning messages emitted by the compiler and linker. Most of the compilation and linking issues were resolved by makefiles modifications. There are several notable calls around time and process APIs that did not have AIX-equivalent libraries for Linux and had to be re-factored entirely.

Some examples are:

  • struct timebasestruct_t had to be refactored to use struct timespec
  • read_real_time() was refactored to use clock_gettime()
  • restimer() had to modified to use clock_getres()
  • The getprocs() subroutine needed to have multiple fixes implemented including:
    • procsinfo.pi_pid was replaced by getpid()instead
    • procsinfo.pi_comm had its program name replaced by parsing ps -aux
    • procsinfo.pi_dsize  was replaced to use /proc/self/status or /proc/pid/status

Also, as part of refactoring the C/C++ code base, all of their code was reorganized to improve maintainability, and avoid circular dependency at linking stages during code compilation. MagellanRx used Valgrind to find, analyze, and resolve memory leaks within their application.

Data migration

Data from Oracle on AIX was migrated to Oracle RDS using AWS Database Migration Service. The cutover schedule enabled downtime for migrating the data from an on-premises datacenter into AWS using a dedicated network connection through AWS Direct Connect.

Solving for High Availability

FirstRx had stringent Recovery Time Objective (RTO) requirements of 2 hours and Recovery Point Objective (RPO) of 20 minutes. To support these requirements, MagellanRx implemented both multi-AZ active and multi-region stand by configurations for the FirstRx Amazon RDS for Oracle databases.

The solution is deployed in us-east-1 as a primary region and us-west-2 was selected as a hot standby environment for needs of regional failover. The Amazon RDS for Oracle database was configured with continuous replication from the us-east-1 primary database to us-west-2 stand by instance; thus, ensuring that RPO and RTO are met, and exceed stated objectives. In case of regional failover, the replica of the database is promoted to take traffic from an application tier in us-west-2 region.  To optimize costs of the environment in us-west-2 region, the application tier was configured in a pilot light configuration to scale-out in case of failover to meet the stated RTO.

AWS Backup service utilizing the backup vault configuration was configured to store backups in us-west-2 and us-east-1 regions.

Migration Process

As an AWS Enterprise Support customer, having a Solution Architect (SA) and Technical Account Manager (TAM) involved from the early stages of the initiative helped deliver the project on time. Both the TAM and SA were involved in the migration planning, well-architected reviews, and even supported the team during cutover testing and the final production cutover. The technical account manager used the AWS Infrastructure Event Management (IEM) process which is part of the Enterprise Support benefits. An IEM offers architecture and scaling guidance as well as operational support during the preparation and execution of planned events, like this migration and architecture optimization.

CI/CD DevOps

MagellanRx believes in automation being written, tested and implemented from the first day of operation on AWS. This includes infrastructure as code via AWS CloudFormation, and automation of scheduled tasks for data ingestions. MagellanRx must adhere to regulatory and compliance requirements where each Amazon Machine Image (AMI) must be certified by their Cloud Engineering and Security teams with pre-installed agents for management. A pipeline using AWS CodeBuild is using approved AMIs for automatically building such environments. This streamlined process saves development time and complexity during feature releases, security patching, and operational activities.

Application infrastructure is built using AWS CloudFormation templates; this infrastructure as code approach ensures that infrastructure can be re-created and similar foundational infrastructure constructs can be leveraged for migrating future applications and solutions from on-premises datacenters into AWS.

To meet the performance requirements of the workload, Magellan Health chose to use Amazon EC2 C6a instances type powered by 3rd generation AMD EPYC processor for its application tier. This compute optimized instance delivered and exceeded performance requirements for the application.

Graviton instances were not considered at the time, as 3rd party mandatory infrastructure services that Magellan Health was deploying on each Amazon EC2 instance were not supporting AWS Graviton Processors due to its ARM architecture.

For the database tier, MagellanRx selected to use Amazon RDS for Oracle service and host the database on an R6i instance to meet performance characteristics of the FirstRx application.

MagellanRx was able to reduce use of the AIX operating system and hardware, as well as reduce operational costs and attain multi-region failover. Performance SLAs of their Claims Adjudication solution were met and exceeded. Automation was introduced for continuous integration and continuous deployment of the environment and supporting infrastructure.

Next Steps

MagellanRx is continuing to innovate and optimize their cost of ownership of the FirstRx application.  MagellanRx is considering migration from Amazon RDS for Oracle service to Amazon Aurora PostgreSQL to further reduce maintenance overhead and cost of ownership.

As ARM support for 3rd party infrastructure services matures, MagellanRx is considering recompiling the FirstRx application C/C++ code for ARM architecture and deploying on Graviton instances; this step will help Magellan Heath to further optimize performance of the application and reduce costs.

About the authors:

Mikhail Vaynshteyn

Mikhail is a Solutions Architect with Amazon Web Services. Mikhail works with Health Care Life Sciences customers and specializes in Data Analytics Services. Mikhail has more than 20 years of industry experience covering a wide range of technologies and sectors.

Jeremy Hall

Jeremy Hall is a Technical Account Manager with Amazon Web Services. Jeremy works in multiple verticals, with a focus on retail and healthcare. Jeremy’s 18+ years’ experience allows him to assist customers with a large range of services in the AWS portfolio. Jeremy’s passion for computing and technology goes back to when he was 6 years old when he built his first computer. He currently lives with his wife, 3 cats, and one golden retriever named “Terabyte Data Hall”(Tera for short!) in the Dallas/Fort-Worth, Texas area.

Krishnakumar Kovath

Krishnakumar Kovath (Krishna) is Sr Director IT Architecture with MagellanRx and Prime Therapeutics. Krishna is primarily focused on implementing cloud-first solutions and currently responsible for the end user systems and API for core business modules. Krishna is passionate about architecting and implementing cloud-based solutions for health care domain, leveraging open-source systems, and AWS best practices.

Viral Thakkar

Viral is a Director of Infrastructure Engineering at Magellan Rx and Prime Therapeutics. Viral’s primary roles are Oracle DBA subject matter expert, and people leader with hands on involvement in the FirstRx migration to AWS using AWS Database Migration Service, and establishing a multi-region data replication. Viral played a key role for all DBA activities during the AWS cloud migration journey.