Migrating Legacy Workloads to AWS in Days with LeapLogic

By Abhijit Phatak, Director of Engineering, LeapLogic – Impetus Technologies
By Gurvinder Arora, Product Marketing Manager, LeapLogic – Impetus Technologies
By Gitika Vijh, Partner Solutions Architect – AWS

Impetus

Most enterprises cite migration costs as a major inhibitor to cloud adoption, but other factors can be potential showstoppers for an enterprise’s modernization journey.

This includes business disruption, asynchronous documentation of the legacy system, lack of sufficient skilled resources, costly manual interventions, limited knowledge of complexities and dependencies across legacy systems, risk of losing business logic, and inability to resolve technical debts on the move.

Businesses must have a robust plan, an automated transformation mechanism, and an experienced guide to help them modernize seamlessly.

Impetus Technologies created its automated cloud transformation accelerator, LeapLogic, to enable faster migration across a range of legacy data and analytic software. Its unique intelligent pattern-based transformation goes beyond line-by-line translation.

From diverse configurations and prescriptive recommendations to intelligent incorporation of the performance tuning and optimization parameters, LeapLogic eliminates unpredictable variables while transforming legacy business code, logic, and workloads to Amazon Web Services (AWS) in an automated environment. This brings cloud benefits to businesses with more accuracy and less disruption.

Impetus Technologies is an AWS Advanced Tier Services Partner and AWS Marketplace Seller with AWS Competencies in Migration, DevOps, and Data and Analytics Consulting.

Ensuring Seamless Modernization with LeapLogic

LeapLogic helps enterprises transform their legacy data warehouse (DW), extract, transform, and load (ETL), Hadoop, and analytics workloads to AWS in four steps:

Assessment of workloads, code profiling, and dependencies with actionable recommendations.
Automated end-to-end transformation, including core business logic to Amazon Redshift-native equivalent.
Validation for pipelines, data, and row and cell-level queries.
End-to-end operationalization of workloads with a controlled price-performance ratio.

Figure 1 – Enabling end-to-end transformation and operationalization of workloads on AWS.

Integrated Assessment of Multiple Types of Workloads

LeapLogic assesses Data Manipulation Language (DML) scripts, Data Definition Language (DDL) scripts, query execution logs, stored procedures, ETL scripts, analytics scripts, and more to prepare a comprehensive list of the entire inventory.

For example, the assessment module lists queries, entities, query types, key resource utilization metrics for applications, users, and files, analysis of orchestrator scripts, active and dead code, and more for a data warehouse. Similarly, for ETL it lists workflows, mappings, sessions, expressions, source queries, procedure blocks, and more, helping the customers to:

Maximize ROI by identifying workload interdependencies (topology), re-engineering/optimization opportunities.
Identify the right target technology stack and architecture based on the business use case.

Figure 2 – Assessment dashboard of LeapLogic.

Let’s delve deeper into how the assessment module of LeapLogic works.

Prioritize High-Value Use Cases to Create a Migration Strategy

LeapLogic provides a phased migration plan with timelines, cost estimates, and a detailed sprint-wise implementation roadmap. In addition, it considers business use cases when prioritizing workloads, which helps define a roadmap and create a sprint-wise backlog for migration.

The platform provides an end-to-end dependency structure (see Figure 3), including triggering orchestration scripts, participating scripts, creating input and output tables, and orchestration sequences.

The dependency structure divides migration into separate logical units, illustrated as a downloadable interactive graph with various filter options on the application user interface (UI), which helps to chart a phase-wise migration scope.

Figure 3 – Dependency structure shown in the LeapLogic dashboard.

Actionable and Prescriptive Recommendations

LeapLogic proposes a roadmap with actionable recommendations for schema and orchestration optimizations to migrate from the legacy architecture to AWS.

The roadmap illustrates how the schema can be optimally designed considering native optimizations such as sort keys, distribution keys, and compression techniques. It also analyzes the hierarchy for interdependent jobs to recommend parallel orchestration opportunities for independent jobs. This helps customers strategize the offload program based on data-driven analysis and insights.

Figure 4 – Orchestration and optimization recommendations.

Create a Phase-Wise, Prioritized Blueprint of Your Migration Plan

LeapLogic shares an architectural blueprint based on your choice of AWS tools, data ingestion needs, security and governance requirements, data lake requirements, and consumption.

It recommends an out-of-the-box architectural blueprint for the AWS tech stack, providing an end-to-end picture of how the two services integrate with one another. As a part of the blueprint, LeapLogic shares a proposed future state architecture (see Figure 5) and the evolution from legacy to a cloud data warehouse.

Figure 5 – Current to future state transition using LeapLogic.

Modernization focuses on all the tech stacks across layers. For instance, it migrates data from the legacy systems to Amazon Simple Storage Service (Amazon S3) to create a data lake that can be processed and then moved to Amazon Redshift for analytics.

Similarly, legacy ETLs can be transformed to AWS Glue for ETL-related operations. For governance and security, native services like AWS Glue Data Catalog and AWS Key Management Service (AWS KMS) can be used, which works seamlessly with other AWS products.

To quickly recap, let’s see what we get from an assessment:

Get to know the starting point for your migration.
Create a phase-wise, prioritized migration blueprint.
Strategize your offload program.
Identify the right target technology stack and architecture.
Create a robust and smooth cutover plan.
Ensure zero business disruption with extensive migration and optimization recommendations.

LeapLogic: Beyond SQL Conversion

Many enterprises assume conversion to be just SQL or query conversion, but this is just the tip of the iceberg. To ensure your migration is successful and reuses existing investments, you must migrate all active/used workloads.

Here is how LeapLogic takes care of the complete business logic and workloads, including conditional statements, procedures, and orchestration logic.

Schema Conversion

LeapLogic inherently manages schema and data types on the target side. However, for specific cases like Character Large Object (CLOB) or Binary Large Object (BLOB), it writes data into Amazon S3 and uses that as a reference in a table.

Likewise, where constraints like primary, foreign, and composite keys exist, LeapLogic automatically converts the queries with a distinct clause, helping customers optimize query performance, meet service-level agreements (SLAs), and control costs.

Some other generic examples are as follows:

text > varchar (scale)
nvarchar > varchar (scale)
clob, blob > varchar (scale), data > timestamp
dec > decimal (precision, scale), varbinary > binary

Some examples for Oracle include:

binary_float > numeric (37,15)
national_character_varying > varchar (scale)
long raw > varchar (scale)
float > decimal (precision, scale)

Some examples for Postgres include:

jsonb > varchar (scale)
int8 > bigint

LeapLogic has ready-to-use mappings that automatically map source and target columns to produce the best results. This helps enterprises store the data in the appropriate format leveraging the right data type on the target side.

SQL Conversion

LeapLogic automatically converts SQL dialects and constructs using an intelligent pattern-based conversion engine. For example, Amazon Redshift does not have out-of-the-box support for MERGE statements with sub-queries, so LeapLogic converts that query:

MERGE INTO WAREHOUSE TB1 USING (SELECT INV_DATE_SK ,INV_ITEM_SK ,INV_WAREHOUSE_SK ,INV_QUANTITY_ON_HAND FROM INVENTORY WHERE INV_DATE_SK >40 AND INV_ITEM_SK > 18509578 )TB2 ON TB1.w_warehouse_sk=TB2.INV_WAREHOUSE_SK WHEN MATCHED THEN UPDATE SET W_STREET_NUMBER=TB2.INV_QUANTITY_ON_HAND;

DROP TABLE IF EXISTS default.public.WAREHOUSEWMMergeUpdate;CREATE TABLE default.public.WAREHOUSEWMMergeUpdate (LIKE default.public.WAREHOUSE );insert into default.public.WAREHOUSEWMMergeUpdate select from default.public.WAREHOUSE as TB1 LEFT OUTER JOIN (SELECT INV_DATE_SK, INV_ITEM_SK, INV_WAREHOUSE_SK, INV_QUANTITY_ON_HAND FROM default.public.INVENTORY WHERE INV_DATE_SK > 40 AND INV_ITEM_SK > 18509578) as TB2 on TB1.w_warehouse_sk = TB2.INV_WAREHOUSE_SK;DROP TABLE IF EXISTS default.public.WAREHOUSE;ALTER TABLE default.public.WAREHOUSEWMMergeUpdate RENAME TO WAREHOUSE;

An example of UDF “QUALIFY” conversion is given below:

SELECT StoreID, SUM(profit) OVER (PARTITION BY StoreID) FROM facts QUALIFY SUM(profit) OVER (PARTITION BY StoreID) > 2;

SELECT pid, county, SUM(sale) as total_sale FROM sale_view GROUP BY CUBE (pid, county) order by 1, 2;

Procedural Code Conversion

Apart from SQL, LeapLogic converts procedural logic like Oracle PL/SQL with cursors using a set-based approach to optimize execution and performance. A lot of code which exists in the legacy inventory corresponds to stored procedures. It may contain conditional logic and statements such as loops and if-else statements, which also need to be transformed into the target equivalent as part of the migration program.

LeapLogic uses Union All to handle –Group Cube and Group By Rollup. It also uses a multi-step method to update old records and insert new records.

ETL Conversion

LeapLogic automates component-to-component transformation for ETL scripts, thereby preserving and optimizing the core business logic. Where direct compatibility cannot be established, it does component-to-query conversion.

Once converted, it packages and orchestrates the converted code as production-ready jobs using an extensible, repeatable, and verifiable methodology.

Figure 6 – Example of conversion to AWS Glue using LeapLogic.

Effort Reduction with Automated Validation

Post-transformation, LeapLogic automatically validates the transformed workloads at different levels, which include:

Sample or customer data-based code validation.
Auto-generation of reconciliation scripts.
SQL/query and business-level validation.
Data validation between table-to-table and file-to-file.
Cell-to-cell validation reports for any mismatches or surplus records.
Data type and entity-level matching between source and target.

Figure 7 – Query validation.

Query being the fundamental entity of different types of scripts getting developed needs to be carefully validated, which can confirm its behavior on datasets. This helps to validate and certify and functioning of the query across happy flows as well as corner cases.

The extensive validation ensures consistency of data and logic, source-target parity, and production-readiness confidence, thereby facilitating migration with a lower risk of business disruption.

Conclusion

In this post, you learned key nuances to consider for successfully migrating workloads to ensure lower risk and faster time to market. A comprehensive assessment of legacy workloads and inventory analysis can help you reuse existing investments and identify optimization opportunities for the target cloud, architectural changes, and a detailed plan of the offload program.

After the assessment, automated transformation of the data estate workloads helps minimize the risk of business disruption and fosters collaboration, sustainability, and new business opportunities.

LeapLogic offers a free self-service assessment program for migrating ETL, Hadoop, and data warehouse workloads to AWS Glue, Amazon EMR, and Amazon Redshift, respectively. It performs an automated assessment of legacy systems and provides instant insights on inventory, complexity, and dependencies, along with a detailed cost-effort estimate.

LeapLogic also helps enterprises with automated transformation, validation, and ensures end-to-end operationalization of the workloads on AWS.

To get 360-degree visibility of your current data ecosystem and make informed decisions for modernization, contact LeapLogic. You can also learn more about LeapLogic in AWS Marketplace.

.

.

Impetus Technologies – AWS Partner Spotlight

Impetus is an AWS Advanced Tier Services Partner and software products and services company focused on creating powerful and intelligent enterprises.

Contact Impetus | Partner Overview | AWS Marketplace

AWS Partner Network (APN) Blog