Cognizant’s Optimized Approach to Modernize Batch Implementations on AWS

By Dipanjan Sengupta, Sr. Director, Technology – Cognizant
By Pijush Kanti Giri, Architect – Cognizant
By Pratip Bagchi, Sr. Partner Solutions Architect – AWS

Cognizant

Enterprise customers are modernizing existing workloads to cloud for various reasons.

Cost optimization has always been a conversation starter, but that leads to application modernization and operational excellence, which brings agility to application development and enables faster go-to market (GTM).

The evolution of the cloud service providers has helped modern applications achieve fluctuating and high demands of infrastructure, software-defined networking and storage, and seamless integration between various managed services.

In a previous AWS blog post, we outlined a detailed way to modernize an integration platform on Amazon Web Services (AWS). Now, we will describe how to bridge the gap between the infrastructure-as-a-service (IaaS) and integration-platform-as-a-service (IPaaS) platform models for modernizing batch processing applications on AWS.

From our experience, this can lead to 60% performance efficiency and 40% operational cost reduction by leaving the IPaaS provider’s complex license model behind.

In the scenario outlined in both blog posts, Cognizant’s customer is one of the world’s most-used business credit report providers, based in Europe and serving 365 million businesses worldwide.

Before this modernization effort, data pertaining to companies and business institutions from various countries and fetched from different data sources are processed and loaded into the customer’s database, and made available to various downstream systems.

Cognizant is an AWS Premier Tier Services Partner with six AWS Competency designations. Cognizant is also a member of the AWS Managed Cloud Services Provider (MSP) and AWS Well-Architected Partner Programs.

Existing Architecture

The customer’s existing batch processing application implementation and delivery were accomplished using an integration software from the IPaaS provider and was hosted on their platform.

The source data from different countries were loaded into on-premises database tables. For every country, there was a different schema available and new data was available in the relevant tables on a daily, weekly, or monthly basis depending on the country.

Cron schedulers were used to initiate the batch process for each country on a daily basis. Source data for a country, if available, was extracted from the country-specific tables and loaded into the staging tables. After extraction, sanitized data from the staging tables was inserted into different target tables.

Broadly, the batch process consists of following activities:

• Cron scheduler: Cron scheduler initiates the batch processes for the various countries on given schedules.
• Cleanup stage: Existing data in the staging tables are deleted before any new data is inserted.
• Extract stage: Source data is fetched from different on-premises source databases.
• Load data stage: Extracted data is loaded to different staging tables
• Load translated data stage: This is an optional stage, where data in foreign (non-English) languages are translated before further processing.
• Delete duplicate data stage: Duplicate records in the staging tables are deleted.
• Load target data stage: In this final stage, sanitized data from the staging tables are loaded into the target tables.

Implementation Scope

The decision to migrate to AWS was made by Cognizant’s customer and driven by the need to scale out with the growing demand and to reduce operating expenses.

Integration platforms typically support a set of components and constructs to define and specify integrations flows using a domain-specific language (DSL) that is unique to the platform.

The applications in our modernization scope were no exceptions:

• Handle multiple extract, transform, load (ETL) batch jobs running concurrently, but at the same time ensure that if a given entity’s batch process is already running then the system shouldn’t start another batch process for the same entity.
• For individual jobs running, we have to capture the job-level statistics.
• Handle approximately 50 million records for every batch job.
• Enabling a new entity’s batch process should be seamless and not require a new production deployment.
• Create a generic and reusable notification mechanism to the customer’s dedicated communication channel.
• Handle compute environment-related failures and develop a notification mechanism to notify the failures.

The solution required business information from 30 different countries (regions) to be consumed as a part of an ETL process. The solution also had to handle several new features:

• Ensuring batch processing for a given entity (country or region) does not overlap at any given time.
• Provide job-level custom metrics.
• Resume processing of failed jobs from the point-of-failure of a batch process.
• Handle approximately 50 million records of data for each entity.

Also, keeping the architecture configuration-driven means that to enable/disable any new batch process requires zero code changes or deployment. Only configuration changes and uploading specific SQL query files would suffice to integrate with this system.

With a wide array of managed and infrastructure services to choose from for compute, network, and storage, designing an elastic, highly available, cost-effective, and scalable solution architecture with all the right components in place was a critical success factor for the modernization project.

Solution Architecture

The modernized solution uses AWS Lambda and AWS Batch Compute on AWS Fargate, a fully managed serverless compute service with workload-aware cluster scaling logic, maintaining event integrations and managing runtimes.

We created compute jobs running as Docker containers and handled the orchestration layer using a serverless architecture.

The solution depends heavily on the availability and reliability of configuration documents for each of the entity’s batch processes. It leverages Amazon Simple Storage Service (Amazon S3), a fully managed object storage service that offers industry-leading scalability, availability, security and performance.

It also leverages the Amazon DynamoDB database, a fully managed, multi-region, multi-active, durable NoSQL database solution.

Figure 1 – Architecture diagram.

The following AWS components were used in the architecture, and here is the context in which they have been used in the solution:

• Amazon DynamoDB is used as an idempotent store to track execution status of different entity’s batch jobs, as well as to maintain the batch jobs audit records and job statistics data.
• AWS Lambda is used to trigger AWS Batch jobs submission to job queue according to defined job dependencies. It checks Amazon DynamoDB for the current batch jobs execution status and then submits the jobs accordingly.
• AWS Batch is used to dynamically provision the optimal quantity and type of compute resources based on volume and specific resource requirements of the batch jobs submitted.
• Amazon Simple Queue Service (SQS) helped to decouple and scale distributed systems and serverless applications. It’s used to de-couple the different AWS service integrations and allow the messaging process between different components.
• Amazon Simple Notification Service (SNS) is used for sending notification messages to Teams channel via Lambda integration.
• Amazon RDS for MySQL is used as the target database system for the data loading jobs.
• Amazon CloudWatch is the primary monitoring solution for applications and infrastructure, providing operational data in the form of logs, events, and metrics. The different entity-specific cron-expressions are configured as CloudWatch Event Rules to trigger the controller layer to initiate the batch jobs for given entity.
• Amazon S3 is used to store the different entity-specific SQL query files, thereby externalizing the queries. It’s also used to store the extracted data as compressed CSV zip file, which is then consumed by various other batch jobs.

Conclusion

In this post, we have shared an enterprise customer case study and viable strategy for modernizing batch processing implementations, migrating from a specialized platform-as-a-service provider to a highly available and scalable platform using AWS.

In this application modernization journey, Cognizant used the right selection of services and functionalities for compute infrastructure and managed services to build a robust, flexible, and high-performing architecture while working backwards from the customer’s challenges and requirements.

Adopting a successful migration approach and using the right set of reusable components, tools, and techniques is important to accelerate the delivery of modernized platforms, while maintaining a high level of design time governance.

.

.

Cognizant – AWS Partner Spotlight

Cognizant is an AWS Premier Tier Consulting Partner and MSP that transforms customers’ business, operating, and technology models for the digital era by helping organizations envision, build, and run more innovative and efficient businesses.

Contact Cognizant | Partner Overview

*Already worked with Cognizant? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.