Automate Your AWS DevOps Validation Pipeline with the Infosys Cloud Infrastructure Validation Solution

By Aalok Prakash Pathak, Sr. Project Manager – Infosys
By Arvind Sundarraman, Engagement Manager – Infosys
By Nidhi Chauhan, Technical Test Lead – Infosys
By Ashutosh Pateriya, Solution Architect – AWS

Infosys

Spurred by the global pandemic, we have seen many organizations shift towards cloud computing to ensure the stability and reliability of their IT applications.

During such migrations, organizations typically focus on the validation of business applications and functional workflows, along with non-functional aspects like security and performance testing.

An often-overlooked aspect of a migration journey is the configuration checks on the underlying cloud infrastructure. In order to make sure the foundational infrastructure is safe, secure and compliant, there is a need to validate the cloud configuration early in the migration cycle.

Further studies suggest most of the breaches that have been observed in cloud environments are primarily due to opportunistic attack, with data left open because of failure to properly configure the cloud platform settings, and not by malware.

Infosys, an AWS Premier Consulting Partner with vast experience in cloud migration and testing, has developed an innovative automation solution. It addresses the need for secure configuration reviews, while ensuring agility and reliability through the migration journey.

In this post, we will discuss how Infosys has taken a secure approach on holistic validation of cloud configuration by performing automated infrastructure validation as part of the DevSecOps pipeline.

This has been achieved by incorporating security and compliance rules early in the lifecycle, thus facilitating the establishment of foundational infrastructure for safe, secure, and compliant cloud adoption.

Cloud Quality Assurance Challenges

A leading U.S. financial services provider embarked on a journey to develop and mature the foundational capabilities for sustainable cloud adoption.

As a quality guard, the Infosys Quality Assurance team developed an automation solution covering more than 15 AWS services and 150+ compliance rules. This was integrated into and then executed as a “stage” in a DevSecOps pipeline to continuously validate cloud infrastructure being provisioned across the environments.

The key challenges were ensuring a proper integration into a DevSecOps provisioning pipeline, and to ensure agility in the overall lifecycle during the migration.

Key aspects that were taken into consideration included:

Time to market: Service rollout is adversely affected if the infrastructure testing remains manually invoked, while infrastructure as a service and application layering is automated.
Infrastructure validation: Validating infrastructure as code (IaC) helps to prevent misconfigured provisioning, which helps avoid security risks and stops the solution from incurring unexpected costs.
Migration complexities: Coverage for mixed and varied formulated migration strategies.
Evolving technologies: Scope to include diverse set of services and features provided by the cloud provider.
Regulatory compliance and security control: Various audit and compliance requirements to be considered for cloud implementation and application migration.
Security practice: Since this solution is interfacing with the customer account, role-based controls are necessary to provide least privileged access to perform the validation. Also, the data at rest needs to be encrypted along with encryption in transit.

Accelerating Configuration Validation with Automation

An approach that relied on automation at the core and integration into DevOps was applied to develop test automation solution using AWS services.

Key principles that were considered included:

Zero touch approach: All steps involved in infrastructure testing including test execution, defect management, and test reporting, are completely automated.
DevSecOps integration: Ensuring the inputs flow in a DevSecOps pipeline required for triggering infrastructure testing stage to avoid any manual intervention.
Fit for purpose tooling: Identifying suitable tools and programming, scripting languages for infrastructure test automation solutioning based on availability in an organization’s technology landscape.
Automated security controls: Comparing deployed infrastructure against security best practices and providing action items as part of the report.

Solution Architecture

The test automation solution developed and integrated can parse the Terraform output files to retrieve the AWS resource information and automatically trigger the test controls based on the resource type.

Figure 1 – Cloud infrastructure test automation in the DevSecOps pipeline.

The following details outline the automation workflow and highlight AWS services and third-party products that helped Infosys establish the cloud infrastructure testing solution.

User submits infrastructure requirements using in-house developed Provisioning Portal.
Requirements captured in the Provisioning Portal are converted to JSON to feed AWS Step Functions. It’s an orchestrator to manage infrastructure provisioning as well as testing stage in the pipeline.
Overall, the solution includes three key steps detailed in the sections below, and the key services leveraged across each steps are:
- AWS Lambda: This was used to execute test scripts corresponding to various service functions (service validation, Jira defect management, Amazon S3 log forwarding) within infrastructure testing stage.
- Amazon EC2: This provides compute capability and multiple EC2 instances are used which host Jira, Kibana, AWS SDKs, and RubyGems to facilitate execution of test scripts.
- AWS KMS: In order to address encryption, customer managed Key Management Systems (KMS) keys are used to securely store and transit test logs. Also, integration of KMS with other AWS services is leveraged to encrypt an Amazon Elastic Block Store (EBS) volume attached to EC2 instance and S3 bucket for securely maintaining test logs at rest, as well as in transit while forwarding to S3.

Step 1: Infrastructure Test Automation Solution

A Lambda function copies validation scripts from the Bitbucket and loads into Amazon Elastic Compute Cloud (Amazon EC2).
AWS Security Token Service (STS) in conjunction AWS Identity and Access Management (IAM) are leveraged to assume the required role and establish connectivity with target AWS account against which validation needs to be performed.
Infrastructure test automation is developed, deployed on EC2 using AWS SDK for Ruby to retrieve information about provisioned AWS resources. This also validates against security compliance and standard requirements defined by the organization.
The test output generated in Step 1 is maintained and stored in EBS for Jira and Kibana dashboard consumption.
Amazon CloudWatch is used for runtime monitoring and debugging of infrastructure testing. This provides detailed insights to the sequence of events related to all three steps of infrastructure testing stage.

Step 2: Automated Jira Defect Management

The second Lambda function triggers and uses test output placed in EBS volume from Step 1 for automated defect management in Jira. AWS Systems Manger Parameter Store is used to securely maintain and encrypt credentials required for Jira connectivity.
Leveraging Jira Rest API and in-house developed RubyGem deployed on EC2, the solution automatically manages defects (create/update/close) in Jira based on test results.

Step 3: Automated Test Metrics Dashboard

The third Lambda function triggers and moves test logs placed in EBS volume from Step 1 to Amazon Simple Storage Service (Amazon S3), which is used as persistent storage for maintaining test logs for Kibana consumption.
Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) is used as a search and analytics engine for data indexing and interacts with Kibana. A wide range of test and adoption metrics are designed and implemented in Kibana.
AWS services (test coverage): This infrastructure test automation solution is providing validation for more than 15 AWS services. More services and automated checks are added to the list based on new, updated requirements from time to time.

Integration within the DevOps Lifecycle

There are multiple stages involved in cloud deployment, and an effective validation approach calls for clearly-defined boundaries between multiple stages and distinct coverage by the testing stage after infrastructure provisioning.

Figure 2 – Infrastructure provisioning stages and testing.

Pre-Provisioning

Linters are integrated into a DevSecOps pipeline before the provisioning stage to scan the infrastructure code and identify any syntactic or semantic errors, deviations from security compliance, or configurations requirements.

In order to avoid the provisioning of incorrect infrastructure, the Linters must stop the DevSecOps pipelines if any issues are uncovered during scan. Once all of the identified issues are fixed and Linter’s scans returns success code, the pipeline should proceed with infrastructure provisioning.

Provisioning

The approach to managing infrastructure is similar to managing an application. The principle is to treat configuration of infrastructure like the software source code is treated.

Cloud technology has wide adoption of using IaC for managing and provisioning infrastructure (like Terraform, Python, and AWS CloudFormation). This stage gets invoked upon successful completion of pre-provisioning and hands over the control to next stage if the success code is returned.

Post-Provisioning

This stage gets invoked upon successful completion of infrastructure provisioning and provides third-party validation of automated cloud infrastructure provisioning. The test automation is developed leveraging Chef InSpec Framework using AWS SDK along with Ruby scripting language.

Chef InSpec is an open source framework for validating infrastructure by describing security and compliance rules. It works by comparing the actual state of provisioned infrastructure with the desired state. This automation validates a wide range of AWS resources being provisioned during build time for security compliance, standards and configurations checklist.

Defect Management and Reporting

As the adoption of DevSecOps pipeline increases across the organization for provisioning infrastructure covering every environment, it starts detecting high number of failures, specifically for test environments, and managing these huge number of issues becomes challenging.

In order to address this challenge and bring completeness to the solution, Infosys developed the in-house capability to automatically create defects in Jira for the failures traced during post-provisioning.

This automation is developed in Ruby scripting language using Jira Rest API to create, update, and close the defects with appropriate comments, and then attaches related logs based on test results. The automation is exported in the form of RubyGem that can be used within and outside DevSecOps pipelines.

The test results of the automated provisioning are stored temporarily in the orchestrator workspace (Jenkins, for instance). Once all of the AWS resources are validated, the test results are copied to an S3 bucket (used for persistent storage).

The Log Stash component periodically scans S3 for new files sets, filters for required values for building test metrics, and transports the data to Kibana using Amazon OpenSearch Service.

A wide range of test and adoption metrics could be drawn using the infrastructure test results in tools like Kibana and Power BI. Some of the key metrics are:

Number of applications validated per platform.
Number of AWS resources validated per application per platform.
Number of test cases executed per application per platform.
Number of failed test cases vs passed test cases.
Number of DevSecOps pipelines adopting infrastructure testing.
Compliance Summary: Number of security violations against violation severity.

Benefits Delivered

With the detection of more than 3,400 defects in the lower environments during the course of a migration journey, the test automation solution helped to reduce additional budget overruns on support due to defects and issues spotted later in the cycle and in production.

It also resulted in foundational infrastructure with improved security, higher operational efficiency, and tighter regulatory compliance before applications are migrated to cloud.

Infosys, in its own way, developed an effective and efficient cloud infrastructure test automation solution having key benefits:

Compliance and control: Improved security and compliance posture without losing agility.
Deep coverage: Across all layers of the infrastructure stack required for an application to deploy.
Reduced costs: Prevented misconfigured infrastructure provisioning. Uncovered thousands of defects across all environments.
Independent of provisioning IaC technology.
Configurable for conditional execution: Select the specific set of test cases related to services being provisioned.
Reusability: Leveraged within or outside DevSecOps pipeline.

Conclusion

Infrastructure is now software in the cloud era; it needs to be treated as code and the need of infrastructure testing is crucial to deliver cloud deployments with expected quality.

At the end, it comes down to whether cloud resources being deployed are configured the way they need to be and are good to consider for baseline. Step left and take an effective and holistic approach to cloud infrastructure validation.

Leverage the foundational capabilities of Infosys to perform continuous validation and provide early and constant feedback about infrastructure quality in a structured and efficient way.

If you’re considering migrating to the AWS Cloud and looking for a testing partner, contact Infosys Infrastructure Testing Services for more information.
.

.

Infosys – AWS Partner Spotlight

Infosys is an AWS Premier Consulting Partner that helps enterprises transform through strategic consulting, operational leadership, and co-creation of solutions in mobility, sustainability, big data, and cloud computing.

Contact Infosys | Partner Overview

*Already worked with Infosys? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.