Automate drift remediation and compliance with AI

This article is part of a multi-part Amazon Web Services (AWS)–authored series that explores the use of modern DevOps tools and AI-based features to accelerate software development and maintenance on AWS. The first article in this series introduces a fictional scenario, with detailed background on the example.com application and the team managing it. This article focuses on the operate phase of the software development lifecycle (SDLC). You will learn how modern functionalities and AI-based features provided by independent software vendors (ISVs), when used alongside AWS services, can help you implementautomatic drift detection and remediation and automated policies that prevent non-compliant infrastructure changes from being introduced into the production environment.

Figure 1: The modernization journey of the example.com application

Background - You are a site reliability engineer (SRE), part of the central SRE team that supports multiple projects, including the example.com application. The application has been going through an overhaul. The engineering team has been transforming the highly coupled architecture into a distributed microservices-based architecture. The team has also been implementing and releasing customer-requested features. It is also being made Payment Card Industry Data Security Standard (PCI DSS) compliant, as it handles credit card payments. Soon, the feature development will be complete, and the application will move back into maintenance mode. Once it moves into maintenance mode, you and the central SRE team will be responsible for the example.com application's infrastructure. Your role will include activities such as system monitoring, incident response and resolution, performance optimization, patching, and ensuring that the application infrastructure stays PCI DSS compliant.

Recently, an on-call SRE intern opened up port 22 in the production environment and accidentally deleted an Amazon Elastic Compute Cloud (Amazon EC2) instance that refreshes the recommendations model. If the application were already PCI DSS compliant, such an activity would have triggered a compliance incident due to infrastructure non-compliance. When you discuss this further with your team, you discover that similar issues have occurred in other projects as well. You must solve the problem by implementing automated systems to address these two requirements:

1. Automated drift detection and remediation
   Although the SRE team needs broader access and permissions to operate the live environment, a mechanism should be in place to detect any changes made to the higher-level environments outside the standard GitHub-driven workflow. The mechanism must restore the application to its last known desired state. This would help reduce customer impact from misconfigurations and manual errors.
2. Implement policies to prevent non-compliant infrastructure changes to production environment
   The SRE team must ensure that the application infrastructure continues to remain PCI DSS compliant and must catch any non-compliance related to infrastructure before it reaches production. Every infrastructure change released to the production environment must be reviewed for compliance.

By solving these two problems for example.com, you can showcase a real-world POC to your senior SRE leadership and scale the approach to other applications your team maintains.

Use case 1: Catch unauthorized changes and fix them automatically

IT teams implement strict gates and limit exposure to higher-level environments, so even tenured engineers do not have access to production systems, and it pays off. By limiting user access, you improve the stability of the application as it is left untouched most of the time. However, certain personas need to have access to higher-level environments because it’s part of their job. For example, an on-call SRE might need to fix something to limit further damage quickly, or the SRE might be changing some configurations to diagnose a problem that’s hard to reproduce in lower-level environments. These changes may result in only a temporary fix, a new problem that impacts the customer, or some other undesirable outcome. It is not a good idea to apply changes to the environment outside your standard deployment process.

While the SRE team leadership mandates processes to keep the production environment in a good state, you need to go one step further to mitigate the risk of human error to ensure that production environment changes go through the right process. Whenever the production environment deviates from its ideal state, typically, the last successful and verified deployment, you must detect the drift and take action to remediate it. As the SRE, you must implement automated safeguards to restore your application, ensuring it quickly returns to its ideal infrastructure state.  

Having learned about recent advances in this area, you start exploring infrastructure management platforms that support automated drift detection and infrastructure reconciliation. You come across Spacelift, an infrastructure orchestration platform.

About Spacelift

The Spacelift orchestration platform supports workflow-based infrastructure provisioning for popular IaC tools you may already be using such as Terraform, CloudFormation, Pulumi, OpenTofu, and Kubernetes. It supports version-control-system-based automated infrastructure provisioning by combining source code, the current infrastructure state, and configuration. Spacelift enables IT teams to gain insights by providing resource visibility, governance with predefined policies, and automated drift detection.

You decide to explore whether Spacelift can work for you by doing a quick POC that emulates the recent incident. You sign up for a free trial and then use Spacelift to provision six Amazon EC2 instances. As you can see in Figure 2, when you navigate to the stacks section of the console, you see all six Amazon EC2 resources listed.

Figure 2: EC2 resources created from the Spacelift platform [Spacelift Drift Detection Overview]

Next, you configure a drift detection schedule that runs every minute and checks whether infrastructure provisioned as part of the stack has drifted from its last recorded state. You associate the drift detection schedule, as shown in Figure 3, with ec2 stack, so whenever there is any change made to the stack outside the Spacelift platform, the Spacelift platform detects the change within a minute.

Figure 3: Drift detection schedule configured for the ec2 stack

To test it out, you delete one of the Amazon EC2 instances monitored by the stack. When you navigate back to the Spacelift console, as shown in Figure 4, you see that an Amazon EC2 instance has drifted from its desired state.

Figure 4: Spacelift detects drifted Amazon EC2 instances and shows details in the section on the right

When you open the tracked runs section of the Spacelift console, you notice that a new tracked run has started in response to the drift. As shown in Figure 5, the tracked run prepares changes required to reconcile the infrastructure to its original state.  However, since you have not configured auto-deployment, it is waiting for your confirmation.

Figure 5: Spacelift reconciles the drifted Amazon EC2 instance

As soon as you provide the confirmation, the Amazon EC2 instance is re-provisioned, and the stack returns to its original state.

By implementing automated drift detection and remediation for your infrastructure, you can ensure that any non-standard changes made to the production environment outside the infrastructure provisioning platform are quickly detected and remediated. This helps you reduce customer impact from unauthorized changes made to your production environment.

Use case 2:  Implement policies to prevent non-compliant infrastructure changes to production environment

Compliance is a regulatory requirement that ensures your application follows the necessary processes to keep customer data safe. However, failing to follow these requirements can lead to legal action, hefty fines, and a decline in customer trust. Because example.com handles credit card data, ensuring the application remains PCI DSS compliant is a legal requirement.

As the SRE team, it falls on you to ensure the application infrastructure remains compliant and that any infrastructure non-compliance is caught before it reaches production. You must monitor and review every change going into the production environment and confirm that it meets compliance requirements. However, manual policy compliance reviews take time, which can delay a deployment. Additionally, having to understand how a regulatory standard affects infrastructure changes requires extensive training and muscle memory to be built within the SRE team. Implementing a comprehensive manual approach can be challenging and more error-prone.

You need a solution that reviews infrastructure changes and flags those that do not meet infrastructure related compliance requirements before the pull request is merged into the main branch. It must also plug into your GitOps-driven infrastructure management CI/CD workflow.

You start exploring and come across the ControlMonkey platform, which lets you automate infrastructure compliance checks with out-of-the-box policies.

About the ControlMonkey platform

The ControlMonkey platform enables you to discover, manage, and automate cloud environments. It seamlessly integrates infrastructure compliance policy bundle checks into your GitOps-based infrastructure deployment workflow, enabling you to automatically check whether your infrastructure changes are meeting policy requirements.

You decide to do a quick POC in example.com’s developer environment by emulating a recent incident.

You sign up and then configure your AWS account within the ControlMonkey platform. Since you do not want to change any existing configurations, you use the ControlMonkey platform’s import functionality to create a new managed resource within the ControlMonkey platform. The Terraform Import console, as shown in Figure 6, shows multiple resources you can import to form a new stack.

Figure 6: ControlMonkey platform Terraform import console [End to End IaC Automation Platform by ControlMonkey platform]

After selecting relevant resources, the ControlMonkey platform provides the Terraform code and a state file, with the option to push the same code to a new repository. After pushing the code to a new repository, you configure control policies so that every time a pull request is created in the managed repository, the ControlMonkey platform policy reviews the code changes and confirms whether they meet the policy requirements for infrastructure application is deployed on.

To test it further, you commit some infrastructure changes to the managed repository while including the change made by the SRE intern that opens port 22. As soon as you create a pull request, the ControlMonkey platform adds a summary of tests and test results. As you can see in Figure 7, the ControlMonkey platform executes several relevant tools and includes the result in a comment. It identifies that the changes in the pull request are failing three control policies, making the application infrastructure non-compliant.

Figure 7: ControlMonkey platform response outlining several policy requirements as part of the GitHub pull request workflow

When you expand the Control Policies section, you see details of the individual tests that failed. As you can see in Figure 8, the creator tag was missing for some resources, and the security group was modified to allow traffic on ports 22 and 3389. The ControlMonkey platform policies specifically note that these changes result in a PCI DSS V3.2.1 compliance incident due to infrastructure changes.

Figure 8: Pull request response outlining PCI DSS policy violations requirements

By implementing automated policy enforcement at the pull request stage, you can catch and remediate issues early and ensure that the production environment stays clean.

To summarize, with the ControlMonkey platform’s out-of-the-box policies, you can automate compliance checks as part of the GitHub workflow, which helps you keep your production environment compliant.

This article explained how implementing an AI-augmented approach while keeping humans in the loop can help you address the following use cases efficiently:

1. Write IaC scripts to automate complex infrastructure provisioning pipelines and speed up software maintenance tasks like version upgrades using AI agents.
2. Separate AWS-native code deployment from feature releases while supporting A/B testing.
3. Combining third-party software solutions from AWS Marketplace with your AWS-native tooling to achieve efficiency in the deployment stage.

 

Figure 9: Improve deployment phase of the software development lifecycle using modern AI-based tools

While this article introduces two solutions, AWS Marketplace is a curated digital storefront, comprised of Agentic and Generative AI third-party software that can help you make the operate stage more efficient. Organizations looking for ways to boost efficiency by integrating AI into various aspects of their SDLC, can explore these tools further, conduct POCs, and then procure the software in AWS Marketplace.

Note on procuring third-party software for your team - Procuring third-party software is made easier in AWS Marketplace because enterprise organizations save procurement time and costs using standard contracts, flexible pricing models, custom pricing and term negotiations with vendors and consulting partners, and consolidated license and spend management.