Tracking and remediating non-compliant resources by integrating AWS Config and Atlassian Jira Service Management through automated webhooks

Organizations require their cloud environment to be secure and compliant according to their governance policies. AWS Config provides customers configuration details of their resources in AWS accounts. Customers can make use of AWS Config managed rules, AWS Config custom rules or conformance packs to get to know the configuration details of their resources quickly. Being aware of any configuration changes in their resources, especially resources that are non-compliant against their governance policies is vital for the security & compliance or for the cloud operations team. When the number of AWS accounts and the resources in each of these accounts increases, then being on top of all non-compliant resources becomes challenging. That’s when automation and integrating with IT Service Management (ITSM) tools is helpful for the monitoring and response team.

In this blog post, I will show you how an AWS resource in AWS Config evaluated as non-compliant can be identified in near real-time in an external ITSM tool like Atlassian Jira Service Management. Also show you how to setup an automation rule to trigger an alert to the required team and create an Incident ticket with required severity through the mechanism of automated webhooks.

Webhooks are messages in serialized form-encoded JSON or XML formats, exchanged over HTTP between applications based on event driven triggers. The payload of the messages is typically sent over to a unique URL for the receiving application. While APIs are manual, webhooks are automated and mostly pushed to the receiving application.

Solution Overview

Figure 1: Overview of the solution

The above figure represents an overview of the solution. Enable AWS Config & choose the required rule(s) for evaluating the compliance status of AWS resources. Create an Amazon EventBridge rule with a connection to Jira Service Management as an API destination. In Jira Service Management Project, an automation rule created for the incoming webhook to handle alert & action. Once the monitoring operations team receives an alert and automated incident ticket, they can take appropriate action to turn that non-compliant resource into compliant.

Detailed Walkthrough of the solution

Step 1: Enabling AWS Config and choosing the required rule(s)

Enable AWS Config in your AWS environment for the required AWS accounts. Select the rules that you want to monitor for any changes to AWS resources when it becomes non-compliant. In this example, I choose AWS managed rule “encrypted-volumes” for this setup. Customers can do the same for any of the rules that they require for compliance.

Figure 2: AWS Config managed rule named encrypted-volumes

Step 2: Creating an API Key for AWS in Jira Service Management

Access your Atlassian Jira Service Management portal and choose Administration

In the API Keys under Settings, create an API Key for AWS with suitable expiry date.

Figure 3: Creating an API key in Atlassian Jira Service Management

Step 3: Creating Incoming Automation Webhook in Jira Service Management Project

In Jira Service Management, under the settings of the desired project, choose Automation. Create a rule.

Figure 4: Creating a rule in Atlassian Jira Service Management for automation in project settings.

I am creating an automation rule for an incoming webhook and executing the rule with “No issues from the webhook” as option. Copy the unique Webhook URL that is provided when the rule is created.

Figure 5: The automation rule showing the setup details of the incoming webhook and execution.

Further in the same rule, a new ticket can be automatically created with the details received through the webhook data.

Figure 6: The automation rule showing the setup details of a trigger action.

I am creating an Incident ticket capturing some basic and essential details from the incoming webhook data. The project and the ticket details can be customized based on the requirement & with Jira Service Management features

Summary: AWS Config - Non-Compliant: {{webhookData.detail.configRuleName}} rule affecting resource {{webhookData.detail.resourceId}}
Description: {{webhookData}}
AWS Account Id: {{webhookData.account}}
AWS Creating Time: {{webhookData.time}}
AWS Incident ARN: {{webhookData.detail.configRuleARN}}
Region: {{webhookData.region}}
AWS Status: {{webhookData.detail.newEvaluationResultcomplianceType}}

Step 4: Creating a Connection in Amazon EventBridge

Navigate to Amazon EventBridge and create a connection with the details provided in Step 2.

Figure 7: Creating a connection in Amazon EventBridge for Atlassian Jira Service Management

When creating a new connection, choose API Key as the Authorization Type under Authorization section and provide the API key (the one that was created in Jira Service Management administration in Step 3) name and its corresponding value.

Figure 8: Creating the connection in Amazon EventBridge with the API key obtained from Atlassian Jira Service Management.

Step 5: Creating an API destination in Amazon EventBridge for the connection.

In the same Amazon EventBridge, create an API destination for the connection that was created in Step 4.

Figure 9: Creating an API destination in Amazon EventBridge for Atlassian Jira Service Management with the earlier created connection.

Supply the unique URL obtained from Step 3 for the API destination endpoint and choose the HTTP method as POST.

Figure 10: Confirming the connection and API destination in Amazon EventBridge for Atlassian Jira Service Management

Step 6: Creating an Amazon EventBridge rule (using the Connection & for the API destination)

For the AWS Config rule that was identified in Step 1, Created an Amazon EventBridge Rule using the connection that was created earlier in Step 4 and for the API destination that was created in Step 5.

Figure 11: Confirming the Amazon EventBridge rule with the target as Atlassian Jira Service Management (as the API destination through a connection)

The Event pattern in a JSON looks like this.

{
  "source": ["aws.config"],
  "detail-type": ["Config Rules Compliance Change"],
  "detail": {
    "messageType": ["ComplianceChangeNotification"],
    "configRuleName": ["encrypted-volumes"],
    "resourceType": ["AWS::EC2::Volume"],
    "newEvaluationResult": {
      "complianceType": ["NON_COMPLIANT"]
    }
  }
}

 Step 7: [Optional] Inspecting details of outgoing webhook originating from AWS Config through Amazon EventBridge Rule.

This step is optional, if you want to use it for the purposes of troubleshooting or inspecting the webhook message details.

To verify, troubleshoot and confirm outgoing webhook calls from AWS Config through Amazon EventBridge rule, for which I set up a separate Connection and API destination in Amazon EventBridge similar to how I set up one for Jira Service Management.

Figure 12: A similar API destination in Amazon EventBridge for webhook site for troubleshooting or inspection purposes.

The webhook details looks like below. Its helps capture the necessary information and the webhook content details.

Figure 13: Confirming the webhook site display details for troubleshooting or inspection.

Here is how the raw content of the webhook looks like.

{
  "version": "0",
  "id": "600ffa36-d38f-dcd6-8e17-bff3e72394c1",
  "detail-type": "Config Rules Compliance Change",
  "source": "aws.config",
  "account": "XXXXXXXXXXXX",
  "time": "2023-01-26T20:54:27Z",
  "region": "us-west-2",
  "resources": [],
  "detail": {
    "resourceId": "vol-0d47ce395abf3e14a",
    "awsRegion": "us-west-2",
    "awsAccountId": "XXXXXXXXXXXX",
    "configRuleName": "encrypted-volumes",
    "recordVersion": "1.0",
    "configRuleARN": "arn:aws:config:us-west-2:XXXXXXXXXXXX:config-rule/aws-service-rule/config-conforms.amazonaws.com/config-rule-09h0sc",
    "messageType": "ComplianceChangeNotification",
    "newEvaluationResult": {
      "evaluationResultIdentifier": {
        "evaluationResultQualifier": {
          "configRuleName": "encrypted-volumes",
          "resourceType": "AWS::EC2::Volume",
          "resourceId": "vol-0d47ce395abf3e14a",
          "evaluationMode": "DETECTIVE"
        },
        "orderingTimestamp": "2023-01-26T20:53:57.118Z"
      },
      "complianceType": "NON_COMPLIANT",
      "resultRecordedTime": "2023-01-26T20:54:26.943Z",
      "configRuleInvokedTime": "2023-01-26T20:54:26.797Z"
    },
    "notificationCreationTime": "2023-01-26T20:54:27.320Z",
    "resourceType": "AWS::EC2::Volume"
  }
}

Step 8: Testing a practical use case

To test this solution and setup, I create an unencrypted EBS volume and attach it to an EC2 instance.

Figure 14: An EC2 Instance showing a non-encrypted volume attached to it.

In a few minutes the AWS Config rule re-evaluates rules and triggers the EBS volume as non-compliant.

Figure 15: AWS Config managed rule showing the evaluation results in the console.

The Amazon EventBridge rule gets triggered and the webhook sends the message to Atlassian Jira Service Management.

Figure 16: Amazon EventBridge rule showing the Invocations and TriggeredRules under Monitoring.

Automated Rule for Incoming Webhook gets triggered.

Figure 17: Confirming the triggering of automation rule in Atlassian Jira Service Management through audit log

Step 9: Alert & Action

An Incident Ticket is at once created in Atlassian Jira Service Management for the non-compliant resource.

Figure 18: Confirming a ticket raised in Atlassian Jira Service Management for the non-compliant resource

An Email Alert is also sent to the Ticket Assignee for requesting appropriate action

Figure 19: A corresponding email sent as an alert to the ticket assignee.

Based on the alert and the ticket details, the ticket assignee for the incident response can take immediate action on the Non-Compliant AWS resources in short time. This solution can help the operations team remain compliant to the organization’s governance policies at all times proactively.

Step 10: Multi-account and multi-region setup

Customers typically have more than one AWS account based on their line-of-business, team function or workload purposes. The compliance findings from multiple regions and multiple AWS accounts can be consolidated by the use of aggregator in AWS Config. An aggregator in AWS Config can bring all the compliance findings into a single view. The above discussed EventBridge rule can setup in the same account where the customer has the aggregator setup in AWS Config, for ease of administration and to cover multi region and multiple accounts.

Figure 20: Schematic representation of multi-account multi-region aggregation in AWS Config through Aggregator

Here are the high level steps to accomplish multi-account multi region aggregation findings and triggering EventBridge rule for appropriate actions.

1. Create an Amazon EventBridge Custom Event Bus in the Management Account where the Config Aggregator is created.
2. Create an Amazon EventBridge Rule for the Custom Event Bus in the same Management Account.
3. The target for the EventBridge rule in the Management Account can be another AWS Service or an external SIEM Tool for appropriate action.
4. In the required Member accounts, Create Config rule to evaluate resources for non-compliance.
5. Setup EventBridge rule in the member accounts connecting the config rule finding and the EventBridge Custom Event Bus in the Management Account created in first step.

The below flow diagram simplifies the above-mentioned steps.

Figure 21: Schematic representation of multi-account multi region aggregation findings and triggering EventBridge rule for appropriate actions

The Resource-based policy for the Management Account Custom Event Bus should look like below. Replace the <ExistingEventBusARN> and <OrganizationID> below.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "CustomEventBusStatement",
    "Effect": "Allow",
    "Principal": {
      "AWS": "*"
    },
    "Action": "events:PutEvents",
    "Resource": "<ExistingEventBusARN>",
    "Condition": {
      "StringEquals": {
        "aws:PrincipalOrgID": "<OrganizationID>"
      }
    }
  }]
}

Figure 22: Confirming an Amazon EventBridge custom bus and its permissions in the console

Cleaning up

If you no longer need this solution, delete the Amazon EC2 Instance, Amazon EventBridge rule, AWS Config rules, AWS Config and revoke API Keys in Atlassian Jira Service Management to avoid incurring charges for the resources that were created following this blog post.

Conclusion

In this blog post, I showed how you can integrate AWS Config and Atlassian Jira Service Management to track and remediate non-compliant AWS Resources through automated webhooks (with Amazon EventBridge rules). Using this solution, you can use Jira Service Management as a single place to track and remediate non-compliance AWS resources while leveraging your existing workflows in Jira.

About the author: