Automating Remediation of Amazon GuardDuty Findings with Dome9 CloudBots

By Sameer Kumar Vasanthapuram, Partner Solutions Architect at AWS
By Pawan Shankar, Sr Product Marketing Manager at Dome9

When Amazon Web Services (AWS) customers turn on Amazon GuardDuty and start seeing security findings roll in, you may not immediately understand how a resource was affected, the attack vector, and how you can prevent it from happening again by taking the appropriate remediations.

Dome9 Security’s integration with GuardDuty brings to the table a way of surfacing these findings, providing context and creating automated remediations.

Bringing together the capabilities of Dome9 and GuardDuty, users that identify a finding can look through their Dome9 console and pinpoint the exact instance, Virtual Private Cloud (VPC), and security group associated with it. This helps customers identify the compromised instance, as well as potential instances that may have a similar posture, thereby allowing you to mitigate the risk before exposure.

In this post, we’ll explore how to identify and remediate threats identified by GuardDuty with Dome9 CloudBots.

CloudBots are a set of automatic remediation solutions built on top of Dome9’s continuous compliance capabilities. They are essentially a library of AWS Lambda functions that enable you to take action on specific violations and reduce the time to resolution for these events.

To set up CloudBots, navigate to the Dome9 CloudBots Github and follow the onboarding steps.

What is Amazon GuardDuty?

Amazon GuardDuty is a threat detection service that continuously monitors for malicious or unauthorized behavior to help customers protect their AWS accounts and workloads.

GuardDuty monitors for activity such as unusual API calls or potentially unauthorized deployments that indicate a possible account compromise. It also detects potentially compromised instances or reconnaissance by attackers.

GuardDuty analyzes continuous streams of metadata generated from a customer’s account and network activity found in AWS CloudTrail Events, Amazon VPC Flow Logs, and Domain Name Service (DNS) logs. It uses integrated threat intelligence, such as known malicious IP addresses, anomaly detection, and machine learning, to identify threats more accurately.

As customers enable GuardDuty and findings start to roll in, you will need to identify the resources that are under threat, undertake remediations, and make sure to identify the attack vector and shut it down.

Who is Dome9?

Dome9 Security offers a comprehensive security and compliance automation solution purpose-built for the cloud.

As an AWS Partner Network (APN) Advanced Technology Partner, Dome9 has built rich integrations with the security services AWS offers, including AWS Config, AWS CloudTrail, VPC Flow Logs, GuardDuty, and Amazon Inspector. Dome9 has AWS Competencies in both Security and Networking.

Identifying the Resource Under Attack

Dome9’s Entity Explorer view provides security context, such as network policies and Identity and Access Management (IAM) privileges, for a specific instance from a single, asset management view.

The Entity Explorer combines GuardDuty findings in the same view to provide additional threat detection insights for a specific instance. Now, you can see the security policies and IAM policies, as well as host vulnerabilities and threat detection alerts, in a unified management pane.

Figure 1 – Entity Explorer view surfaces security groups, inbound rules, and NACLs associated to an instance.

Within Entity Explorer, you can drill down into GuardDuty findings on the Alerts & Findings tab.

Figure 2 – Use the Alerts & Findings tab to drill down into GuardDuty findings.

In addition to the ability to drill down to specific resources in the Entity Explorer, you can visualize your entire configuration and assess the security posture of your VPC the instance belongs to. This allows you to check if there are any security misconfigurations or configuration drift that has occurred and resulted in the anomalous behavior.

Clarity View on Dome9 provides a network topology based on how security groups are configured and easily shows if an instance is exposed to the public internet or not.

Figure 3 – Clarity View provides a network topology based on how security groups are configured.

Remediation with Dome9 CloudBots

Now that we have surfaced findings and can drill down to the resource and understand the context around its security posture, we can identify methods to automate the remediation as findings roll in.

Here, we’ll specifically explore a malicious IP talking to an Amazon Elastic Compute Cloud (Amazon EC2) instance in your environment.

Figure 4 – The AWS console shows GuardDuty findings for all resources within an environment.

On the GuardDuty console, you can specifically filter and drill down into the finding related to the compromised instance. Choosing a specific finding allows you to get more information on the specific resource affected, but it does not provide context around why that specific resource was affected.

Figure 5 – Choose a specific finding to get more information on the specific resource affected.

When situations arise, the most important thing is to have a predefined security playbook or runbook that can be a guiding process for incident response and remediation.

You can achieve this by deploying Dome9 CloudBots that can take predefined remediation actions for specific types of GuardDuty findings.

CloudBots Remediation Architecture

GuardDuty findings are sent to an Amazon Simple Notification Service (SNS) topic via CloudWatch Events, to which the Dome9 CloudBot Lambda Function subscribes to.

The following actions are taken by Dome9 CloudBots code:

1. GuardDuty identifies malicious activity and sends its finding as a CloudWatch Event.
2. Parse the SNS notification that’s sent to the subscribed SNS topic.
3. Transform the GuardDuty SNS notification into a Dome9 finding format (both are JSON).
4. Check the finding type against the GD_ACTIONS environment variable in Lambda and launch specific CloudBot, if defined.
5. Remediate with predefined action,

Figure 6 – Dome9 CloudBots remediation architecture for GuardDuty findings.

Parsing SNS Notifications

The index.py code parses the GuardDuty event from SNS and looks for the message value. As long as the source of the finding is an “aws.guardduty” finding, it calls GD_transform_event module:

### code snippet###
{
  #Bring the data in and parse the SNS message
def lambda_handler(event, context):
    raw_message = event['Records'][0]['Sns']['Message']
    source_message = json.loads(raw_message)
    try: 
        try:
            source_message = json.loads(raw_message)
        except: # If the event comes through as a dict, take it as it comes
            source_message = raw_message
        # Check for source. Transform it to "Dome9" format if it's not originating from Dome9. 
        # This expects that GD is triggering lambda via SNS. This is needed for running cross-region GD events. 
        if "source" in source_message and source_message["source"] == "aws.guardduty": # GuardDuty event source via CW Events
            text_output_array.append("Event Source: GuardDuty\n")
            gd_transform_module = importlib.import_module('transform_gd_event')
            found_action, text_output, source_message = gd_transform_module.transform_gd_event(source_message)
            text_output_array.append(text_output)
            if not found_action:
                print(text_output_array)
                return      
}

Transform GuardDuty Findings

This GD_transform_event module parses the SNS event and transforms the message (transform code below) into a Dome9 format to allow appropriate action to take place:

### code snippet###
{
 def transform_gd_event(unformatted_message):
    found_action = False
    formatted_message = ""
    text_output = ""
    #Check the OS variables to get the list of what we want to do for the different GD actions
    try:
        gd_actions = json.loads(os.environ['GD_ACTIONS'])

        for gd_finding_type, action in gd_actions.items():
            if unformatted_message["detail"]["type"] == gd_finding_type and "AUTO:" in action:
                text_output = "Found a defined rule for GD finding %s. Continuing\n" % gd_finding_type
                found_action = True
                break    

# code snippet
   try:
        # Make the main structure of the formatted message
        formatted_message = {
            "reportTime": unformatted_message["detail"]["createdAt"],
            "rule": {
                "name": action,
                "complianceTags": action
            },
            "status": "Failed",
            "account": {
                "id": unformatted_message["detail"]["accountId"],
                "vendor": "AWS"
            },
            "entity": {
                "region": unformatted_message["detail"]["region"]
            }
        }
}

How CloudBots Knows What to do with a GuardDuty Finding

In AWS Lambda, there’s an environment variable that defines the actions. They key is GD_ACTIONS and the value is a JSON blob that defines findings we’re looking for. Here’s an example:

{
    “UnauthorizedAccess:EC2/MaliciousIPCaller.Custom”: "AUTO: ec2_stop_instance",
    "Backdoor:EC2/Spambot": "AUTO: ec2_quarantine_instance"
}

If the finding from GuardDuty matches one that was defined in GD_ACTIONS, then the corresponding bot will be called. If the action is undefined, it’ll be ignored.

{
def handle_event(message,text_output_array):
    post_to_sns = True
    #Break out the values from the JSON payload from Dome9
    rule_name = message['rule']['name']
    status = message['status']
    entity_id = message['entity']['id']
    entity_name = message['entity']['name']
    region = message['entity']['region']
    compliance_tags = message['rule']['complianceTags'].split("|")
## removed code ##
  for tag in compliance_tags:
        tag = tag.strip() #Sometimes the tags come through with trailing or leading spaces. 
        #Check the tag to see if we have AUTO: in it
        pattern = re.compile("^AUTO:\s.+")
        if pattern.match(tag):
            text_output_array.append("Rule violation found: %s \nID: %s | Name: %s \nRemediation bot: %s \n" % (rule_name, entity_id, entity_name, tag))
            # Pull out only the bot verb to run as a function
            # The format is AUTO: bot_name param1 param2
            arr = tag.split(' ')

The function then calls run_action in ec2_stop_instance.py to stop the affected Amazon EC2 instance:

## Run the bot ##
   bot_msg = bot_module.run_action(boto_session,message['rule'],message['entity'],params)

Auto-Remediation and Output

Once the GuardDuty finding has been parsed and CloudBots have identified the exact finding type and action to take, the auto-remediation is triggered. A snippet of the code to stop an Amazon EC2 instance is below:

{
  ### Turn off EC2 instance ###
def run_action(boto_session,rule,entity,params):
instance = entity['id']
ec2_client = boto_session.client('ec2')

result = ec2_client.stop_instances(InstanceIds=[instance])
responseCode = result['ResponseMetadata']['HTTPStatusCode'] if responseCode >= 400: text_output = "Unexpected error: %s \n" % str(result) else: text_output = "Instance stopped: %s \n" % instance return text_output
}

Once this action has been performed, you should see the compromised Amazon EC2 instance stopped.

Figure 7 – Amazon EC2 console showing the affected instance has been stopped by CloudBots auto-remediation.

Conclusion

Amazon GuardDuty allows customers to identify malicious activity within their AWS deployments.

With Dome9’s Entity Explorer, you can easily drill down and analyze the basis of why a resource was affected. Customers can then leverage the CloudBots auto-remediation framework in combination with well-defined and mature policies to automate incident response and reduce time to respond to security events.

For more information please visit cloudbots.io and the Dome9 Github repository.

.

Dome9 Security – APN Partner Spotlight

Dome9 is an AWS Competency Partner. They offer a comprehensive security and compliance automation solution purpose-built for the cloud. Dome9 has built rich integrations with the security services AWS offers.

Contact Dome9 | Solution Overview | Buy on Marketplace

*Already worked with Dome9? Rate this Partner

*To review an APN Partner, you must be an AWS customer that has worked with them directly on a project.