Cloud incident response at UNSW with digital forensics powered by AWS

Introduction

In the digital age, universities face increasing cyber threats that put valuable data at risk. The University of New South Wales (UNSW), a renowned research institution, is taking proactive measures to address this growing concern.

By collaborating with Amazon Web Services (AWS), UNSW aims to modernize its IT infrastructure and bolster its cybersecurity defenses as part of its cloud transformation program. As part of the transformation program, AWS partnered with CyberCX, a leading cybersecurity company in Australia, and proposed leveraging the Automated Forensics Orchestrator for Amazon EC2. The Forensics Orchestrator deploys a mechanism using AWS services to orchestrate and automate key digital forensics processes and activities for Amazon Elastic Compute Cloud (Amazon EC2) instances in the event a potential security issue is detected.

This post provides a walkthrough of how UNSW achieved this milestone, through enhancing the Automated Forensics Orchestrator for Amazon EC2 to meet their requirements and ensuring an efficient and effective incident response and investigation plan in the event of a cyberattack.

Forensics Orchestrator framework

Automated Forensics Orchestrator for Amazon EC2 is an AWS solution that provides customers the ability to integrate a serverless framework that automates the end-to-end forensic containment and analysis process for Amazon EC2 instances.

Using this solution, incident response (IR) teams can focus efforts on the investigation of potentially compromised Amazon EC2 instances, removing the need to manually gather artifacts such as volumes and memory.

The framework is deployed using a series of predefined AWS CloudFormation templates. These templates are deployed across AWS workload accounts, the security account (capture memory, artifacts), and the forensic account (investigate and analyze).

Key components of the framework include:

1. Incident identification and response – Automated monitoring using AWS Security Hub and Amazon GuardDuty to trigger alerts for security teams.
2. Data collection and preservation – Seamless collection and preservation of critical data to ensure evidence remains untainted.
3. Scalable analysis – By using the scalability of the AWS Cloud infrastructure, UNSW can analyze large volumes of data efficiently. This capability accelerates the forensic investigation process, reducing downtime and minimizing the impact of cyber incidents.

Figure 1 shows the major components of the Automated Forensics Orchestrator for Amazon EC2 including the accounts involved, AWS Lambda and CloudFormation for automation, and Amazon Simple Storage Service (Amazon S3) for storing data.

Figure 1. High-level architecture of Automated Forensics Orchestrator for Amazon EC2.

Approach

Working with the UNSW incident response team, the Automated Forensics Orchestrator for Amazon EC2 served as a baseline for developing a tailored forensics solution to meet the university’s specific requirements. To ensure the principle of least privilege across all accounts, new features were identified and incorporated into the solution, including:

Graphical interface for investigations

To facilitate investigations and make the process more user-friendly, the team developed a graphical interface for the Orchestrator. This interface allowed incident responders to visually navigate through the forensic data and perform necessary actions with greater ease and efficiency, simplifying the overall investigation process.

Just-in-time (JIT) access

The JIT feature enhances the security posture of the incident response system, ensuring that only trusted individuals have temporary access to essential automation features while protecting against potential misuse by unauthorized users.

Enhanced automation

Recognizing the importance of automation in expediting response times and minimizing human errors, the team emphasized enhancing automation capabilities within the Orchestrator. Automated tasks, such as evidence collection, analysis, and reporting, were introduced to further optimize incident response procedures.

Improved deployment pipeline

The team focused on enhancing the deployment pipeline to streamline the process of deploying and updating the Orchestrator. By implementing continuous integration and continuous deployment (CI/CD) practices, the solution could be efficiently managed and delivered, reducing downtime and ensuring the latest updates were promptly available.

By addressing these specific areas and incorporating UNSW’s unique requirements, the Orchestrator was finely tuned to seamlessly integrate into their AWS environment. The result was an efficient and effective forensics solution capable of meeting the university’s needs while adhering to the principle of least privilege.

Solution overview

Graphical interface for investigations

The Orchestrator solution was designed to use the SANS Investigative Forensics Toolkit (SIFT) Workstation for IR investigations on Amazon EC2 memory and disks. The UNSW team requested the ability to perform investigations using a graphical user interface (GUI), instead of the standard command line interface (CLI), which comes as standard with SIFT.

To securely provide IR admins with access to a SIFT GUI, the following features in Table 1 required significant enhancement:

Table 1

Figure 2 shows how the IR administrator accesses the SIFT instance via a hardened bastion to perform investigation tasks.

Figure 2. AWS Systems Manager Agent (SSM Agent) access to forensic GUI.

The hardened bastion was deployed through the existing Amazon EC2 Image Builder pipeline using UNSW preapproved patterns and security tooling, including antivirus and anti-malware technologies. Enhancements were also made to the forensic IR AMI to provide a GUI on the instance, and additionally lock down the interface using secrets injected at image runtime from AWS Secrets Manager.

This new design provides the IR administrator with secure access to a forensic Amazon EC2 instance through a hardened bastion host using Systems Manager. The bastion and forensic instances are both designed to be ephemeral, meaning once investigations are complete, they are destroyed.

Secure access has been provided using Systems Manager endpoints to the Windows bastion, with locked-down security groups allowing only RDP access to the forensic IR instance. This minimizes the effect in a worst-case scenario when investigating potentially malicious malware.

JIT access

The Forensics Orchestrator solution provides two options when performing IR in the AWS workload accounts:

1. Contain – Revoke the ability for an Amazon EC2 instance to communicate by completely isolating it.
2. Analyze – Securely snapshot and share the volumes associated with the Amazon EC2 instance for investigation in the forensic account.

Users assign a set of key-value tags to the instance to trigger the automation steps for each instance. For example, when GuardDuty detects a potential malicious event, the IR administrator can apply the workload instance with the tag to trigger the functions. The concern was that these tags could be applied by anyone in the account that had knowledge of the values, which exposes a data leakage and application availability risk to UNSW.

UNSW faced a problem in that it has a large subset of users across many workload accounts that could trigger the IR automation features. An internal actor could cause damage across accounts that could lead to either data being removed from the account (analyze function) or availability issues (containing applications). All they would need to do is tag Amazon EC2 instances, which many users have permission to perform.

To solve the issue, they provisioned automated JIT access to an authorized subset of users across the landing zone.

The resulting JIT access solution uses temporary credentials assigned to a small number of administrators using role-based access control (RBAC). When an IR administrator requests a temporary elevated role, Lambda functions are triggered to create the role and then add time-bound limits. Once no longer required, the role is terminated, and users can no longer trigger automation functions using the tags.

This function was built entirely using AWS services:

• AWS Service Catalog is used to provide a re-usable template to create required roles for temporary privilege.
• Lambda serverless functions trigger required actions, such as temporary privilege elevation assignments to users.
• Amazon EventBridge rules trigger functions and actions as required (scheduled, event driven), such as triggering function to expire roles.

Figure 3 shows how the JIT role is assigned from Service Catalog to allow authorized security administrators to perform investigation.

Figure 3. Simplified view of the JIT access provisioning process.

Enhanced automation

The most requested enhancement of the Orchestrator was the ability to simultaneously contain and analyze Amazon EC2 instances using a new tag value. The tagging features were designed to allow IR investigators to contain or analyze an instance, but not both at the same time.

In the case of an incident, you had to first trigger the analysis functions to process Amazon EC2 memory and disk volumes and then trigger the function again to contain the instance. This leaves a window of approximately 30 minutes while the analysis functions trigger before the instance can be isolated in the environment.

To minimize this time window and control the potential effect, an additional tag value was engineered called the quarantine tag. This tag allows for the simultaneous containment of an Amazon EC2 instance from the environment and secure sharing of the instance volumes to the forensic account.

This enhancement was developed using Lambda functions to call on AWS APIs. The Lambda triggers based on an AWS CloudTrail API event  (when an Amazon EC2 instance is tagged with the quarantine value) and then:

• If the JIT role has been approved and the tag value matches exactly the required tag, Lambda will trigger.
• Lambda will make API calls using Boto3 to isolate the instance:
  • Revoke security group rules, so that no traffic is allowed inbound and outbound.
  • Revoke the identity access management (IAM) role on the instance and add a new IAM role with minimal permissions.
  • Revoke instance termination protection.
• Lambda passes the instanceID and other volume details to the security account step function for disk and memory capture.

Figure 4 shows how Lambda leverages tags to contain and isolate a compromised instance using CloudTrail API and EventBridge.

Figure 4. Diagram showing the tagging function for IR.

Improved deployment pipeline

UNSW used customizations for AWS Control Tower, which is a highly customized CI/CD implementation to deploy resources and governance at scale. The Orchestrator framework required significant modification to integrate templates into the customized CI/CD process. These modifications included:

• Update templates to use existing infrastructure, removing duplication of resources.
  • Existing Amazon EC2 Image Builder integration.
  • Existing network integration in security account. Ensuring networks meet preapproved patterns such as allowed classless inter-domain routing (CIDR) ranges.
  • Adding parameterization to templates, enabling the customization to automatically deploy the forensic framework to all existing and new accounts moving forward.
• Update template syntax for consistency
  • Updating infrastructure as code (IaC) syntax to YAML from JSON.
  • Fix formatting issues found when linting templates.

 Example – Update templates to use the existing Amazon EC2 Image Builder

UNSW deployed the Amazon EC2 Image Builder pipeline, which is the intended tool to be used across an organization for creating and sharing golden images. The Forensics Orchestrator framework would have deployed a similar tool in the forensic account, unnecessarily duplicating resources.

Figure 5 shows the high-level architectural view of the forensic Amazon EC2 Image Builder pipeline template used in existing UNSW deployments. This creates a golden AMI used to investigate compromised memory and disk for Amazon EC2 instances.

Figure 5. AWS Forensics Orchestrator integration template.

The proposal moving forward was to modify the IaC templates to achieve the following:

• Reduce cloud costs
  • Use the existing Amazon EC2 Image Builder pipeline and supply the AMI configuration to build the image. The AMI is then shared directly with the forensic account.
• Enhance security
  • Use existing customer managed AWS Key Management Service (AWS KMS) keys for encryption of the AMI upon creation.
  • Hardened AMI configuration using injectable secrets at runtime with Secrets Manager.
  • Sharing the forensic AMI with the forensic account securely over AWS PrivateLink.
  • Removal of the internet gateway after all required data for AMI configuration was moved to a local Amazon S3.

Figure 6 shows the improved deployment pipeline with enhanced security incorporating AWS KMS to encrypt the AMI and leveraging PrivateLink to share the forensic AMI.

Figure 6. Enhanced integration into UNSW infrastructure.

Conclusion

Through the enhancements made to the AWS Automated Forensics Orchestrator for Amazon EC2 solution, UNSW achieved several important benefits. These improvements led to significant positive outcomes for the university’s cybersecurity capabilities.

The modifications to the Orchestrator’s deployment process streamlined the implementation and management of the solution, reducing complexity and potential errors during setup. By leveraging the existing networking architecture and avoiding the creation of redundant resources, UNSW minimized unnecessary expenses associated with deploying the forensics solution.

The enhanced solution provided better visibility into cyber events and incidents across the university’s AWS environment, enabling quicker detection and response to potential threats. With the integration of automated JIT access and adherence to the principle of least privilege, the risk of unauthorized access and misuse of the Orchestrator’s features was significantly mitigated. The graphical interface for the incident response team members simplified the investigation process, allowing for more intuitive navigation and efficient data analysis.

UNSW’s collaboration with AWS in developing this cloud-centered forensics solution demonstrates the university’s dedication to safeguarding data and improving incident response capabilities. By continuously innovating and fostering a resilient cybersecurity culture, UNSW sets an inspiring example for other educational institutions seeking to protect their digital assets in today’s interconnected world.

CyberCX is an AWS Premier Services Partner and a leading provider of professional cybersecurity and cloud services across Australia and New Zealand. If you represent a higher education institution and are interested in learning how AWS can help guide your institution through your security journey, please contact us.