The key components of CISA’s Malcolm on Amazon EKS

Understanding your network traffic is key to improving your security posture. Network traffic analysis tools are one way to dive into network findings and events. Malcolm is a powerful, open source network traffic analysis tool suite created by the Cybersecurity and Infrastructure Security Agency (CISA) to aid public and private sector customers in improving their network security monitoring and incident response. Malcolm is most commonly used for incident response, network monitoring, threat hunting, training, and research, but can be adapted for other use cases.

Malcolm is deployable on-premises or in the cloud. In the Amazon Web Services (AWS) Cloud, Malcolm can be deployed as a standalone Amazon Elastic Compute Cloud (Amazon EC2) instance using an Amazon Machine Image (AMI). For full deployments intended to monitor production traffic, it can be deployed as an Amazon Elastic Kubernetes Service (Amazon EKS) cluster. In this post, we introduce you to the key components of Malcolm on Amazon EKS.

Solution overview

Before we start exploring Malcolm on Amazon EKS, it’s important to understand how Malcolm works. Let’s break down Malcolm’s data ingestion, processing, analysis, and visualization capabilities. The following diagram shows the high-level architecture of Malcolm.

Figure 1. Architectural diagram of Malcolm.

Data ingestion

Malcolm accepts network traffic data in the form of full packet capture (PCAP) files, Zeek logs, and Suricata alerts. You can upload these files using a browser-based interface, or they can be passively captured live and forwarded to Malcolm using lightweight forwarders.

Data processing and analysis

Once the PCAP files are ingested, Suricata and Zeek analyze the packets to extract metadata, detect anomalies, and identify potential threats. Then, the extracted metadata and logs are processed by Logstash and indexed into OpenSearch so that the data is searchable and analyzable (Note: Malcolm supports Elasticsearch as an alternative to OpenSearch for organizations where Elasticsearch is already in place). Arkime also feeds into OpenSearch. For searching detailed packet data, users can drill down to specific network events or investigate suspicious activity.

Visualization and search

After the data is processed and analyzed, users can interact with the processed data through OpenSearch Dashboards and Arkime. This data can help organizations and security teams gain a more comprehensive view of their network protocols and help identify the network sessions in a suspected security incident. Although all the open source tools that make up Malcolm are already available and in general use, Malcolm provides a framework of interconnectivity that makes it greater than the sum of its parts.

Malcolm on Amazon EKS

Malcolm is composed of several widely used open source tools, making it an attractive alternative to security solutions requiring paid licenses. Malcolm is available as an Open Container Initiative (OCI) compliant container image. Amazon EKS is a managed Kubernetes service to run Kubernetes in the AWS Cloud and in on-premises data centers. The high-level design of Malcolm on Amazon EKS is shown in the following diagram.

Figure 2. Architectural diagram of Malcolm on Amazon EKS.

The workflow is as follows:

1. Malcolm container images are stored and retrievable from CISA’s GitHub. These images are OCI compliant and can be stored in any container registry of your choice, including Amazon Elastic Container Registry (Amazon ECR).
2. Next, the Malcolm container images are deployed into the EKS cluster worker nodes (CISA has tested and published instructions on running Malcolm on Amazon EKS at Deploying Malcolm on Amazon Elastic Kubernetes Service). You can use your existing DevOps pipeline or GitOps to deploy the images like your other containers workload.
3. Malcolm is designed to work with Kubernetes ingress. As part of the published setup, ingress traffic to access the Malcolm UI is handled by AWS Load Balancer Controller.
4. All the Malcolm analysis results are stored in Amazon Elastic File System (Amazon EFS).

You can deploy infrastructure in minutes and scale up and down automatically by deploying Malcolm on Amazon EKS. Amazon EKS automatically manages the availability and scalability of the Kubernetes control plane nodes responsible for scheduling Malcolm containers, managing application availability, and storing cluster data, so you can spend time on tasks that have a direct impact on your organization. This also means that during peak hours, Malcolm will have the resources it needs, but during off hours you aren’t paying for resources you’re not using. Malcolm on Amazon EKS can be deployed in multiple Availability Zones, providing increased resilience compared with deploying in a single data center.

From a security standpoint, customers that deploy Malcolm on Amazon EKS benefit from AWS secure infrastructure and best practices and have additional tooling available to help them enforce security at every level of their stack. For example, customers can use Amazon Virtual Private Cloud (Amazon VPC) Traffic Mirroring to send network traffic to out-of-band security and monitoring appliances for content inspection, threat monitoring, and troubleshooting. They can also use tools such as AWS Identity and Access Management (IAM) to create more fine-grained permission sets and enforce the principle of least privilege. 

Conclusion

Malcolm is an open source, seamlessly deployable network traffic analysis tool created by CISA. This tool can aid US government agencies and others who are looking to improve their network monitoring and incident response, especially those still seeking to meet the requirements of the Office of Management and Budget’s M-21-31: Improving the Federal Government’s Investigative and Remediation Capabilities Related to Cybersecurity Incidents. By using Amazon EKS, customers can not only enjoy the benefits of the cloud, they can also take advantage of the built-in benefits of Amazon EKS. To learn more about Malcolm on Amazon EKS, visit the Malcolm GitHub or Malcolm YouTube channel. Happy building!