Building a network traffic analysis system: Deploying Malcolm on Amazon EC2

Cybersecurity organizations face significant pressure to maintain comprehensive network visibility while managing costs effectively. Malcolm, an open source network traffic analysis suite developed by Idaho National Laboratory and the U.S. Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency (CISA), has emerged as a powerful solution for organizations seeking enterprise-grade security monitoring capabilities without the burden of expensive commercial licenses.

In this post, we provide step-by-step guidance for deploying Malcolm natively on Amazon Elastic Compute Cloud (Amazon EC2), using the on-demand compute, elasticity, and scalability of Amazon Web Services (AWS) to eliminate traditional infrastructure overhead. This approach is particularly beneficial for cost-conscious public sector (federal, state, and local), utility, and industrial customers who require network and security monitoring to deliver on their missions.

Solution overview

Malcolm is deployed on a single EC2 instance within an Amazon Virtual Private Cloud (Amazon VPC) endpoint. The instance resides in a public subnet for direct internet access. An internet gateway enables communication between the VPC and the internet. This architecture can be scaled by increasing the EC2 instance size or the Amazon Elastic Block Store (Amazon EBS).

Figure 1: Malcolm deployed on Amazon EC2 architecture

1. EC2 instance – The Malcolm platform runs on Ubuntu 24.04 LTS and has minimum specifications of 30 GB of memory and 8 vCPUs. The Ubuntu operating system (OS) hosts multiple Docker containers running Malcolm’s components.
   • For this blog we use an arm64 c8g.8xlarge. We recommend using the latest generation instance from either the Compute Optimized or General Purpose family.
2. EBS storage – An EBS volume with a minimum of 100 GB of storage on a gp3 (general-purpose) SSD volume provides persistent storage for Malcolm’s data and logs.
3. Malcolm components – All the Malcolm components are run as Docker containers on our Malcolm EC2 instance. The key Malcolm services include: Zeek, Suricata, OpenSearch, Logstash, Arkime, and NGINX.
4. Security and accessibility – A security group restricts inbound traffic to HTTPS (443 from specified IP addresses. Outbound internet access is enabled through the internet gateway for updates and package installation.

Prerequisites

To perform the solution, you need to have the following prerequisites:

• An AWS account
• AWS Command Line Interface (AWS CLI) installed
• Familiarity with command line environment

Now that we have an understanding of the architecture, let’s build it! First, we will create our EC2 instance, then we will set up Malcolm within the instance using docker containers, and finally we will deploy Malcolm.

Amazon EC2 instance creation

You need to perform these steps in a command line environment such as AWS CloudShell, Linux, Windows, or macOS system with the AWS CLI installed.

1. Create an IAM role for your EC2 instances by running the following sets of commands. The commands create a trust policy, an IAM role, attaches the AWS managed policy we will be using and then creates an instance profile and attaches the role.

   cat << EOF > trust-policy.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Service": "ec2.amazonaws.com"
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }
   EOF	
   
   
   aws iam create-role \
       --role-name EC2-SSM-Role \
       --assume-role-policy-document file://trust-policy.json

   

    

   aws iam attach-role-policy \
       --role-name EC2-SSM-Role \
       --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
   
   
   aws iam create-instance-profile \
       --instance-profile-name EC2-SSM-Role
   
   
   aws iam add-role-to-instance-profile \
       --instance-profile-name EC2-SSM-Role \
       --role-name EC2-SSM-Role

2. Create a security group for the EC2 instance:

   $ aws ec2 create-security-group \
       --group-name malcolm-sg \
       --description "Malcolm SG"

   

3. Set the inbound security group (SG) rules. Replace YOUR_PUBLIC_IP with your public IP address in the following commands so you can access Malcolm on the EC2 instance from your workstation using HTTPS (port 443):

   $ aws ec2 authorize-security-group-ingress \
       --group-name malcolm-sg \
       --protocol tcp \
       --port 443 \
       --cidr YOUR_PUBLIC_IP/32

4. Run the following command and replace ARCH with the desired architecture (amd64 or arm64). For this post we will use arm64. Copy the AMI ID for the next step.

   $ aws ssm get-parameter --name /aws/service/canonical/ubuntu/server-minimal/24.04/stable/current/ARCH/hvm/ebs-gp3/ami-id --query Parameter.Value --output text

   

5. Run the following command. Replace AMI_ID with the AMI ID from the previous step. Replace INSTANCE_TYPE with the desired instance type (we will use 8xlarge). The size of the storage volume will vary depending on the amount of data you plan to process and retain in Malcolm. The example here uses 100 GB; adjust as needed for your use case.

   $ aws ec2 run-instances \
       --image-id AMI_ID \
       --instance-type INSTANCE_TYPE \
       --iam-instance-profile Name=EC2-SSM-Role
       --security-group-ids malcolm-sg \
       --block-device-mappings "[{\"DeviceName\":\"/dev/sda1\",\"Ebs\":{\"VolumeSize\":100,\"VolumeType\":\"gp3\"}}]" \
       --count 1 \
       --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=Malcolm}]"

6. Get the instance details and check the status:

   $ aws ec2 describe-instances \
       --filters "Name=tag:Name,Values=Malcolm" \
       --query "Reservations[].Instances[].{ID:InstanceId,IP:PublicIpAddress,State:State.Name}"

Malcolm setup

You need to perform the following steps as the Ubuntu user inside the EC2 instance. Connect via AWS Systems Manager Session Manager.

1. Switch from the ssm-user to the ubuntu user.

   $ sudo --login --user ubuntu

2. Install curl, unzip, and Python dependencies:

   $ sudo apt-get -y update
   
   $ sudo apt-get -y install curl unzip python3 python3-dialog python3-dotenv python3-pip python3-ruamel.yaml

3. Run the following command to download the latest Malcolm release .zip file. We use Malcolm version 25.06.0. When you run it, make sure to navigate to the Malcolm releases page and use the version number of the latest Malcolm release.

   $ curl -OJsSLf https://github.com/cisagov/Malcolm/releases/latest/download/malcolm-25.06.0-docker_install.zip
   
   $ ls -l malcolm*.zip

4. Extract the Malcolm release .zip file:

   $ unzip malcolm-25.06.0-docker_install.zip

   

5. Now let’s install and configure Malcolm. Run the command below. When prompted to extract the Malcolm runtime files from the .tar.gz archive, select Yes and accept the default installation path.

   $ ./install.py \
     --defaults \
     --restart-malcolm \
     --delete-old-pcap \
     --delete-pcap-threshold '7%' \
     --delete-index-threshold '75%' \
     --extracted-file-max-size-threshold '5G' \
     --file-extraction interesting \
     --file-preservation quarantined \
     --extracted-file-server \
     --extracted-file-server-password infected \
     --extracted-file-server-zip \
     --netbox local \
     --netbox-autopopulate \
     --netbox-auto-prefixes \
     --zeek-ics

   The Malcolm installation script sets up necessary dependencies and configures Malcolm. It can be run non-interactively, which is what we chose.

   ( If you want to customize Malcolm’s configuration instead, run ./install.py and answer the installation and configuration questions one-by-one to suit your use case).

6. Reboot the instance and reconnect. Rebooting EC2 instance allows for the new system configuration to take effect. Then, reconnect through Session Manager.

   $ sudo reboot

Set up authentication in Malcolm

Now let’s set up authentication.

1. First, switch back into the Ubuntu user:

   $ sudo --login --user ubuntu

2. Configure the authentication setup with the following command:

   $ ~/malcolm/scripts/auth_setup \
      --auth-noninteractive \
       --auth-method basic \
       --auth-generate-webcerts \
       --auth-generate-fwcerts \
       --auth-generate-netbox-passwords \
       --auth-generate-redis-password \
       --auth-generate-postgres-password \
       --auth-generate-opensearch-internal-creds

3. Set up the administrator credentials by running the auth_setup script again (without any parameters).

   $ ~/malcolm/scripts/auth_setup

   1. • Use arrow keys to navigate to “Store administrator username/password for basic HTTP authentication“
      • Press Space to select it and Enter to confirm
      • Enter your desired administrator username and password when prompted

   You can reference the Malcolm documentation for more information on the auth_setup prompts.

Running Malcolm

You need to perform the following steps as the Ubuntu user inside the EC2 instance.

1. Start Malcolm using the start script in the Malcolm installation directory:

   $ cd ~/malcolm
   $ ./scripts/start

   
   Malcolm takes a few minutes to start. During this time, you might see text scroll past from the containers’ logs that look like error messages. This is normal while Malcolm’s services synchronize among themselves.When Malcolm is running, the start script will output Started Malcolm, and you’ll be returned to the command prompt:

   logstash-1 | [2025-04-10T15:03:28,294][INFO ][logstash.agent ] Pipelines running {:count=>6, :running_pipelines=>[:"malcolm-input", :"malcolm-output", :"malcolm-suricata", :"malcolm-enrichment", :"malcolm-beats", :"malcolm-zeek"], :non_running_pipelines=>[]}
   
   Started Malcolm
   
   Malcolm services can be accessed at https://<IP address>/

   

2. Use the status script in the Malcolm installation directory to show the status of Malcolm’s services:

   $ cd ~/malcolm
   $ ./scripts/status

3. Connect to Malcolm’s web interface. From your workstation in a browser, navigate to the public IP address of the instance. The IP address was captured in Amazon EC2 instance creation step Log in with the credentials specified when setting up authentication in the preceding step.

Conclusion

Malcolm is an open source, seamlessly deployable network traffic analysis suite created by INL and CISA. In this post, we demonstrated how to install and run Malcolm natively on Amazon EC2. Using this approach, you can benefit from the flexibility and cost-efficiency of cloud-based infrastructure, affording enterprise-grade network monitoring capabilities without the constraints of conventional on-premises deployments. Happy building!

Additional Malcolm Resources:

• Malcolm Documentation
• Learning Tree
• Training Videos
• Discussion Board