How to analyze AWS Network Firewall logs using Amazon OpenSearch Service – Part 2
In part 1 of this blog-post series, we walked you through steps to configure Amazon OpenSearch Service to receive logs from AWS Network Firewall using Amazon Kinesis Data Firehose. In this part 2, we cover steps to generate test alerts, validating them and configure dashboards in Amazon OpenSearch Service to visualize and analyze log data.
Generate test alerts
- To generate Network Firewall alert logs, use testmynids.org which provides testing for the detection of malicious events by network intrusion detection systems (NIDS) with test files and scripts that simulate test NIDS activities.
- Connect to the Web server EC2 instance created as part of the CloudFormation stack using Session Manager. To do this, go to the EC2 Instances Console, select Web server instance, and select Connect.
- Then, choose Session Manager and select Connect. This will open a browser tab with terminal access to the Web server.
- Run the following commands
sudo yum install -y nc # installing ncat<br /> curl -sSL https://raw.githubusercontent.com/3CORESec/testmynids.org/master/tmNIDS -o /tmp/tmNIDS && chmod +x /tmp/tmNIDS && /tmp/tmNIDS -h<br />/tmp/tmNIDS -99
- While this command is running, go to Network Firewall, select Network Firewall → Monitoring, and check Stateful Received Packets, Passed Packets, and Dropped Packets.
- Moreover, look at the Monitoring of Kinesis Data Firehose delivery stream by navigating to Amazon Kinesis delivery streams → select required delivery stream → Monitoring.
- As a last step, confirm data reception to the required index at Amazon OpenSearch Dashboard → Discover.
- This confirms that you’re receiving Network Firewall logs to the Amazon OpenSearch Service domain. Now you can start creating dashboards and charts on log data.
Creating dashboard to analyze Logs
You can create visualizations to look at different metrics, and then combine them to create a dashboard that gives you the complete analysis of logs. Here we describe how to create Tag Cloud and Pie Chart visualizations.
Tag Cloud visualization
- Log in to Amazon OpenSearch service, select the Visualize option from the menu, and then select Create visualization. Select the Tag Cloud visualization.
- Select the index that was created for logs, and then configure the visualization options. In the Data section, under the Metrics option, select Tag size and leave the default Count under the Aggregation. In the Buckets, select Add -> Tags from the dropdown -> Significant Terms in the Aggregation dropdown. -> event.app_proto.keyword under the Field dropdown. In the Size dropdown, enter a value based on how many words you want to see in the visualization.
- In the Options section, you can change the Orientations and Font size of the words. Apply your changes by selecting Update.
- You’ll see the visualization similar to this based on the data in the index.
- Select Save, give a name in the Title, and give some Description of the visualization to save the visualization.
Pie Chart visualization
- Select the Visualize option from the menu and then select Create visualization. Select the Pie.
- Select the same index and then configure the visualization options. In the Data section, under the Metrics option, select Slice size, and select Sum under Aggregation dropdown and netflow.bytes under the Field dropdown. In the Buckets, select Add, select Split slices from the dropdown, select Terms in the Aggregation dropdown, and then select event.src_port under the Field dropdown. In the Size dropdown, enter a value based on how many values you want to see in the visualization. Select Add again, select Split slices from the dropdown, select Terms in the Aggregation dropdown, and then select event.dest_port under the Field dropdown. In the Size dropdown, enter a value based on how many values you want to see in the visualization.
- In the Options section, you can change the Pie settings and Label settings. Apply your changes by selecting Update.
- Select Save, provide a name in the Title, and give some Description to save the visualization.
You can combine visualizations that show all of the relevant information about logs. Select Dashboard from the menu and then select Create dashboard. Select Add an existing to add visualizations to the dashboard. The panel will show the visualizations and select all of the necessary visualizations. Once done, all of the selected ones will be added to the dashboard. The size of the visualizations and some more formatting changes can be completed here to arrange the visualizations properly in the dashboard. Once done, select Save to save the dashboard and provide a Title and Description.
In the following sample dashboard, there are multiple visualizations, such as Pie chart, Donut chart, Horizontal and Vertical bar charts, and Tag Cloud. These focus on different metrics such as Source and Destination by Bytes Transferred and Flow count by different dimensions. These include application protocol, Source and destination IPs, Protocol, Source and Destination ports, and TCP flags.
- First, clean up Network Firewall by navigating to AWS CloudFormation Stacks, select the stack that you have created earlier, and Delete.
- Next, go to Network Firewall rule groups and delete the two Suricata Stateful rule groups that you created.
- Next, delete the Kinesis Data Firehose delivery stream by navigating to Delivery streams, selecting the delivery stream that you have created, and Delete.
- Then, go to IAM Roles and delete the Service role created by Kinesis Data Firehose delivery stream. Find the Service role required by filtering via the Kinesis Data Firehose delivery stream name.
- Then, go to S3 buckets and delete the bucket that you created to store the failed data of the delivery stream.
- Lastly, clean up Amazon OpenSearch Service domain by navigating to Domains, selecting the domain that you have created, and Delete.
Altogether, this two-part blog series demonstrated the steps involved in analyzing AWS Network Firewall We walked through how to setup Amazon OpenSearch Service Index-specific permission for Kinesis Data Firehose Service role. Furthermore, we demonstrated how to configure rules in Network Firewall and generate test alerts. Moreover, we demonstrated how to create a dashboard and visualize different metrics in Amazon OpenSearch Service. You can also get hands-on experience with AWS Services using Network Firewall Workshop and Amazon OpenSearch Service Workshops.
About the authors: