AWS Business Intelligence Blog
Visualize AWS Network Firewall logs with Amazon QuickSight dashboards
Businesses transitioning to cloud-centered or digital-native applications use various cloud services to innovate and improve customer offerings. These applications are crucial for business operations, but monitoring network traffic from different segments is equally important, especially for securing sensitive data. An analytics dashboard becomes vital to extract insights from network events in Amazon Web Services (AWS) multi-account environments. You can use an analytics dashboard to prioritize and respond to security threats, creating a proactive approach to network management and security. Using analytics and visualizations help to optimize security for cloud-based applications.
This blog post explains how to create interactive dashboards with Amazon QuickSight to visualize your AWS Network Firewall logs. Additionally, it covers tracking mechanisms that you can use to gain visibility into your Amazon Virtual Private Cloud (Amazon VPC) network traffic, aiding in the identification of security threats and networking issues. This seamless integration empowers visual interpretation of network traffic flows and analysis of firewall logs, thereby enabling comprehensive insights and agile decision-making.
Solution overview:
The following architecture diagram illustrates the log collection and visualization solution for Network Firewall logs using QuickSight. Network traffic, originating from an Amazon Elastic Compute Cloud (Amazon EC2) instance and destined for the internet, traverses through the Network Firewall endpoint deployed in a subnet in an inspection VPC. The Network Firewall, operating according to its configured firewall policies, inspects the traffic and generates and stores the logs in JSON format in an Amazon Simple Storage Service (Amazon S3) bucket.
Next, AWS Glue crawler accesses the S3 bucket to catalog the logs and create table definitions in AWS Glue Data Catalog. Lastly, Amazon Athena is used to create views, to be visualized as dashboards in QuickSight.
Prerequisites:
Alternatively, for the purpose of following along with our solution, you can deploy a Network Firewall and other related resources using this AWS CloudFormation template. The stack deploys an Amazon VPC, a Network Firewall, and an EC2 instance that might incur costs when in use. See AWS Network Firewall and Amazon EC2 Pricing.
- Create an S3 bucket that stores the Network Firewall logs.
- Enable and set up Athena.
- Sign up for QuickSight.
Step 1: Configure Network Firewall logs destination as your S3 bucket
- Sign in to the AWS Management Console and go to Amazon VPC.
- Navigate to your VPC dashboard, then to Network Firewall and Firewalls.
- Select the name of the firewall that you want to configure to view its details.
- On the Firewall details tab, go to the section named Logging and choose Edit.
- Under Log type, select the check boxes for Alert and Flow to view further options.
- Both for the Alert log and Flow log destinations, choose S3, and enter the S3 bucket name.
- Choose Save to update the firewall logging configuration.
Step 2: Generate the AWS Network Firewall logs
Whenever any network traffic interacts with Network Firewall, logs are generated. If you have a network firewall deployed, it’s already generating firewall logs and pushing them to the S3 bucket you chose in the preceding step. You can proceed to step 3.
Alternatively, if you used the sample CloudFormation template from the prerequisites section, connect to your EC2 instance {Stack Name}-test-instance using AWS Systems Manager Session Manager, and use the Ncat command to generate network traffic. This will generate firewall logs and store them to your S3 bucket.
Now that you’ve set up your network firewall to send its logs to an Amazon S3 bucket, let us prepare to analyze these logs. Next, you will create the necessary analytics components, beginning with the Data Catalog database and an AWS Glue crawler.
Step 3: Create a Data Catalog database and an AWS Glue crawler
A database in the Data Catalog is a container that holds the tables. Create a database under Data Catalog and name it anflogsdb
.
The purpose of an AWS Glue crawler is to examine the raw data in the S3 bucket to understand its format, structure, and other details. Then the crawler organizes the data into tables and keeps track of everything in the Data Catalog. To create a crawler that reads the log files stored on Amazon S3:
- On the AWS Glue console, choose Data Catalog in the navigation pane, and then choose Crawlers and Create crawler.
- Enter
network_firewall_log_data
as the Crawler name, add an optional description and tags, and choose Next. - On the Choose data sources and classifiers page, choose Add a data source and select S3 as the data source.
- For the option Location of S3 data, select In this account, and enter the path of S3 bucket where the firewall logs are stored as the S3 path. You can browse S3 to choose the folder that contains both logs (alert and flow logs). Choose Add an S3 data source to save and choose Next.
In this blog post we are considering both flow and alert firewall logs.
- The crawler needs permissions to access the data store and create objects in the Data Catalog. Choose Create an IAM role.
The AWS Identity and Access Management (IAM) role name starts with AWSGlueServiceRole- which is prefilled. Append network_firewall_logs
to the field. Choose Create and Next.
For the purpose of this demo, see security best practices in IAM to configure the IAM role following the principle of least privilege.
- Crawlers create tables in your Data Catalog database. Select anflogsdb (created at the start of this step) as your Target database. Optionally, add a Table name prefix to identify your tables. For example:
nfwlogs
. - Expand Advanced options and select Update all new and existing partitions with metadata from the table. This allows partitions to inherit metadata properties such as their classification, input and output format, SerDe information, and schema from their parent table.
- Under Crawler schedule, select your preferred Frequency. The crawler will look for new or changed logs in S3 according to this schedule. After the first full run, the crawler adds new partitions found since its previous run. Choose Next.
- Review the crawler configuration and choose Create crawler. See Adding an AWS Glue crawler for more information.
- On the newly created crawler’s details page, choose Run crawler. When completed, two tables will show up in the Table changes column—one each for flow and alert logs—to query the network firewall logs. If you’re using AWS Lake Formation and you have any Insufficient Lake formation permission issues, see this AWS re:Post article.
Step 4: Create Athena views
To build custom analytics, you can create views in Athena. These are logical tables that you can use to query a subset of data. Views help you to hide complexity, perform basic transformation and aggregations on columns and minimize maintenance when querying large tables. For example, here we will flatten the JSON data by extracting the fields from the event JSON field and convert unixtime format data to timestamp format.
In this step, you will query the Data Catalog database tables created in the previous step and create views in Athena, which will eventually be used by the QuickSight datasets. You will generate multiple views to analyze alert logs for reporting on traffic matching stateful rules with alert actions, and flow logs for standard network traffic flow logs.
Create the Amazon Athena alert log view using the following steps:
- Go to the Athena console and choose Query editor from the navigation pane, and then choose Tables and views. Choose Create, and then select Create view.
- Replace the current template with this SQL create view query. Make sure to substitute
<<alertlogs_tablename>>
with the name of the alert log table you created using the AWS Glue crawler in the previous step. - Choose Run to create the view.
Create the Amazon Athena flow log view using the following steps:
- Go to the Athena console and choose Query editor from the navigation pane, and choose Tables and views. Choose Create, and then select Create view.
- Replace the current template with this SQL create view query. Make sure to substitute
<<flowlogs_tablename>>
with the name of the flow log table you created in the previous step via the Glue crawler. - Choose Run to create the view.
Step 5: Create Amazon QuickSight datasets
Before you can analyze and visualize the Network Firewall logs data in QuickSight, you need to authorize QuickSight access to Athena and the S3 bucket containing the firewall logs.
- Go to the QuickSight console and choose the profile icon at the upper right corner. Choose Manage QuickSight and then choose Security & permissions and Manage.
- Depending on whether you are a first-time user or have QuickSight already set up, choose Add or Remove or Manage.
- Locate Amazon S3 from the list of AWS services and choose one of the following actions to open the screen where you can choose S3 buckets:
- If the check box is clear, select Amazon S3.
- If the check box isn’t clear, choose Select S3 buckets.
- On the S3 buckets Linked to QuickSight Account tab, select the box for the S3 bucket containing the Network Firewall logs and also the corresponding box for Write permission for Athena Workgroup. Choose Finish.
- On the QuickSight access to AWS services page, select Amazon Athena and choose Save.
Next, create the QuickSight datasets for the following Athena views: nfw_flow_firewall_logs
and nfw_alert_firewall_logs
.
- On the QuickSight console, choose Datasets from the navigation pane and then choose New dataset.
- Create a new Athena data source:
- From the available list, select the Athena data source card.
- Enter a descriptive Data source name such as
anflogs_datasource
. - Choose Athena workgroup and then select your workgroup.
- Choose Validate connection to test the connection.
- Choose Create data source.
- On the Choose your tablewindow, select the Glue database anflogsdb and the Athena view created for the flow logs nfw_flow_firewall_logs.
- Choose Select and then Edit/Preview data to create the dataset nfw_flow_firewall_logs.
- Choose Save & publish to create the nfw_flow_firewall_logs dataset and then choose
- Choose New dataset and select the already created data source anflogs_datasource.
- Choose Create dataset and then repeat steps 3, 4, and 5 to create another dataset by choosing the alert logs Athena view nfw_alert_firewall_logs under the database anflogsdb.
Step 6: Create your analysis
Amazon QuickSight analysis let you build and publish dashboards with interactive visualizations that you can use to explore data, uncover trends, and gain insights. In this step, you will add your Athena datasets to QuickSight for analysis.
- On the QuickSight console, choose Analyses from the navigation pane and then choose New analysis.
- Select one of the Athena datasets created in the previous step and choose USE IN ANALYSIS.
- Keep the default options and choose
- Choose Dataset and select Add a new dataset. Choose the other Athena dataset that you created. Choose Select and Close.
Step 7: Visualize Network Firewall logs in QuickSight
The Athena datasets, added to QuickSight, can now be used to build visualizations to analyze AWS Network Firewall logs. You can use various attributes from the datasets to create your visualizations as needed.
As an example, create a dashboard to get the list of source IPs, sorted by the number of times it appears in the flow logs. This can help you understand the outliers and identify potentially malicious activity in your AWS account. You can mitigate the risk in a timely manner by taking necessary steps such as blocking suspicious traffic or implementing rate limits.
Create a visual for analyzing the most active source IPs on your network:
- On the Analysis page select the flow logs dataset nfw_flow_firewall_logs from the Dataset list under the Data
- From the menu bar choose Insert and then Add visual. A new, blank visual is displayed, and AutoGraph is selected by default.
- On the Visuals pane, choose the tile for Table.
- For this example, add the src_ip field to the Value This shows the data without any aggregation. To have an aggregated view of the Source IPs, choose scr_ip again from the Data pane and add it to the Group by field.
- Next, navigate to the ellipses beside the Value column and use Sort Order to sort by descending order.
You can explore various field attributes and create different visuals to analyze the Network Firewall logs, as shown in the following figure. The following are some of the use cases that you can consider:
- Analysis of the most active source and destination IPs.
- Trend of network traffic according to protocol types.
- The number of blocked and allowed events within the alert logs.
- Trends of packets and bytes on a daily, monthly, or quarterly basis.
- Hostname overview.
- Logs analysis for a particular time period or specific firewall by using date filter or multi-select filter respectively.
Step 8: Publish the Analysis as Dashboard and also share within your organization
- On the Analysis page, choose Publish at the upper right corner.
- On the Publish a dashboard window, choose the option that meets your requirement:
- To create a new dashboard, choose Publish new dashboard as, and enter a name.
- To replace an existing dashboard, you have two options:
- To update a dashboard with your changes, choose Replace an existing dashboard and then select a dashboard from the dropdown.
- To rename a dashboard, choose Replace an existing dashboard, select the dashboard from the list, choose Rename, and enter the new name.
Replacing a dashboard updates it without altering security or report schedule configuration.
- Choose Publish dashboard.
- Next, share the dashboard with users or groups within your organization:
- Go to the published dashboard and choose the Share icon at the upper right corner. Select Share dashboard.
- Search for the user email or group name under Invite users and groups to dashboard. Only active users and groups appear in the search results.
- For the users or groups that you want to grant access to the dashboard, choose Add.
- For the access level, select Viewer or Co-owner, depending on the user’s QuickSight role.
- This will also send out email notifications containing the dashboard link to the users.
- Alternatively, you can use the toggle to enable dashboard access for everyone in the account.
Clean up
Use the following steps to clean up the AWS resources deployed in this post and avoid incurring additional charges:
- If you created the resources using the sample CloudFormation template, delete the CloudFormation stack.
- Delete the network firewall and firewall policy if you created them for the purpose of this post.
- Empty the S3 bucket containing the sample logs and delete the bucket.
- Delete the Glue database and Glue crawler.
- Delete the IAM role AWSGlueServiceRole-network_firewall_logs.
- Delete the Amazon QuickSight assets, keeping the QuickSight subscription active:
- To cancel the QuickSight subscription, see Cancelling Your Amazon QuickSight Subscription. Delete the Amazon QuickSight account only if you’re sure that it isn’t being used by any other users.
Conclusion
In this post we discussed the process of creating a comprehensive view of AWS Network Firewall logs using Amazon QuickSight. We explained the steps and resources to construct a tailored analytics dashboard within QuickSight, enabling a better understanding of network events and traffic patterns. This approach aids in identifying anomalies, threats, and network events with fine-grained insights during troubleshooting. For further insights into how QuickSight can benefit businesses with dashboards and reports, see Getting started with Amazon QuickSight.
About the authors
Yash Bindlish is an Enterprise Support Manager at Amazon Web Services. He has more than 18 years of industry experience including roles in cloud architecture, systems engineering, and infrastructure. He works with Global Enterprise customers and help them build, scalable, modern and cost effective solutions on their growth journey with AWS. He loves solving complex problems with his solution-oriented approach.
Harsh Chheda is a Technical Account Manager at Amazon Web Services with 9 years of industry experience. He supports Global Enterprise customers throughout their AWS journey, helping them maintain highly resilient, cost-effective, and secure cloud infrastructure. He is part of AWS data analytics expert cohort and supports customers with their data visualization use cases.
Abhishek Dey is a Technical Account Manager for the Financial Services Industry at Amazon, bringing over 12 years of industry experience. He leads the Networking Field Community for Enterprise Support in India and is a lifelong learner and tech enthusiast, with a strong focus on Generative AI and cutting-edge technologies.
Ashish Jain is a Technical Account Manager at Amazon Web Services, committed to accelerate the cloud journey for AWS Global Enterprise customers. He has 16+ years of industry experience across systems engineering, cloud operations, Microsoft workloads, cloud infrastructure management and customer advocacy. He is passionate about cloud technologies and strives to leverage them towards his customers’ success on AWS cloud.