Visualize and gain insights into your VPC with Amazon Q in Amazon QuickSight

Introduction

AWS services generate rich log and metric data, enabling you to create comprehensive dashboards that reveal valuable insights, including detailed visibility into Virtual Private Cloud (VPC) connectivity patterns. This post demonstrates how Amazon QuickSight and Amazon Q in QuickSight enable data visualization from any source. We focus on visualizing connectivity patterns in VPCs to showcase the benefits of QuickSight for users with varying technical expertise.

Amazon QuickSight is a fully managed, cloud-scale business intelligence (BI) service that enables users to create and share interactive dashboards, analyze data, and gain insights through visualizations. On April 30th, 2024, AWS announced the general availability of Amazon Q in QuickSight, which enhances QuickSight by providing generative artificial intelligence (AI) capabilities to visualize data using natural language queries.

In the following sections, we explore two use cases that collect data from different sources, enhance it, and visualize it using QuickSight. These examples demonstrate how Amazon Q can quickly generate visualizations from common natural language queries, showcasing the ease and efficiency of using QuickSight and Q for generating insightful dashboards.

Overview

This post presents two use cases that demonstrate how to visualize VPC Flow log data using Amazon Q in QuickSight. These examples work regardless of the source solution used to capture and enrich the data. Although we developed these use cases for this post, you can also use this as a reference for any use case that you have.

Use case #1: Enriched VPC Flow Logs with security group and cost tags

Figure 1: Architecture diagram – Enriched VPC Flow Logs with Amazon Q in QuickSight

VPC Flow Logs generate records using predefined fields. AWS Lambda can enrich these Flow Logs with more fields, creating a customized view of VPC traffic flows.

In the architecture shown in Figure 1, VPC Flow Logs are generated and sent to Amazon Data Firehose. Logs are then forwarded to Lambda for enrichment. Lambda adds more attributes to the flows, such as the security group and cost tag for the Amazon Elastic Compute Cloud (Amazon EC2) instances in the VPC Flow Log entry, before delivering them to Amazon S3. AWS Glue Catalog stores the table definitions for the enriched VPC Flow logs. This allows Amazon Athena to query the data efficiently. Finally, QuickSight visualizes the data from Athena. The data flow is as follows:

1. VPC Flow Logs are enabled on an existing VPC by the AWS CloudFormation template.
2. Log records are sent to Data Firehose.
3. Data Firehose is configured to emit metrics and logs to Amazon CloudWatch for troubleshooting if necessary.
4. Data Firehose invokes a Lambda to enrich the VPC Flow Logs with more attributes, such as Amazon EC2 security group and cost tag.
5. Lambda is configured to emit metrics and logs to CloudWatch for troubleshooting if necessary.
6. Enriched VPC Flow Logs are delivered to an S3 bucket.
7. AWS Glue crawls the S3 bucket to catalog the data for querying.
8. Athena queries the cataloged data from AWS Glue.
9. QuickSight uses Athena and Amazon S3 as a data source for the dataset. A blank analysis is created ready to begin visualizing the VPC Flow Logs.
10. Amazon Q is enabled on the analysis through a Q topic. The Q topic allows Quicksight to interpret natural language queries to facilitate the development of visuals in the analysis.

Use Case #2: VPC Flow Logs and Amazon Route 53 Resolver Logs

Figure 2: Architecture diagram – Enhanced VPC Traffic Analysis with Route 53 Resolver Logs using Amazon Q in QuickSight

VPC Flow Logs and Amazon Route 53 Resolver Logs provide complementary information about network traffic in your VPC.

In the architecture shown in Figure 2, VPC Flow Logs capture detailed information on traffic in your VPC between different IP addresses, but do not include domain names. This combination offers a comprehensive view of your VPC network traffic. The data flow is as follows:

1. VPC Flow Logs are enabled on an existing VPC by the CloudFormation template.
2. VPC Flow logs are sent to Amazon S3.
3. Route 53 Resolver Logs are also created and associated to the VPC for which the VPC Flow Logs were created.
4. Route 53 Resolver Logs are sent to Amazon S3
5. Lambda is used to execute Athena queries for merging VPC Flow Logs and Route 53 Resolver Logs.
6. Both VPC Flow Logs and Route 53 Resolver Logs are combined based on the IP addresses each day. These map network traffic in the VPC Flow Logs to the domain names and keep the analysis up to date on a daily basis. Changes to the domain name to IP mapping are updated accordingly.
7. QuickSight uses Athena and Amazon S3 as a data source for the dataset. A blank analysis is created ready to begin visualizing the VPC Flow Logs.
8. Amazon Q is enabled on the analysis through a Q topic. The Q topic allows Quicksight to interpret natural language queries to facilitate the development of visuals in the analysis.

This post provides a walkthrough for the first use case. For more detailed information on both use cases, such as walkthroughs and related CloudFormation templates, visit the AWS Samples GitHub repository.

Prerequisites

Before we begin, follow these steps:

1. Deploy one of the CloudFormationtemplates provided in the aws-samples GitHub repository. Each example in this repository has its own detailed documentation hosted in the aws-samples GitHub repository.
2. Make sure your QuickSight user has a PRO role to access the Amazon Q functionality demonstrated in this post.

This post provides two examples. We may add more examples to the aws-samples repository in the future. The walkthrough steps in this post apply to all provided examples.

Walkthrough: Gain insights into your data with Q in QuickSight

1. Review the preconfigured Q topics in the deployed CloudFormation template. Q topics streamline visualization queries by establishing natural language synonyms to data fields.
2. Query Q in QuickSight for insights into your VPC Flow Log data.

1. Review Q topics in QuickSight

In this section we review the Q topic deployed by the CloudFormation template.

Follow these steps as shown in Figure 3.

1. Open the AWS Console and navigate to QuickSight.
2. In the left-hand pane, choose Topics.
3. Choose the topic name.

Figure 3: Q topics

Choose the tabs at the top, navigate to Data, then DATA FIELDS. QuickSight automatically populates the data fields with the fields from the dataset. It also populates synonyms for the corresponding fields where possible. In this example, the synonyms have been defined in the CloudFormation template. Synonyms help Amazon Q in correlating your queries with the fields in the data set. You can further customize the data fields to match your organization’s terminology. This results in more meaningful and accurate responses from Amazon Q. Refresh the Q topic indexes to reflect these changes in future analyses.

Figure 4: Q topics – edit topic

Another helpful customization is Field Value Synonyms. This allows you to add synonyms to values in the VPC Flow Logs data. In the following example, synonyms have been added to IP protocols numbers. This allows Q to correctly interpret queries of VPC Flow Logs with the words icmp, tcp, udp.

Figure 5: Q topics – edit synonyms

2. Query Q for insights into the VPC Flow Log data

We ask Q in QuickSight a question regarding the VPC Flow Log data. Choose Ask a question about at the top center of the QuickSight screen.

Figure 6: Q topics – ask a question

As shown in the following Figure 7, we have asked Amazon Q “which cost tags had the most megabytes in October 2024 by hour”. Amazon Q has processed our question and, using the defined synonyms, interpreted the question as “Total Megabytes by Start Datetime hour and Cost Tag for Start Datetime in October 2024”. Amazon Q provided a response to the question, supported by detailed visual data backing the response.

Figure 7: Q topics – question result

Walkthrough: populate QuickSight Analysis with Amazon Q

1. Link the Q topic with the QuickSight Analysis
2. Populate the QuickSight Analysis by adding visuals generated by Amazon Q. This post provides a few example queries.

1. Link QuickSight Analysis to Q topic

After reviewing Q topics, we explore Analyses. An Analyses is a collection of visualizations, interactive dashboards, and data insights that are created and organized on one or more sheets to explore and present data effectively.

We create our first visualizations in the Analyses. The CloudFormation template has created a blank Analysis linked to the dataset that imports VPC Flow Logs data from Athena. Choose the Analysis name to begin, as shown in Figure 8.

Figure 8: Amazon Q Analyses

To run queries on Amazon Q, link the Q topic to the Analysis. On the center of the top bar, choose the vertical ellipsis (⋮) next to Build visual, and choose Topic Linking from the menu. Enable the Link topic for Build Visual and Q&A option, and choose the topic deployed by CloudFormation from the drop down. Choose APPLY CHANGES, as shown in Figure 9.

Figure 9: Amazon Q Analyses – topic linking

2. Populate QuickSight Analysis

On the center of the top bar, choose Build visual. The Build a visual right pane is revealed. Begin by typing the first natural language question and choose BUILD. We have provided four sample questions to get you started for the first use case. The QuickSight documentation provides types of questions supported by Q with more sample questions. QuickSight Enterprise Edition is prerequisite to ask questions using Amazon Q.

Sample questions:

• show top source and destination ip by gigabyte
• show top source ip and path internet gateway by gigabyte
• show top security group by megabytes
• show megabytes egress date start May by hour

The second use case has its own sample questions documented in the GitHub repository.

Amazon Q interprets the questions and derives a query based on the field from the dataset and the defined synonyms in the linked topic. Amazon Q also chooses a visual type, which can be changed. On the top right of the visual, choose the image of a bar graph and choose the visual type that best visually represents the data for you. Choose ADD TO ANALYSIS, as shown in Figure 10.

Figure 10: Amazon Q Analyses – build visual

The following dashboard has been populated with the visuals from the four examples questions, as shown in Figure 11.

Figure 11: Amazon Q Analyses – deployed visuals

Conclusion

In this post, you have learned how to use Amazon QuickSight to visualize data from various data sources using natural language queries. Amazon Q in QuickSight democratizes data access, empowering everyone in your organization to make data-driven decisions. Organizing your data into intuitive topics and enabling natural language queries allows Amazon Q users to gain insights to VPC flow. To help your team get started with QuickSight, we recommend watching the following tutorials What is Amazon QuickSight Q and Best practices for QuickSight Q Authors.

About the authors