Networking & Content Delivery

Visualize and gain insights into your VPC with Amazon Q in AWS Quick Sight

Introduction

AWS services generate rich log and metric data, enabling you to create comprehensive dashboards that reveal valuable insights, including detailed visibility into Virtual Private Cloud (VPC) connectivity patterns. This post demonstrates how AWS Quick Sight and AWS Quick Sight enable data visualization from any source. We focus on visualizing connectivity patterns in VPCs to showcase the benefits of Quick Sight for users with varying technical expertise.

AWS Quick Sight is a fully managed, cloud-scale business intelligence (BI) service that enables users to create and share interactive dashboards, analyze data, and gain insights through visualizations. On April 30th, 2024, AWS announced the general availability of AWS Quick Sight, which enhances Quick Sight by providing generative artificial intelligence (AI) capabilities to visualize data using natural language queries.

In the following sections, we explore two use cases that collect data from different sources, enhance it, and visualize it using Quick Sight. These examples demonstrate how Amazon Q can quickly generate visualizations from common natural language queries, showcasing the ease and efficiency of using Quick Sight and Q for generating insightful dashboards.

Overview

This post presents two use cases that demonstrate how to visualize VPC Flow log data using AWS Quick Sight. These examples work regardless of the source solution used to capture and enrich the data. Although we developed these use cases for this post, you can also use this as a reference for any use case that you have.

Use case #1: Enriched VPC Flow Logs with security group and cost tags

Figure 1 - Architecture diagram – Enriched VPC Flow Logs with Amazon Q in QuickSight

Figure 1: Architecture diagram – Enriched VPC Flow Logs with AWS Quick Sight

VPC Flow Logs generate records using predefined fields. AWS Lambda can enrich these Flow Logs with more fields, creating a customized view of VPC traffic flows.

In the architecture shown in Figure 1, VPC Flow Logs are generated and sent to Amazon Data Firehose. Logs are then forwarded to Lambda for enrichment. Lambda adds more attributes to the flows, such as the security group and cost tag for the Amazon Elastic Compute Cloud (Amazon EC2) instances in the VPC Flow Log entry, before delivering them to Amazon S3. AWS Glue Catalog stores the table definitions for the enriched VPC Flow logs. This allows Amazon Athena to query the data efficiently. Finally, Quick Sight visualizes the data from Athena. The data flow is as follows:

  1. VPC Flow Logs are enabled on an existing VPC by the AWS CloudFormation template.
  2. Log records are sent to Data Firehose.
  3. Data Firehose is configured to emit metrics and logs to Amazon CloudWatch for troubleshooting if necessary.
  4. Data Firehose invokes a Lambda to enrich the VPC Flow Logs with more attributes, such as Amazon EC2 security group and cost tag.
  5. Lambda is configured to emit metrics and logs to CloudWatch for troubleshooting if necessary.
  6. Enriched VPC Flow Logs are delivered to an S3 bucket.
  7. AWS Glue crawls the S3 bucket to catalog the data for querying.
  8. Athena queries the cataloged data from AWS Glue.
  9. Quick Sight uses Athena and Amazon S3 as a data source for the dataset. A blank analysis is created ready to begin visualizing the VPC Flow Logs.
  10. Amazon Q is enabled on the analysis through a Q topic. The Q topic allows Quick Sight to interpret natural language queries to facilitate the development of visuals in the analysis.

Use Case #2: VPC Flow Logs and Amazon Route 53 Resolver Logs

Figure 2 - Architecture diagram – Enhanced VPC Traffic Analysis with Route 53 Resolver Logs using Amazon Q in QuickSight

Figure 2: Architecture diagram – Enhanced VPC Traffic Analysis with Route 53 Resolver Logs using AWS Quick Sight

VPC Flow Logs and Amazon Route 53 Resolver Logs provide complementary information about network traffic in your VPC.

In the architecture shown in Figure 2, VPC Flow Logs capture detailed information on traffic in your VPC between different IP addresses, but do not include domain names. This combination offers a comprehensive view of your VPC network traffic. The data flow is as follows:

  1. VPC Flow Logs are enabled on an existing VPC by the CloudFormation template.
  2. VPC Flow logs are sent to Amazon S3.
  3. Route 53 Resolver Logs are also created and associated to the VPC for which the VPC Flow Logs were created.
  4. Route 53 Resolver Logs are sent to Amazon S3
  5. Lambda is used to execute Athena queries for merging VPC Flow Logs and Route 53 Resolver Logs.
  6. Both VPC Flow Logs and Route 53 Resolver Logs are combined based on the IP addresses each day. These map network traffic in the VPC Flow Logs to the domain names and keep the analysis up to date on a daily basis. Changes to the domain name to IP mapping are updated accordingly.
  7. Quick Sight uses Athena and Amazon S3 as a data source for the dataset. A blank analysis is created ready to begin visualizing the VPC Flow Logs.
  8. Amazon Q is enabled on the analysis through a Q topic. The Q topic allows Quick Sight to interpret natural language queries to facilitate the development of visuals in the analysis.

This post provides a walkthrough for the first use case. For more detailed information on both use cases, such as walkthroughs and related CloudFormation templates, visit the AWS Samples GitHub repository.

Prerequisites

Before we begin, follow these steps:

  1. Deploy one of the CloudFormationtemplates provided in the aws-samples GitHub repository. Each example in this repository has its own detailed documentation hosted in the aws-samples GitHub repository.
  2. Make sure your Quick Sight user has a PRO role to access the Amazon Q functionality demonstrated in this post.

This post provides two examples. We may add more examples to the aws-samples repository in the future. The walkthrough steps in this post apply to all provided examples.

Walkthrough: Gain insights into your data with Q in Quick Sight

  1. Review the preconfigured Q topics in the deployed CloudFormation template. Q topics streamline visualization queries by establishing natural language synonyms to data fields.
  2. Query Q in Quick Sight for insights into your VPC Flow Log data.

1. Review Q topics in Quick Sight

In this section we review the Q topic deployed by the CloudFormation template.

Follow these steps as shown in Figure 3.

  1. Open the AWS Console and navigate to Quick Sight.
  2. In the left-hand pane, choose Topics.
  3. Choose the topic name.

Figure 3 - Q topics

Figure 3: Q topics

Choose the tabs at the top, navigate to Data, then DATA FIELDS. Quick Sight automatically populates the data fields with the fields from the dataset. It also populates synonyms for the corresponding fields where possible. In this example, the synonyms have been defined in the CloudFormation template. Synonyms help Amazon Q in correlating your queries with the fields in the data set. You can further customize the data fields to match your organization’s terminology. This results in more meaningful and accurate responses from Amazon Q. Refresh the Q topic indexes to reflect these changes in future analyses.

Figure 4 - Q topics – edit topic

Figure 4: Q topics – edit topic

Another helpful customization is Field Value Synonyms. This allows you to add synonyms to values in the VPC Flow Logs data. In the following example, synonyms have been added to IP protocols numbers. This allows Q to correctly interpret queries of VPC Flow Logs with the words icmp, tcp, udp.

Figure 5 - Q topics – edit synonyms

Figure 5: Q topics – edit synonyms

2. Query Q for insights into the VPC Flow Log data

We ask Q in Quick Sight a question regarding the VPC Flow Log data. Choose Ask a question about at the top center of the Quick Sight screen.

Figure 6 - Q topics – ask a question

Figure 6: Q topics – ask a question

As shown in the following Figure 7, we have asked Amazon Q “which cost tags had the most megabytes in October 2024 by hour”. Amazon Q has processed our question and, using the defined synonyms, interpreted the question as “Total Megabytes by Start Datetime hour and Cost Tag for Start Datetime in October 2024”. Amazon Q provided a response to the question, supported by detailed visual data backing the response.

Figure 7 - Q topics – question result

Figure 7: Q topics – question result

Walkthrough: populate Quick Sight Analysis with Amazon Q

  1. Link the Q topic with the Quick Sight Analysis
  2. Populate the Quick Sight Analysis by adding visuals generated by Amazon Q. This post provides a few example queries.

1. Link Quick Sight Analysis to Q topic

After reviewing Q topics, we explore Analyses. An Analyses is a collection of visualizations, interactive dashboards, and data insights that are created and organized on one or more sheets to explore and present data effectively.

We create our first visualizations in the Analyses. The CloudFormation template has created a blank Analysis linked to the dataset that imports VPC Flow Logs data from Athena. Choose the Analysis name to begin, as shown in Figure 8.

Figure 8 - Amazon Q Analyses

Figure 8: Amazon Q Analyses

To run queries on Amazon Q, link the Q topic to the Analysis. On the center of the top bar, choose the vertical ellipsis (⋮) next to Build visual, and choose Topic Linking from the menu. Enable the Link topic for Build Visual and Q&A option, and choose the topic deployed by CloudFormation from the drop down. Choose APPLY CHANGES, as shown in Figure 9.

Figure 9 - Amazon Q Analyses – topic linking

Figure 9: Amazon Q Analyses – topic linking

2. Populate Quick Sight Analysis

On the center of the top bar, choose Build visual. The Build a visual right pane is revealed. Begin by typing the first natural language question and choose BUILD. We have provided four sample questions to get you started for the first use case. The Quick Sight documentation provides types of questions supported by Q with more sample questions. Quick Sight Enterprise Edition is prerequisite to ask questions using Amazon Q.

Sample questions:

  • show top source and destination ip by gigabyte
  • show top source ip and path internet gateway by gigabyte
  • show top security group by megabytes
  • show megabytes egress date start May by hour

The second use case has its own sample questions documented in the GitHub repository.

Amazon Q interprets the questions and derives a query based on the field from the dataset and the defined synonyms in the linked topic. Amazon Q also chooses a visual type, which can be changed. On the top right of the visual, choose the image of a bar graph and choose the visual type that best visually represents the data for you. Choose ADD TO ANALYSIS, as shown in Figure 10.

Figure 10 - Amazon Q Analyses – build visual

Figure 10: Amazon Q Analyses – build visual

The following dashboard has been populated with the visuals from the four examples questions, as shown in Figure 11.

Figure 11 - Amazon Q Analyses – deployed visuals

Figure 11: Amazon Q Analyses – deployed visuals

Conclusion

In this post, you have learned how to use AWS Quick Sight to visualize data from various data sources using natural language queries. AWS Quick Sight democratizes data access, empowering everyone in your organization to make data-driven decisions. Organizing your data into intuitive topics and enabling natural language queries allows Amazon Q users to gain insights to VPC flow. To help your team get started with Quick Sight, we recommend watching the following tutorials What is AWS Quick Sight Q and Best practices for Quick Sight Q Authors.

About the authors

Author - Rashmiman Ray

Rashmiman Ray

Rashmiman is a Technical Account Manager at AWS, based out of New Jersey. He works with AWS Enterprise customers, providing technical guidance and best practice recommendations to help them succeed in the cloud. Outside of work, he enjoys hiking on trails, playing cricket, and cooking Indian delicacies.

Opeoluwa Victor Babasanmi

Opeoluwa Victor Babasanmi

Victor is a Sr. Networking Specialist Solutions Architect at AWS. He focuses on providing customers with technical guidance on planning and building solutions using best practices, and proactively keeps their AWS environments operationally healthy. When he is not helping customers, you may find him playing soccer, working out, or looking for a new adventure somewhere.

Author - Diego Hernandez

Diego Hernandez

Diego Hernandez is a Technical Account Manager based in Canada. Diego passion is all things networking. In his spare time, Diego enjoys spending time with his family and snowboarding.