Visualize live analytics from Amazon QuickSight connected to Amazon OpenSearch Service

Live analytics refers to the process of preparing and measuring data as soon as it enters the database or persistent store. In other words, you get insights or arrive at conclusions immediately. Live analytics enables businesses to respond to events without delay. You can seize opportunities or prevent problems before they happen. Speed is the main benefit of live analytics. The faster a business can use data for insights, the faster they can act on critical decisions.

Some live analytics use cases include:

• Analyzing access logs and application logs from servers to identify any server performance issues that could lead to application downtime or help detect unusual activity. For instance, analyzing monitoring data from a manufacturing line can help early intervention before machinery malfunctions.
• Targeting individual customers in retail outlets with promotions and incentives while the customers are in the store and close to the merchandise.

We see customers using real-time analytics using our ELK stack. The ELK stack is an acronym used to describe a stack that comprises three popular open-source projects: Elasticsearch, Logstash, and Kibana. Often referred to as Elasticsearch, the ELK stack gives you the ability to aggregate logs from all your systems and applications, analyze these logs, and create visualizations for application and infrastructure monitoring, faster troubleshooting, security analytics, and more. In this post, we extend the live analytics visualizations using Amazon QuickSight.

Solution overview

Amazon OpenSearch Service is a fully managed service that makes it easy for you to deploy, secure, and run OpenSearch cost-effectively at scale. You can build, monitor, and troubleshoot your applications using the tools you love at the scale you need. The service provides support for open-source OpenSearch APIs, managed Kibana, integration with Logstash and other AWS services, and built-in alerting and SQL querying. In addition, Amazon OpenSearch Service lets you pay only for what you use—there are no upfront costs or usage requirements. With Amazon OpenSearch Service, you get the ELK stack you need without the operational overhead.

QuickSight is a scalable, serverless, embeddable, machine learning (ML)-powered business intelligence (BI) service built for the cloud. QuickSight lets you easily create and publish interactive BI dashboards that include ML-powered insights. QuickSight dashboards can be accessed from any device and seamlessly embedded into your applications, portals, and websites.

This post helps you visualize the Centralized Logging solution using QuickSight. Centralized logging helps organizations collect, analyze, and display Amazon CloudWatch logs in a single dashboard in QuickSight.

This solution consolidates, manages, and analyzes log files from various sources. You can collect CloudWatch logs from multiple accounts and AWS Regions. Access log information can be beneficial in security and access audits. It can also help you learn about your customer base and understand your Amazon Simple Storage Service (Amazon S3) bill.

The following diagram illustrates the solution architecture.

For more information about the solution, see Centralized Logging.

Prerequisites

Before you implement the solution, complete the prerequisite steps in this section.

Provision your resources

Launch the following AWS CloudFormation template to launch the Centralized Logging solution:

After you create the stack, you receive an email (to the administrator email address) with your login information, as shown in the following screenshot.

Launch QuickSight in a VPC

Sign up for a QuickSight subscription with the Enterprise license.

QuickSight Enterprise Edition is fully integrated with Amazon Virtual Private Cloud (Amazon VPC). A VPC based on this service closely resembles a traditional network that you operate in your own data center. It enables you to secure and isolate traffic between resources.

Allow QuickSight to access Amazon OpenSearch Service

Make sure QuickSight has access to both the VPC and Amazon OpenSearch Service.

1. On the QuickSight dashboard, choose the user icon and choose Manage QuickSight.
2. Choose Security & permissions in the navigation pane.
3. Choose Add or Remove to update QuickSight access to AWS services.

4. For Allow access and autodiscovery for these recourses, select Amazon OpenSearch Service.

Manage the VPC and security group connections

You need to give permissions on the QuickSight console to connect to Amazon OpenSearch Service. After you enable Amazon OpenSearch Service on the Security & permissions page, you add a VPC connection with the same VPC and subnet as your Amazon OpenSearch Service domain and create a new security group.

You first create a security group for QuickSight.

1. Add an inbound rule to allow all communication from the Amazon OpenSearch Service domain.
2. For Type, choose All TCP.
3. For Source, select Custom, then enter the ID of the security group used by your Amazon OpenSearch Service domain.
4. Add an outbound rule to allow all traffic to the Amazon OpenSearch Service domain.
5. For Type, choose Custom TCP Rule.
6. For Port Range, enter 443.
7. For Destination, select Custom, then enter the ID of the security group used by your Amazon OpenSearch Service domain.

Next, you create a security group for the Amazon OpenSearch Service domain.

8. Add an inbound rule that allows all incoming traffic from the QuickSight security group.
9. For Type, choose Custom TCP.
10. For Port Range, enter 443.
11. For Source, select Custom, then enter the QuickSight security group ID.
12. Add an outbound rule that allows all traffic to the QuickSight security group.
13. For Type, choose All TCP.
14. For Destination, select Custom, then enter the QuickSight security group ID.

Choose your datasets

To validate the connection and create the data source, complete the following steps:

1. On the QuickSight console, choose Datasets.
2. Choose Create dataset.
3. Choose Amazon OpenSearch Service.
4. For Data source name, enter a name.
5. Depending on your Amazon OpenSearch Service connections of either public or VPC, choose your connection type and Amazon OpenSearch Service domain.
6. Choose Validate connection.
7. Choose Create data source.

8. Choose Tables.
9. Select the table in the data source you created.
   
10. Review your settings and choose Visualize.
    

Visualize the data loaded

QuickSight, with its wide array of visuals available, allows you to create meaningful visuals from Amazon OpenSearch Service data.

When you choose Visualize from the previous steps, you start creating an analysis. QuickSight provides a range of visual types to display data, such as graphs, tables, heat maps, scatter plots, line charts, pie charts, and more. The following steps allow you to add a visual type to display the data from the datasets.

1. On the Add menu, choose Add visual.
   
2. Choose your visual type.
   
3. Add fields to the field wells to bring data into the visuals to be displayed.
   

The following screenshot shows a sample group of visuals.

Automatically refresh your data

You can access and visualize your data through direct queries. Your data is queried live each time a visual is rendered. This gives you live access to your data. Additionally, you can automatically refresh the visuals every 1–60 minutes, so that you don’t have to reload the page to see the most up-to-date information. The following screenshot shows the auto-refresh settings while preparing to publish your dashboard.

For more information about the auto-refresh option, see Using Amazon OpenSearch with Amazon QuickSight.

The following screenshot shows an example visualization.

Clean up

When you’re done using this solution, to avoid incurring future charges, delete the resources you created in this walkthrough, including your S3 buckets,  Amazon OpenSearch Service cluster, and other associated resources.

Summary

This post demonstrated how to extend your ELK stack with QuickSight in a secure way for analyzing access logs. The application logs help you identify any server performance issues that could lead to application downtime. They can also help detect unusual activity.

As always, AWS welcomes feedback. Please submit comments or questions in the comments section.

About the Authors

Lokesh Yellanur is a Solutions Architect at AWS. He helps customers with data and analytics solutions in AWS.

Joshua Morrison is a Senior Solutions Architect at AWS based in Richmond, Virginia. He spends time working with customers to help with their adoption of modern cloud technology and security best practices. He enjoys being a father and picking up heavy objects.

Suresh Patnam is a Sr Solutions Architect at AWS; He works with customers to build IT strategy, making digital transformation through the cloud more accessible, focusing on big data, data lakes, and AI/ML. In his spare time, Suresh enjoys playing tennis and spending time with his family. Connect him on LinkedIn.