Amazon Redshift Administration and FinOps Using LTI Canvas Glide

By Rajan Seshadri, LTI AWS Architect – LTI COE (DATA)
By Shivaji Murkute, LTI AWS Architect – LTI COE (DATA)
By Francois van Rensburg, Sr. Partner Management Solutions Architect – AWS

LTI

As organizations acquire more and more data, the need to effectively store and query the data has become very important.

Organizations have been moving their data to Amazon Web Services (AWS) and Amazon Redshift, a fully managed, petabyte-scale data warehouse service that makes it easy and cost effective to analyze data.

This has also brought about the need for resources with a financial background, such as CFOs, financial planners, business analysts, procurement, sourcing, and accounts payable. These skills can help a company get a good understanding of the cloud billing model so they can optimize costs.

Success requires financial and technology resources to collaborate in the procurement process, incentive tracking, cost allocation, and by creating standard operating procedures for handling cloud spend using a common denominator for communicating on financial concepts. All of this has created new requirements from technology resources to understand organizational billing dynamics.

Canvas Glide from Larsen and Toubro Infotech (LTI) is a tool for administering Amazon Redshift’s day-to-day routine tasks. It also helps organizations manage all financial aspects related to cost allocation, chargeback, monitoring, and planning of Amazon Redshift clusters.

In this post, we will cover how Canvas Glide assists administrators to successfully manage their Amazon Redshift clusters—both technically and financially—and guides them on common maintenance tasks.

LTI is an AWS Premier Tier Consulting Partner with the Financial Services Competency and Amazon Redshift service delivery designation. LTI helps more than 400 clients succeed in a converging world as a global technology consulting and digital solutions company.

Introducing Canvas Glide

Canvas Glide is a tool developed and maintained by LTI that connects to Amazon Redshift and enables an organization to administer the data warehouse.

Administering a data warehouse of petabyte scale requires skills in monitoring cluster performance, assigning priorities to various workloads, creating, and maintaining users and their access privileges, creating data shares and database performance monitoring, and more.

Canvas Glide helps with Amazon Redshift database tasks through a graphical user interface (GUI) tool and does the heavy lifting by executing software commands behind the scenes. Glide saves on cost and streamlines operations by providing insights into cluster usage, providing reports, predicting spend, and sending notifications on events.

Following, we discuss some of the key features of Canvas Glide from LTI.

Database 360

Database 360 displays a 360-degree view of the Amazon Redshift cluster, including the number of clusters and number of databases and tables in it. Additionally, information such as average performance and storage usage of the cluster are also displayed.

Cost Monitoring

Cost monitoring enables users to view, analyze, and manage the costs associated with various Amazon Redshift clusters and get charges incurred by each cluster, which can further break down by individual business units that access the cluster.

Using a simple cost projection, the process projects the future cost of the Amazon Redshift cluster based on current usage.

Workload Management

Workload management enables users extract, transform, load (ETL) workloads to be grouped based on the homogeneity of the workloads, or based on the business groups they are serving.

By assigning priority to groups, using workload management slots based on criticality, optimal performance can be achieved on the query execution.

Using Glide, authorized users can manage the following:

Priority of user groups by altering the number of execution units they are assigned.
Rules relating to query execution and other configuration parameters.
Concurrency scaling for a queue; this feature routes queries automatically to a transient cluster.
Manage parameter groups and the queues contained.

Data Sharing

Data sharing enables sharing data for read purposes across many Amazon Redshift clusters within or outside of an organization’s accounts. Access is granted based on the consumer’s AWS account ID and user.

Glide provides the ability to create and modify data shares that include the permissions to access the underlying objects, and to grant them to consumers. Glide internally manages all of the Amazon Redshift commands relating to data sharing actions, thus simplifying the management process.

User Management

Glide provides a user-friendly way to manage all cluster user management tasks in the form of:

Create and delete database users.
Create and delete user groups.
Assign and unassign users to user groups.
Grant and revoke privileges (select, update, and delete) on database objects to user groups.

Health Analysis

Canvas Glide enables organizations to configure and track the health of Amazon Redshift cluster usage using the below metrics and use them for cluster health monitoring and performance analysis:

CPU utilization
Disk space utilization
Database connections
Health status
Query duration
Query throughput
Concurrency scaling activity

Analytics and Insights

Using historical data points by various metrics, graphical charts are presented for visualization. A few of the charts are explained below:

Performance analysis: Monitoring the performance of the cluster is important for better utilization and user experience. Canvas Glide uses metrics such as CPU utilization, read throughput and write throughput, and conducts historical analysis providing recommendations on resizing along with Amazon Redshift node types.
Cost analysis: When spinning up an Amazon Redshift cluster, most of the monthly costs are static. There are a few exceptions to that rule, however:
- Concurrency scaling – For bursty workloads, this feature allows an Amazon Redshift cluster to scale horizontally to process requests in parallel. Organizations pay only for actual usage with per-second billing and accumulate one hour of concurrency scaling cluster credits every 24 hours while the main cluster is running.
- Managed storage – When using nodes of type RA3, the managed storage is billed per GB up to 64 TB. The only way to monitor storage consumption is the Percentage Disk Space Used metric. Glide facilities monitoring the concurrency scaling seconds metrics and provides analysis on historical data and options to notify whenever the sum within 24 hours is greater than 3,600. The tool also facilities monitoring storage consumption on historical data of the Percentage Disk Space Used metric.
Storage quotas analysis: The smallest node types for Amazon Redshift come with 160 GB storage, and the biggest node type offers up to 64 TB. Glide facilities monitoring the Percentage Disk Space Used metric closely to avoid that a cluster will not run out of storage capacity and Administrator can take necessary action at the right time.
Cluster health analysis: The health of an Amazon Redshift cluster is the most important to monitor. Admins should know whenever there’s an issue with the cluster, or when the cluster is unavailable due to scheduled AWS maintenance work. Glide facilities the Health Status metric that indicates whether a cluster is healthy and reports historical health analysis providing additional information required for contacting AWS support.

How it Works

Canvas Glide is deployed within a customer’s AWS account and is a container-based solution.

It can be deployed on an Amazon Elastic Container Service (Amazon ECS) or AWS Fargate across multiple AWS Availability Zones (AZs) and placed behind a load balancer. The list of Amazon Redshift clusters that should be managed are passed as parameters through an initialization file.

Additionally, a local database based upon Postgres created within this virtual private cloud’s DB subnet is included in this initialization file and used by Glide to maintain historical details about the various functionalities it manages.

Users access Glide through HTML-based web pages. Authorizations to the pages are controlled through an identity management solution using OAUTH2 / SAML protocol. The URL of the IDM that governs users’ access is passed as an input to the initialization file.

The deployable for Canvas Glide is a Docker image which can be run on the AWS container service like ECS along with the mandatory parameters, as described above, and passed through the initialization file.

Figure 1 – Physical architecture of Canvas Glide.

The diagram above includes the essential components such as load balancer, container service, Postgres database, and the identity management system required as part of the Canvas Glide installation. As the solution is more focused on managing the Amazon Redshift cluster, it’s expected to be access purely within the corporate net.

The logical architecture represented below has three important layers: presentation, business, and database.

Figure 2 – Logical architecture of Canvas Glide.

The presentation layer features the HTML pages driven by various widgets needed for presentation of the Glide user interface.

The business layer has all of the in-built business logic essential for the overall functioning of the application. This layer interacts with the Amazon Redshift cluster and database for all of the important activities highlighted above. In addition, this layer connects with a local database to store all statistics for historical analysis.

Finally, the database layer features connection objects to interact with the Amazon Redshift clusters that Glide manages, and a PostgreSQL database where historical data is stored.

Installation

Canvas Glide and its dependencies are containerized into a Docker image and, hence, can be executed in any environment that supports Docker. Amazon ECS and AWS Fargate are supported as deployment targets.

The list of clusters to administer along with its JDBC URL can be passed as a parameter file to the Docker image as part of the installation. The local database required for storing historical data for the purpose of analysis is also containerized with a PostgreSQL opensource database. As an alternate, Amazon RDS for PostgreSQL can be used.

Post successful installation, Canvas Glide can be accessed connecting via a browser to the user interface.

Summary

In this post, we explained how Amazon Redshift is a key service for organizations to manage vast amounts of data in a cost-efficient and reliable way. We also discussed the responsibility placed on technical resources such as CFOs, financial planners, and business analysts to provide solutions to address organizational financial requirements.

For this, LTI has developed Canvas Glide for managing Amazon Redshift clusters and its important features. Glide assists technical resources in forecasting cluster requirements and simplifies financial management and billback of cluster resources.

Glide saves administrators time and effort while providing the necessary financial inputs required by an organization’s financial teams.

.

.

Larsen and Toubro Infotech (LTI) – AWS Partner Spotlight

LTI is an AWS Premier Tier Consulting Partner that helps more than 400 clients succeed in a converging world as a global technology consulting and digital solutions company.

Contact LTI | Partner Overview

*Already worked with LTI? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.

AWS Partner Network (APN) Blog