Automate Deployment of Amazon QuickSight Assets with AWS Step Functions and QuickSight Asset Bundle APIs

By Denys Tyshetskyy, Principal Data Specialist – CyberCX
By Jyothi Madanlal, Solutions Architect – AWS

CyberCX

Business intelligence (BI) and IT operations (BIOps) teams aim to automate and accelerate the deployment of BI assets to ensure business continuity.

The data team at CyberCX has extensive expertise in building end-to-end customer centric data analytics and machine learning (ML) solutions. In this post, we will share an architecture pattern to automate the the promotion of Amazon QuickSight assets through a typical multi-environment setup using AWS Step Functions and the simplified QuickSight Asset Bundle APIs.

This solution has been developed as a reusable module in the context of a wider CyberCX Serverless Data Platform designed and built by the CyberCX team. The same pattern can be adapted to a scenario which necessitates the creation of one or more staging environments from a production environment.

Amazon QuickSight is an easy-to-use, cloud-scale business intelligence tool that provides insights and analytics capabilities. When migrating a QuickSight visualization, the following assets are typically recreated per environment:

Analysis: Collection of visualizations that encompass visual elements such as diagrams, charts, and graphs.
Dashboard: Published version of an analysis.
Data source: Access point or foundation of data, which may be Amazon Athena, Amazon Redshift, Amazon RDS, or other service.
Dataset: Actual data from the data source that is used in the analysis.

Prior to the launch of the QuickSight Asset Bundle APIs, migrating these components from one environment to another was an intricate process that involved exporting each component individually, in a coordinated sequence, and subsequently importing them into the desired AWS account using QuickSight APIs. This was a multi-step process, prone to failures and errors that could lead to high operational complexity.

The QuickSight Asset Bundle APIs allow for the creation of a cohesive artifact, comprising the top-level analysis component and its dependencies, which can be bundled in a single step and exported into the destination AWS account. This simplifies and reduces operational overhead of the migration process while enabling bulk migrations.

CyberCX is an AWS Premier Tier Services Partner and AWS Marketplace Seller that embraces an AWS-first digital transformation approach through its deep AWS specialization.

Solution Overview

This solution relies on the seamless integration of AWS services to automate Amazon QuickSight asset migration and facilitate accelerated development.

The solution leverages:

AWS Step Functions: Serving as the orchestration framework, Step Functions simplify the coordination of the different asynchronous steps involved in QuickSight asset migration.
AWS Lambda: AWS Lambda functions enable the execution of intermediate computations and essential QuickSight API invocations required for the QuickSight asset migration.
Amazon S3: Used as a temporary storage for the exported artifact.

The entire solution is designed with security in mind and follows the principles of least privilege access. Encryption is turned on for the data stored in the Amazon Simple Storage Service (Amazon S3) bucket. and the bucket is also configured to be private.

A typical BIOps workflow may be as follows:

Business analyst creates a QuickSight analysis in the development environment.
Once validated, the analyst triggers the Step Function workflow to export the analysis from the development environment into S3, and then import it into the staging environment in the staging AWS account.

Figure 1 – Amazon QuickSight asset migration deployment diagram.

The sequence of the API calls made within the Step Function workflow is discussed below:

StartAssetBundleExportJob is triggered to start the export of one or more analyses, identified by the Amazon Resource Name (ARN) and dependencies, as shown below. You can specify an appropriate JOBID to easily identify this instance of an export job.

aws quicksight start-asset-bundle-export-job --include-all-dependencies --export-format QUICKSIGHT_JSON --aws-account-id <DEV_AWSACCOUNTID> --asset-bundle-export-job-id <JOBID> --resource-arns '[ "arn:aws:quicksight:<REGION>:<DEV_AWSACCOUNTID>:analysis/<ANALYSIS-ID_1>", "arn:aws:quicksight:<REGION>:<DEV_AWSACCOUNTID>:analysis/<ANALYSIS-ID_2>", ...]'

Poll for the completion of the export job uses the below command. On successful completion of the job, a {DOWNLOAD-URL} is returned in the output, which is used in the next step.

aws quicksight describe-asset-bundle-export-job --aws-account-id <DEV_AWSACCOUNTID> --asset-bundle-export-job-id <JOBID>

StartAssetBundleImportJob is triggered to import the assets into the environment deployed in account with {STAGING_AWSACCOUNTID}.

aws quicksight start-asset-bundle-import-job --aws-account-id <STAGING_AWSACCOUNTID> --asset-bundle-import-job-id <JOBID> --asset-bundle-import-source '{"S3Uri": "<DOWNLOAD-URL>"}' --failure-action ROLLBACK

The above process can be repeated for promotion of the QuickSight assets from staging and other non-production environments to one or more production environments.

The pattern was implemented to migrate manually-created QuickSight assets from the production environment into the development and staging accounts and simultaneously convert into an easily-managed and deployable artifact.

This pattern is also applicable to the following use cases:

Migration of assets between non-production environments.
Migration of assets from non-production to one or more production environment.
One-time migration of assets to recreate functionality in a new AWS region or new account.
Backup and restore of QuickSight assets.

For this solution, the transition to BIOps was implemented in two stages:

Migration took place as per the diagram in Figure 1. With an execution time in minutes, all of the artifacts were migrated in one run of the Step Function, from production into development and staging environments.
Day-to-day BIOps stage where a QuickSight analysis is created or updated in the development account, and is promoted to the staging and production environments.

Note the migration stage is not a prerequisite for the implementation of BIOps, but can be used should the need arise to create additional environments.

Customer Benefits

Improved productivity: The automated orchestration using AWS Step Functions the Amazon QuickSight Asset Bundle APIs boosts overall productivity and gives greater confidence in the time to deploy new or modified assets.
Innovation: With the deployment of a QuickSight asset being automated, the focus can be on innovation and deriving better insights from the data.
Scale and reduced time to market: Quickly deploy assets in new environments, accounts, or regions with a repeatable and fully managed Step Function workflow.
Flexible: As it does in the real world, this pattern is applicable for the first QuickSight asset as well as for implementing BIOps for already published and live QuickSight dashboards and assets.

CyberCX Serverless Data Lake Platform

This architecture pattern has been developed as a reusable module in the context of a wider data platform designed and built by the CyberCX data team. Here are some of the platform’s benefits:

AWS-native: Seamlessly integrating with AWS services, CyberCX’s Serverless Data Lake Platform is fortified with the versatility to leverage tailored tools for every task.
High modularity: With inherent modularity, the platform harmoniously integrates into existing data platforms to ensure seamless coexistence.
Speed to market: The rapid deployment of the data lake foundation (within 10-15 days) to the customer’s production environment ensures swift results.
DevOps and infrastructure as code (IaC): The platform provisions infrastructure across multiple environments, facilitating rigorous testing before deployment to production.

Note the presence of the data platform is not a prerequisite for the Amazon QuickSight deployment and migration solution.

Summary

This post described a pattern to automate the manual process of deploying of Amazon QuickSight assets in a typical multi-environment, multi-account setup. The approach increases deployment confidence, reducing manual error with a boost to overall productivity.

To learn more about CyberCX’s Serverless Data Lake Platform, including customer references, check out the CyberCX website and reach out.

Resources:

.

.

CyberCX – AWS Partner Spotlight

CyberCX is an AWS Premier Tier Services Partner that embraces an AWS-first digital transformation approach through its deep AWS specialization.

Contact Partner | Partner Overview | AWS Marketplace

AWS Partner Network (APN) Blog