Databricks Simplifies Deployment Using AWS Quick Start

When Databricks was faced with the challenge of reducing complex configuration steps and time to deployment of Databricks workspaces to the Amazon Web Services (AWS) Cloud, it worked with the AWS Integration and Automation team to design an AWS Quick Start, an automated reference architecture built on AWS CloudFormation templates with integrated best practices. As a result, the Quick Start significantly reduced errors and time to deployment, replaced complex configuration steps with a simple interface, and eliminated the need for customers to become AWS experts. The Quick Start is now a preferred method of deployment for hundreds of customers and Databricks solutions architects.

Since its start in 2013, Databricks has helped more than five thousand customers worldwide enable massive-scale data engineering, collaborative data science, full-lifecycle machine learning, and business analytics. The Databricks workspace is the core component that customers deploy to their environments to access Databricks assets, objects, and computational resources such as clusters and jobs. Deploying the Databricks workspace on AWS requires customers to switch back and forth between consoles while they follow along with detailed documentation. Deployment steps include configuring a virtual private cloud (VPC), setting up security groups, creating a cross-account AWS Identity and Access Management (IAM) role, and adding all of the AWS services used in the workspace. The deployment can take over an hour and usually requires the help of a Databricks solutions architect who is familiar with AWS. 

With Databricks growing at a fast rate, sometimes doubling its customer base in just a year, the company needed a simpler and faster solution for deploying Databricks on AWS – a solution that didn’t require excessive context switching, step-heavy documentation, and AWS expertise. And, with more security requirements always needed to keep up with the latest innovations and technology, the deployment was bound to get more complicated with even more steps and decisions to make along the way. When Denis Dubeau joined Databricks as the AWS Partner Solutions Architect in 2019, he soon became the go-to person for Databricks solutions architects needing his help deploying workspaces on AWS. Denis immediately recognized the complexity of the process and decided to look for a better solution. He took his quest to re:Invent, AWS’s premier events for the cloud computing community. 

At re:Invent, Denis learned about AWS Quick Starts and how they are specifically designed to help save time by eliminating hundreds of manual installation and configuration steps through automated AWS Cloud deployments. He learned that Quick Starts also implement AWS best practices, so they're designed with security and availability from the start. Denis soon engaged with the AWS Integration and Automation team to discuss requirements and start planning a new Databricks on AWS Quick Start. Together, they created the source control for artifacts, the development environment for building the Quick Start, and the CI/CD pipeline for continuously building and testing code changes. They wrote and tested the AWS CloudFormation templates on several AWS Regions with multiple input parameters, and finally packaged it all together as a Quick Start.

After presenting the Quick Start to Databricks executives and performing several proof-of-concept tests with some of the more complicated security options, the Quick Start was ready for customers. The CloudFormation templates are written in YAML and extended by an AWS Lambda-backed custom resource written in Python. The templates create and configure the AWS resources required to deploy and configure the Databricks workspace by invoking API calls for a given Databricks account, AWS account, and Region. Customers can even download and customize the templates. To deploy Databricks on AWS using the Quick Start, customers simply sign in to the AWS Management Console, select the CloudFormation template and Region, fill in a handful of parameter values, and choose to deploy. And within about five minutes, the Databricks workspace is fully deployed and running.

The architecture of the Quick Start is designed so that customers with a wide variety of environments can take advantage of the deployment architecture. It includes: 

• A highly available (HA) architecture that spans at least two Availability Zones.
• A Databricks-managed or customer-managed virtual private cloud (VPC) in the customer's AWS account. This VPC is configured with private subnets and a public subnet, according to AWS best practices, to provide you with your own virtual network on AWS.
• In the private subnets:
  • Databricks clusters of Amazon Elastic Compute Cloud (Amazon EC2) instances.
  • One or more security groups to enable secure cluster connectivity.
• In the public subnet:
  • A network address translation (NAT) gateway to allow outbound internet access.
• Amazon CloudWatch for the Databricks workspace instance logs.
• (Optional) A customer-managed AWS Key Management Service (AWS KMS) key to encrypt notebooks.
• An Amazon Simple Storage Service (Amazon S3) bucket to store objects such as cluster logs, notebook revisions, and job results.
• AWS Security Token Service (AWS STS) for requesting temporary, limited-privilege credentials for users to authenticate.
• A VPC endpoint for accessing S3 artifacts and logs.
• A cross-account IAM role to enable Databricks to deploy clusters in the VPC for the new workspace. Customers can either create the IAM role during deployment or use an existing IAM role. 

The Quick Start comes with a deployment guide with details about the architecture, guidance on the CloudFormation templates and parameters, troubleshooting tips, license information, and product and contact information.

With the Quick Start, instead of having to follow along with step-heavy documentation and switch back and forth between various applications and interfaces, including the database environment and the AWS Management Console, customers can simply launch a CloudFormation template, fill in a few parameter fields, and deploy a fully functioning Databricks workspace in just a few minutes. 

Before the Quick Start, customers almost always needed the help of a solutions architect to deploy the Databricks workspace on AWS, an error-prone process that used to take at least an hour. Now, customers can use the Quick Start to automate their deployment without needing assistance, giving the solutions architects more time to attend to their job duties and perform business-critical tasks.

The CloudFormation templates used in the Quick Start automatically create and configure the required AWS resources needed to deploy and configure the Databricks workspace. Customers and Databricks solutions architects don’t need to be experts in AWS and solutions architects don’t have to go through extensive AWS training before assisting customers with their deployments. In most situations, the architects can simply tell customers which parameter values they need (account credentials, Databricks workspace name, and AWS Region) for the two deployment options and point them to the Quick Start. And with the fast rate that Databricks is gaining customers, this type of self-service model is just what the company needs to keep up with growing demands for product enhancements, security upgrades, and user experience improvements.

As part of an effort to incorporate best practices into its deployment process, Databricks is planning to deliver a series of updates over the next few months so that customers can take advantage of those improvements. Typically, rolling out these updates requires customers to perform additional steps and piece-meal modifications to already existing deployments. The Quick Start eliminates these manual steps by always being up to date with the latest updates from both Databricks and AWS. This means that customers already have the mechanisms built into the modular design of the Quick Start to automatically receive those updates. For example, the Quick Start automatically adds a lifecycle management policy on the S3 bucket directly at deployment time. AWS recommends this best practice to help customers manage their storage space in the most cost-effective manner; however, some customers and solutions architects who aren’t experts in AWS overlook these valuable components.

Since the official launch of the Quick Start in September 2020, customers have used it to deploy over 800 Databricks workspaces with the following results:

• Reduction in deployment time from over an hour to 5 minutes.
• A 95% reduction in deployment errors.
• 90% adoption rate by solutions architects as their preferred method for deployment.

The Quick Start automates the manual tasks that require customers to switch back and forth between interfaces and applications, and the self-service nature of the Quick Start lets customers deploy their own workspaces without the need for assistance. With such a short deployment time, customers can more easily troubleshoot issues and redeploy as many times as needed. 

The Quick Start is now used to deploy workspaces used in Databricks hands-on workshops. Preparing deployed workspaces for these workshops used to require several days of setup and planning by a third party. In May 2021, 200 workspaces were deployed by customers with a 100% success rate. 

The Quick Start is also used in the 14-day trial version of Databricks and has increased the success rate of deployment to over 80%. Before the Quick Start, when customers signed up for the trial, they would receive a welcome email with their account information and a link to instructions that required a lot of context switching and decision making. Now the trial launches the Quick Start template, prompts for the password, and customers are off and running with the deployment.