Scheduling Provisioning and Termination of Amazon EMR Clusters with AWS Service Catalog Connector for ServiceNow

By Itai Njanji, Senior Consultant, OI at AWS

Data scientists across various institutions are utilizing Amazon EMR clusters to run analytics workloads. These data scientists require prescriptive guidance to automatically schedule the provisioning and termination of Amazon EMR clusters to keep up with the rate of innovation, and to save cost.

Scheduling when to provision clusters allows data scientists to run analytics workloads at their own schedule. Scheduling when to terminate clusters allows them to run analytics workloads only when they need them.

In this post, I will demonstrates how to automatically schedule provisioning and termination of Amazon EMR clusters by utilizing AWS Service Catalog Connector for ServiceNow. The capability discussed in this post can also be extended to other AWS services.

ServiceNow is an AWS Partner Network (APN) Advanced Technology Partner that gives enterprises complete visibility and control of their IT environment, including virtualized and cloud infrastructure.

ServiceNow simplifies service mapping, delivery, and assurance, consolidating IT service and infrastructure data into a single system of record.

Definitions

AWS Service Catalog allows you to centrally manage commonly deployed AWS services and provisioned software products. It helps organizations achieve consistent governance and compliance requirements, while enabling users to quickly deploy only the approved AWS services they need.

ServiceNow is an enterprise service management platform that places a service‑oriented lens on the activities, tasks, and processes to enable a modern work environment.

ServiceNow Service Catalog is a self-service application that end users can leverage to order IT services based on request fulfillment approvals and workflows.

AWS Service Catalog Connector for ServiceNow is an integration that allows data scientists using ServiceNow to automatically schedule the provisioning and termination of Amazon EMR clusters that reside in AWS Service Catalog.

Prerequisites

Before continuing, please review the blog post titled How to Install and Configure the AWS Service Catalog Connector for ServiceNow. This post will help you gain familiarity with the AWS Service Catalog Connector application, and ensure you have it installed before proceeding.

The steps in this post work for both the 1.67 version and the 2.02 (latest) version of the connector.

ServiceNow Workflows

ServiceNow workflows automate multi‑step processes that occur between any combination of people and systems to help companies achieve better business outcomes. Workflows make the processes that drive your business more effective and efficient with end‑to‑end automation that removes the bottlenecks.

A few examples of workflows that can be built in ServiceNow to support AWS resources are:

Automatic Approval Workflows

Enterprises want to provide their data scientists with agile capabilities to automate the provision of AWS services without approvals. For example, as a data scientist, I want to self-provision Amazon EMR clusters in my development environment without approvals.

This workflow allows developers to get this kind of access without managerial approval, increasing the rate of development and innovation. Operational guardrails should be implemented to avoid abuse.

Approval Workflows

Enterprises want to provide their data scientists with agile capabilities to self-provision guarded AWS services with approvals. For example, as a manager, I want to approve services that my data scientists provision because of various reasons to meet enterprise change management, IT governance, and security compliance policies.

This workflow allows data scientists to request a service and have different levels of approvers confirm acceptance before the resource is provisioned.

To allow data scientists to automatically schedule when to provision and terminate Amazon EMR clusters, we designed and built ServiceNow workflows as defined in the next section.

Amazon EMR Cluster Workflows

The workflows designed to enable data scientists to schedule when to provision and terminate Amazon EMR clusters are defined by the following capabilities:

Schedule a time to provision an Amazon EMR cluster.
Schedule a time to automatically terminate an Amazon EMR cluster.
Schedules can be one-time or recurring. For example, as a user I can provision an EMR cluster that terminates at 5 p.m., or I can schedule a cluster that comes up every day at 8 a.m. and terminates at 5 p.m. for a specified period.

The image below shows the overall workflow diagram that summarizes the design. Click any image to enlarge.

Installation Steps

You need to have AWS Service Catalog Connector for ServiceNow installed and have at least an Amazon Elastic Compute Cloud (Amazon EC2) instance running.

This installation guide builds upon the basic setup detailed in the blog post titled How to Install and Configure the AWS Service Catalog Connector for ServiceNow.

Installation Steps as ServiceNow Admin

In your current ServiceNow instance (Kingston/London/Madrid), navigate to Retrieved Update Sets:

Download the XML files that will be used for installations from this GitHub folder.

In ServiceNow, under Retrieved Update Sets, install the first xml file: AWS Provisioning – Re-provisioning Merged Set.

Preview the installed update set.

Possible Error: For the errors shown below, please click on Accept Remote Update to ignore the error while previewing the update set.

Once Update Set has been previewed successfully, click Commit and confirm a successful commit.

Repeat these steps for the second XML file: AWS Provisioning – Re-provisioning Merged Set (Global).

Expected Error: For the error shown below, click on Skip Remote Update to ignore the error while previewing the update set.

After successfully installing both update sets, your view should like this:

Testing Steps as ServiceNow Admin

To test the new workflows, you have to first assign them to your product. In this post, we will use a simple Amazon EC2 instance to test. We use an Amazon EC2 instance to represent a simple EMR cluster. In practice, EMR clusters can take a long time to provision; hence using the Amazon EC2 instance for testing.

You need to have AWS Service Catalog Connector for ServiceNow installed and have at least an Amazon EC2 instance running or any product. This installation guide builds upon the basic setup detailed in the blog post titled How to Install and Configure the AWS Service Catalog Connector for ServiceNow.

Assign New Workflows to Your Amazon EC2 Instance

In ServiceNow, navigate to the AWS Service Catalog and click on a sample product (your product name could be different).

Select the magnifying glass next to workflow, and from the resulting list select the updated workflow. Next, update the record.

Test Your New Workflow

In ServiceNow, navigate to the AWS Service Catalog and click on a product you assigned the workflow to. In this example, it’s “EC2 Machines.”
.
Confirm you have ability to schedule provisioning and termination of this product.
.
Click on One-Time Termination. This is the easiest test, as it does not create a repeating schedule. Fill in your desired times; there is logic to limit time for business from 8 a.m. to 5 p.m.

Provision the product as you would normally. The difference is that the provision and termination is now going to automatically happen at your scheduled time.
.
After the time has passed, navigate to your provisioned product. You should see the product at the scheduled time provisioned. If you navigate to this window before the scheduled time, there would be no product yet.
.
In this example, I scheduled provisioning to occur at 8 a.m. on August 2, 2019, and termination at 9 a.m. on August 2, 2019. Once you click on the product, you can see all the details.

To follow along on how the workflows are progressing, you will need your requested item number.

A completed workflow, end-to-end, will be similar to the image below, with all the paths taken highlighted.

Wrapping Up the Installation

Once you have gone through all of the testing processes, you have successfully completed the installation.

Updates to a schedule are not possible in this proof of concept. If a new schedule for a similar product is needed, the user has to terminate the old schedule and create a new one.

This new capability can be extended to any product that benefits from scheduling. Amazon EMR clusters are one example of products you can schedule provisioning, since they take time to provision and you can utilize scheduled termination to reduce costs.

A special thanks to Linium, an APN Select Consulting Partner, who worked on the ServiceNow architecture. Linium is a ServiceNow Elite Partner and enables business solutions and processes for the platform. With a focus on architecting solutions for AWS services, Linium helps customers use their existing investment to manage their AWS environment more effectively and realize their goals for improved functionality and usability.

Summary

Enterprises can utilize ServiceNow workflows to empower data scientists with automatic workflows that can be used to schedule provisioning and termination of Amazon EMR clusters.

This post reviewed the various workflows that can be built on top of the AWS Service Catalog Connector for ServiceNow.

An Amazon EC2 instance was used to test this workflow, and the capability can be extended to any product that can benefit from scheduling. For example, you can schedule when to schedule provisioning and termination of an AWS account using an Account Vending Machine product in AWS Service Catalog.

You can also use it for simpler tasks, such as as scheduling termination of other AWS resources that a vendor may need for a short period of time.

Utilizing information shared in this post, enterprises can accelerate adoption of Amazon EMR clusters and AWS products by utilizing the AWS Service Catalog Connector for ServiceNow and leveraging the provided workflows.

ServiceNow – APN Partner Spotlight

ServiceNow is an APN Advanced Technology Partner. Their solution gives enterprises complete visibility and control of the entire IT environment, including virtualized and cloud infrastructure.

Contact ServiceNow | Solution Overview

*Already worked with ServiceNow? Rate this Partner

*To review an APN Partner, you must be an AWS customer that has worked with them directly on a project.

AWS Partner Network (APN) Blog