Enhancing configuration management at Verizon using AWS Systems Manager

In large enterprise organizations, it’s challenging to maintain standardization across environments. This is especially true if these environments are provisioned in a self-service manner—and even more so when new users access these provisioning services.

In this post, I describe how we at Verizon found a balance operating between agility, governance, and standardization for our AWS resources. I walk you through one of the solutions that we use to enable new users to provision AWS resources and configure application software. The solution uses ServiceNow and the following AWS services:

• AWS Systems Manager
• AWS Service Catalog
• AWS CloudFormation

Overview
Verizon seeks to provide a standardized AWS resource-provisioning service to new users. We needed a solution that incorporates auditing best practices and post-deployment configuration management to any newly provisioned environment. These best practices must work within a fully auditable self-service model and require that:

• All appropriate resource-provisioning service requests are life-cycle appropriate.
• The configuration management is defined and automatically applied as needed.

Solution
We wanted to provide a better user experience for our new users and help them provision resources in compliance with Verizon’s Governance and Security practices.

Shopping cart experience using AWS Service Catalog and ServiceNow
To accomplish these requirements, we use AWS Service Catalog to manage all our blueprint AWS CloudFormation templates (after being cleared through CFN-Nag). We then publish them as products in ServiceNow using the AWS Service Catalog Connector for ServiceNow (for example, EC2 CloudFormation as a product).

End users get a shopping cart-like experience whenever they provision resources in their account. This process helps us maintain provisioned resources consistent across all accounts and meet our compliance requirements.

The products or AWS CloudFormation templates are published to AWS Service Catalog using an automated Jenkins pipeline triggered from a Git repository, as shown in the following diagram.

All the products or AWS CloudFormation templates are retrieved from the AWS Service Catalog using the AWS Service Catalog Connector for ServiceNow and display as products. Users see the following list of compliant products from the Service Portal UI on ServiceNow.

When the user selects a product and provisions it in their account, ServiceNow makes backend calls to Verizon applications to do compliance checks. Then, it makes a call to AWS Service Catalog to provision the product. After the provisioning is successful, the user sees the list of provisioned products. The user can also use the API to provision the product.

Configuration management using Systems Manager
After the product is provisioned, users need the ability to configure their instances in a secure way using native AWS services. As shown earlier, a user uses the EC2 product and provisions it using the AWS Service Catalog. The user has an EC2 instance to configure his application.

At Verizon, we use Ansible for post-provisioning the configuration management of EC2 instances. After evaluating several options, we decided that Systems Manager was a perfect fit to use as an AWS native configuration-management solution. We leveraged Systems Manager agents already baked into our AMIs. For example, we use the Systems Manager Run Command with a run ansible document to execute Ansible playbooks and a run a shell-script document to run bash commands. For more information, see Running Ansible Playbooks using EC2 Systems Manager Run Command and State Manager.

The flow
In the previous provisioning section, you saw how users provision resources using AWS CloudFormation. ServiceNow maintains information on what types of resources users try to provision. For example, if there’s a product with an EC2 resource, you can enable the Systems Manager Run Command to deploy the EC2 product from the ServiceNow UI, as shown in the following screenshot.

When a user selects the Systems Manager Run Command, it allows users to include inline shell scripts or an Ansible Playbook. They can then submit the script as part of the configuration management, as shown in the following sample script:

---

- hosts: local

  tasks:

   - name: Install Nginx

     apt: pkg=nginx state=installed update_cache=true

     notify:

      - Start Nginx

  handlers:

   - name: Start Nginx

     service: name=nginx state=started

ServiceNow stores the information in its database for audit before it makes a Systems Manager API call to run the command on the selected EC2 instance. ServiceNow fetches the output using the command id from the previous command and shows it on the UI, as shown in the following screenshot.

We call this a post-provisioning workflow in ServiceNow, because it lets users do configuration actions after the provisioning is successful.

Additional configurations
Connector for ServiceNow version 2.0.2 includes additional operational actions, ServiceNow admin capability and ServiceNow Service Portal features. The Connector for ServiceNow documentation link includes the ServiceNow Additional Administrator Features details. Details include the ability to delete an AWS Service Catalog product in ServiceNow that does not have self-service actions associated. The ServiceNow Service Portal configurations also details the ability to order AWS Service Catalog products through the ServiceNow Service Portal using the Service Catalog and Order Something views.

Summary
This solution is just one of many ways that Verizon helps users provision Verizon-compliant resources and deploy their applications in the AWS Cloud. We want to empower new cloud users to provision resources faster, with fewer clicks, but also in a secure manner that follows audit and compliance requirements.

About the Author

Krishna Gadiraju (GK) is an architect for the Cloud Governance and Cloud User Experience product teams at Verizon. He actively assists development teams with the migration of on-premise applications to the cloud while ensuring that the Verizon AWS accounts meet all security and other compliances. GK has AWS DevOps Professional and GCP Associate certifications. He is an active presenter at cloud conferences and can be reached at https://www.linkedin.com/in/chaitanya-gk/.