Building a cloud CMDB on AWS for consistent resource configuration in hybrid environments

In this post, we will show you how you can design and implement a configuration management database (CMDB) strategy as part of your cloud operating model. We are going to review some common needs when building a CMDB and the relevant AWS services that will help you build a comprehensive solution. We will talk about where to track the CIs, which items to track, how to track them, and tagging strategies for simplified tracking.

The concept of a CMDB is not new. Enterprises have relied on CMDB solutions either as part of their ITSM suite or as a standalone product (COTS or custom Microsoft Excel based inventory) to support ITIL processes and change, configuration, and incident management. The CMDB data is represented as a point-in-time snapshot of the configuration item (CI) record covering the resource attributes, ownership, relationship to other Cis, and major lifecycle timestamps like creation, decommissioned dates, and so on. These attributes were updated per the periodic discovery cycle. At times, this could lead to stale inventory data because the CMDB update was not in real time. In many cases, this was an acceptable tradeoff.

With cloud CIs coming into play, the periodic discovery and update of CDMB is further challenged because of the considerations outlined below. As you go through these considerations, you will also find possible options to build an optimal CMDB strategy for your hybrid estate.

Where do you track the configuration items?

When integrating a cloud strategy into your CMDB design, you need to decide where to track the CIs.

1. Existing CMDB solution remains the authoritative source for the on-premises and cloud CIs. You can integrate cloud CI changes with the existing CMDB either in an event-driven or batch mode of flat file import at the predefined intervals.
2. Federated CMDB architecture where the existing CMDB still remains the authoritative source for the CIs but you also create a satellite cloud CMDB for all the cloud CI. This opens the possibility of a bi-directional synchronization between the existing CMDB and cloud CMDB and also delivers an entity isolation at the CMDB level.
3. Cloud service provider CMDB becomes the authoritative CMDB integrating both the on-premises and cloud CIs. To avoid any confusion with the semantics, the existing CMDB solution might very well be a cloud-hosted inventory itself but they were designed and built with a focus on the on-premises inventory and then extended to cover the cloud CIs too. On the other hand, a cloud service provider CMDB natively integrates with the cloud CIs and extends to cover the on-premises CIs.

For more information, see the ITIL Asset and Configuration Management in the Cloud whitepaper.

Each of these options has these considerations in common:

• Keep a single authoritative CMDB that integrates with your ticketing tool for the incident CI linkage. This is important because you want to have a single source of truth for all the business services and supporting technology resources.
• Understand what will be tracked in the CMDB and why. For example, you can choose not to populate the CMDB with details of a stateless instance or a transient cluster but instead track the Auto Scaling group as a cloud CI.

The discovery process varies depending on the CMDB solution you choose. If you need to run your own discovery solution and integrate with the existing CMDB, the recommended way is to gather AWS resource inventory by querying the AWS Config APIs. AWS Config makes tracking your resources easy by continuously monitoring and recording your AWS resource configurations and maintaining a history of configuration changes to those resources. AWS Config is central to your CMDB design. AWS Config also has the concept of a configuration item. I will refer to the AWS Config-supported resources as the AWS Config CI.

Depending on the maturity and landscape of your cloud adoption, one of the three options might meet your requirements. As enterprises set-out on the legacy modernization journey, the federated CMDB option not only seamlessly integrates with the existing ITIL processes, but also enables the analysis of the cloud inventory independent of the existing CMDB capabilities. I’ll share some examples.

Consider the ServiceNow CMDB as the existing CMDB and AWS Config as the cloud CMDB. The ServiceNow CMDB uses AWS Service Management Connector for ServiceNow to integrate with AWS Config. As a cloud CMDB, AWS Config provides a detailed view of the configuration of AWS resources, their relationship with other resources, and the historical configuration. Integration with the ServiceNow CMDB enables you to view the consolidated cloud resources using an AWS Config aggregator from the authoritative CMDB and offers cloud resource transparency across multiple AWS accounts. But there’s more; AWS Config goes beyond being just a plain resource inventory store and offers several possibilities in combination with the integrated AWS services.

1. You can use AWS Config rules to evaluate whether your AWS resources comply with the desired configuration and also remediate the non-compliant resources. For example, you can create a managed rule to determine whether your Amazon Elastic Compute Cloud (Amazon EC2) instances are using the specified AMIs. In addition to using AWS Config managed rules, you can also create a custom rule for your specific configuration evaluation requirements.
2. Because now you have a bi-directional sync with the authoritative CMDB through the Service Management Connector for ServiceNow, you can import existing CMDB CI records as custom resources in AWS Config. This enables you to write AWS Config rules for these custom resources to continuously monitor compliance.
3. AWS Config will provide information about the EC2 instance, including host, ENI, and security group, but what if you need to capture even more granular instance details (for example, application version, OS drivers, agents, packages, or Windows registry details)? Knowing which applications ran on which instances could be critical information for the application based view of the CMDB. For granular node-level visibility, you can use AWS Systems Manager Inventory. You can view the instance metadata history and track changes using AWS Config.
4. AWS Config also delivers a periodic snapshot of the latest resource configurations to your Amazon Simple Storage Service (Amazon S3) bucket. You can use an interactive query service like Amazon Athena to analyze data in the S3 bucket using standard SQL and gain customized views of your AWS resources (for example, IAM users who are using inline policies or instances missing an update). Although the method outlined in this blog post makes it possible to query the CI delivered to your S3 bucket, what if you want to get the current configuration state of your AWS resources? You can use the advanced query feature in AWS Config to run SQL type property-based queries and aggregations on the current AWS resource.
5. Last but not least is the ease of building a notification engine. You can use AWS Config to send configuration change notifications to an Amazon SNS Your security team can subscribe to the topic to get notified when the compliance state of your resources changes to noncompliant per the AWS Config rule evaluation.

Which configuration items to track

1. Traditionally, CMDB inventory covered resources like server hardware, virtual machines (with database and web server flags), and network equipment. With the AWS Cloud, you have additional resources like AWS Lambda functions, Amazon SQS standard queues, Amazon Kinesis data streams, etc. that can go beyond the semantics of the traditional CMDB. You must analyze which cloud CIs you want to federate from the cloud CMDB back to your authoritative CMDB. If you pay for your CMDB license on a per-CI basis, consider the impact of importing every AWS Config CI from hundreds of AWS accounts. AWS Config allows you to select the types of resources (EC2 instance, IAM user, and so on) that you want AWS Config to record. You create a configuration recorder and specify the desired resource types in the AWS Config recorder settings.
2. In rare cases, you might find that the AWS resource you want to reconcile with the authoritative CMDB is not tracked natively in AWS Config or it could be a self-managed Kubernetes cluster that you are running on EC2 instances. Just like the bi-directional sync use case, you can track AWS resources using the AWS Config custom resource feature. When you combine this feature with Amazon CloudWatch Events, AWS Lambda, and AWS Config rules as described in using AWS Config custom resources to track any resource on AWS, you can track, monitor, and achieve compliance even for resources not currently supported on AWS.
3. Filtering synchronization with the authoritative CMDB does not take away the fact that you still have all the AWS Config CIs being tracked and maintained in your AWS Config aggregator for all the accounts. What this means is that you can apply analytics on top of this aggregated AWS Config CI data across the AWS accounts to figure out how the use of AWS services has increased or decreased and then co-relate these findings to your application modernization, data strategy, and personnel training plans.
4. Dynamic nature of the cloud resource lifecycle (for example, Application Auto Scaling, Amazon EMR transient clusters, Amazon SageMaker training instances, short-lived Amazon ECS containers, and Amazon EKS pods). For such cases, tracking the guiding configuration template in the CMDB might be more meaningful than tracking the transient resources. For example, you can cover your security and compliance requirements better by tracking the Auto Scaling group configuration (AMI, key pair, security groups, user data, instance profile, tenancy) as a configuration item against the scaled-in and scaled-out nodes it triggered.
5. Licensing information. Generally, customers use specialized software asset management software that can use the CMDB discovery process to capture and populate the license details and drive the subsequent reconciliation with the license entitlement. On the AWS Cloud, you can use AWS License Manager to effectively manage your BYOL software licenses for Amazon EC2, Amazon RDS, and AWS Marketplace. You can also integrate License Manager API notifications with your existing ticketing tool to track license consumption.

How to track the configuration items

1. When populating the CI records in the existing CMDB, the cloud resources record format can be different compared to the existing CMDB schema. Cloud CI records for one cloud resource can also be different from the other cloud resource. Most leading CMDB discovery solutions integrate with AWS to consume cloud CIs, but check if you need to build a workflow around importing the AWS configuration dump and applying any transformation before it can be consumed by your CMDB service.
2. You can use the following services to define a desired state for the cloud resources and track the drift.
   1. AWS Config rules are a great way to define the desired state per AWS resource type and remediate the impacted resource.
   2. If you are using infrastructure as code (IaC) mode of deployment using AWS CloudFormation, you can use the drift detection feature to identify stack resources to which configuration changes have been made outside of CloudFormation management.
   3. You can use AWS Systems Manager State Manager to define and enforce a desired instance configuration for Amazon EC2. For example, you can use State Manager associations to specify that antivirus software must be installed and running on your instances or that certain ports must be closed.
   4. You can build a notification engine to alert any configuration drift from the desired state of the resource. You can use Amazon SNS to have multiple accounts publishing AWS Config noncompliant alerts to a central topic subscribed by your security team.

Tagging your configuration items

1. Another important mechanism of tracking the cloud CI is classifying and grouping the AWS resources based on tags such as cost center, security requirements, and workload. Tagging is an important step in enabling compliance because it allows you to group and bring transparency to the underlying cloud CIs (along with the interfacing cloud CIs) of a critical infrastructure. Some compliance authorities explicitly define this requirement as described in this article from BaFin’s 2017 annual report. After you have the visibility of the dependent components of the critical infrastructure, you can take special steps for the availability, reliability, and security of these components and track and report the aggregated critical infrastructure per audit requirements.
2. As a first step in your tagging strategy, implement a consistent tagging discipline across your multi-account AWS environment. Start by creating and applying the tag policies that contain the tagging rules (key-value pairs) at the organization level or the organizational unit (OU) level. After you have applied the rules, you should also ensure that your users are adhering to the prescribed controls. You can take a preventive or a reactive approach or combine both approaches as two levels of control:
   1. Preventive tagging control. You can use service control policies or IAM policies to prevent the IAM entities from creating certain resource types if the API request doesn’t include the specified tags.
   2. Reactive tagging control. You can use the required-tags AWS Config managed rule to evaluate resource compliance by checking if any one of a supplied list of tags is present on the resource. Start by configuring a remediation action on the resource with missing tags to be appropriately tagged or terminated. You can also view and download the tagged resource report from the Tag policies page in the Resource Groups console to check compliance.

It’s a best practice to define and apply a consistent tagging policy across accounts. Enforce this tagging policy using the preventive controls. You should also apply the reactive tagging control to check for the resources that don’t support tagging on launch or to validate tag value correctness.

Is your CMDB design well-architected?

The AWS Well-Architected Framework will help you evaluate and align your design choices with  cloud best practices. The five pillars of the framework are operational excellence, security, reliability, performance efficiency, and cost optimization.

AWS recently launched a new AWS Well-Architected Management and Governance Lens to help enterprise customers innovate faster and maintain control over the cost, compliance, and security. The Management and Governance Lens provides prescriptive guidance to customers for achieving visibility and operations at scale across the resources, data, and applications in a dynamic cloud environment.

Conclusion

Managing a reliable CMDB helps you to run, use, operate, and recover IT workloads to levels that meet the requirements of your regulators and business stakeholders. With AWS Config at the center of your cloud CMDB requirements, you can inventory your virtual IT assets, simplify compliance auditing, and improve your security posture.

About the authors