Run compliance enforcement and view compliant and non-compliant instances using AWS Systems Manager and PowerShell DSC

Point-and-click management randomizes servers with every single change made. It’s great to configure your servers with code. In addition, having automation to manage server configuration and compliance helps companies save time, improve availability, and lower the risks associated with up-time and security. However, the learning experience and administrative effort required to maintain such an environment can be challenging.

Today AWS Systems Manager publishes a new public document named AWS-ApplyDSCMofs. This document simplifies fleet configuration enforcement using a declarative approach based on PowerShell Desired State Configuration (DSC).

AWS Systems Manager is a management service that helps you automatically collect software inventory, apply OS patches, create system images, and configure Windows and Linux operating systems for both AWS and on-premises systems. These features help you define and track system configurations, avoid configuration drift, and maintain compliance of your Amazon EC2 and on-premises configurations.

PowerShell DSC is a declarative platform used for configuration, deployment, and management of Windows systems. If you’ve worked with configuration management tools like Chef or Puppet, you’ll notice that PowerShell DSC provides a familiar framework. PowerShell DSC allows administrators to use simple text documents called DSC to describe configurations that they want for a server. A DSC configuration is a specialized PowerShell script that states what to do, not how to do it. Executing the configuration produces a MOF file. The MOF file can be applied to one or more servers to achieve the desired configuration for those servers. DSC resources do the actual work of enforcing configuration. DSC comes with several built-in resources that manage ordinary server items, such as registry entries, services, and Windows features. You can learn more about built-in DSC resources here. There are public repositories of DSC resources such as PowerShell Gallery and GitHub PowerShell DSC Resources. You can also develop your own DSC resources if you have specialized needs. Related DSC resources are packaged into a PowerShell module.

In this blog post, we show you how you can remotely configure and maintain your Amazon EC2 Windows instances using PowerShell DSC’s declarative model. In addition, we show you how a Systems Manager for PowerShell DSC document makes it easier for you.

How it works

You can use the AWS-ApplyDSCMofs document with the AWS Systems Manager Run Command feature to apply one or more MOFs to one or more Amazon EC2 or managed instances. You can use the AWS-ApplyDSCMofs document with AWS Systems Manager State Manager to periodically apply one or MOFs to one or more EC2 or managed instances based on a schedule you specify.  Using State Manager ensures that the desired configuration for your servers is enforced on a regular basis, even if someone alters the configuration manually.  You can create new MOFs and resources, or use your existing ones.

Let’s look at how the AWS-ApplyDSCMofs document works. The following diagram illustrates using Run Command or State Manager to apply one or more MOFs to one or more instances.

1. The AWS-ApplyDSCMofs document is executed by Run Command or State Manager with arguments you specify on the instances you specify.
2. The specified MOFs are downloaded from Amazon S3, file system shares, or via HTTP request. The required resource modules are determined for each instance.
3. The required resource modules are loaded from PowerShell Gallery (if permitted) or Amazon S3 for each instance.
4. Passwords for required credentials are fetched from either Systems Manager Parameter Store or Secrets Manager and then credentials are resolved for each instance.
5. The specified MOFs are applied on each instance.
6. Compliance is verified on each instance.
7. Compliance reports are uploaded to Amazon S3.
8. Compliance is reported to Systems Manager Compliance for each instance.

Document parameters

You can view the AWS-ApplyDSCMofs document in the AWS Systems Manager console, on the Documents page:

The document’s parameters will let you provide all the values that are necessary to execute the configuration end-to-end. These parameters include:

• MofsToApply – A comma separated list of MOF files to apply. Files can be from S3 (e.g., s3:bucketName:fileKey), HTTP/S (e.g., https://mydomain.com/MyMof.mof) or File System (e.g., \\MyServer\MyShare\MyMof.mof).
• MofOperationMode – Indicates the action to be taken on noncompliant resources:
  • ReportOnly mode – Reports noncompliance but will not take any corrective action.
  • Apply mode – Attempt to bring the noncompliant resource into compliance.
• ServicePath – The prefix used when writing compliance and status reports to Amazon S3.
• ReportBucketName – The name of the S3 bucket to write the detailed compliance reports to.
• StatusBucketName – The name of the S3 bucket to write the instance compliance summary information to.
• ProxyUri – (Optional) The Uri of a proxy server to use when accessing external resources (S3, PSGallery etc.).
• ModuleSourceBucketName – (Optional) The name of the S3 bucket containing module files. If this is not specified, the ‘AllowPSGalleryModuleSource’ parameter should be ‘True’.
• AllowPSGalleryModuleSource – (Optional) True if modules can be downloaded from PSGallery. If this is false, the ModuleSourceBucketName parameter should be used. The default is ‘True’.
• RebootBehavior – (Optional) The reboot behavior that the script will have should it need to reboot the machine. Default is ‘AfterMof’.
• UseComputerNameForReporting – (Optional) Use the ComputerName for reporting instead of the Instance ID. Default is ‘False’ (that is, instance ID will be used).
• EnableVerboseLogging – (Optional) Enables verbose logging, which is useful when deploying MOFs for the first time. Make sure you have lifecycle policies enabled on your logging bucket if you use this all the time.
• EnableDebugLogging – (Optional) Enables debug logging, which is useful when troubleshooting a failing MOF. Don’t use in day-to-day operation.
• ComplianceType – (Optional) The compliance type to use when reporting compliance. Default is Custom:DSC. You can create multiple associations to a single target, but each association should have a different ComplianceType value to ensure that the compliance information for each association is not overwritten.
• PreRebootScript – (Optional) A script to execute prior to the instance rebooting, if the configuration has indicated that a reboot is necessary. Your script must be a single line, but can be separated by semicolons.

For the ReportBucketName, StatusBucketName, and ModuleSourceBucketName parameters you can optionally prefix the bucket names with the Region that the bucket is located in (e.g., us-west-2:mymofbucket). If you are using a proxy with whitelisted URLs/IP addresses for S3 endpoints in a specific Region that does not include us-east-1, you need to prefix the bucket names with the Region. If the bucket name is not prefixed, it will automatically discover the bucket Region using the us-east-1 endpoint.

How about MOF files?

AWS Systems Manager for PowerShell DSC supports precompiled MOF files as inputs to Systems Manager.

MOFs can be stored in three different places:

• S3 – E.g., s3:bucketName:fileKey or s3:bucketRegion:bucketName:fileKey
• HTTP/HTTPS – E.g., https://mydomain.com/Mofs/MyMof.mof
• File System – E.g., \\MyFileServer\Mofs\MyMof.mof or C:\Mofs\MyMof.mof

If Amazon S3 or HTTP/HTTPS are used as the source of MOFs, Systems Manager for PowerShell DSC will only download new or changed version of these MOFs. So repeated invocations will only download the MOFs when necessary. This reduces the costs of executing this document. In order for this to work in Amazon S3, encryption must be disabled on the bucket.

Key benefits

• PowerShell DSC Configuration can leverage AWS Systems Manager for cloud scale management.
• Dynamic Token Substitution allows you to insert “tokens” into your configurations that are resolved at runtime, from a variety of sources including environment variables, managed instance tags, Parameter Store entries, or combinations (check Tags first, then check Parameter Store). This feature allows you to reuse common MOFs that have instance-specific values, which can substantially reduce the number of MOFs that you need to manage (for example, one per service instead of one per instance).
• Integration with Systems Manager Compliance – a queryable “compliant/notcompliant” answer, as well as “what resource failed/is not compliant”. You can also enable detailed JSON-formatted reports to be stored in Amazon S3, including all verbose messages that were output by the resources. The output logging from the document itself can be standard, verbose, or debug level, which will give you all the information required to troubleshoot the PowerShell DSC application. You can see example reports in the Walkthrough coming up next.
• PowerShell DSC integration doesn’t rely on a DSC compliance server. MOF files can be stored in S3, an HTTP/HTTPS endpoint, an SMB share, or a local file system.
• Ability to run a “Prereboot script”, so if any DSC resources require a reboot, the script will be executed first. This is useful for server farms where you might want to remove an instance from a load balancer, or you might want to evacuate cluster resources before rebooting.

Walk through

In this example we use a Systems Manager for PowerShell DSC document to install and configure our IIS server-based web farm hosted in EC2.

The example MOF enforces the following configurations:

• IIS server and required components (ASP.NET) are installed.
• The default website is stopped.
• A new website is created under the specified directory.
• IIS logging is enabled in a custom folder.
• Required registry keys are configured based on the Systems Manager parameter store values.
• Required ports are configured in Windows Firewall for remote management.

Requirements

There are a few requirements for the configuration:

Platform requirements: On target instances:

• PowerShell v5.0 or greater.
• SSM Agent v2.2 or greater.
• AWSPowerShell module 3.3.261.0 or greater.
  • If AWSPowerShell is not present or out of date this module will install or upgrade from PSGallery.

IAM role requirements: As a basic requirement, target instances should have AmazonEC2RoleForSSM IAM managed policy. Also, in order to use SecretsManager and Tags, you will also need a policy similar to the one that follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SSMPermissionsForTagAccess",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ssm:ListTagsForResource"
            ],
            "Resource": "*"
        },
    ]
}

Configuration

To run AWS-ApplyDSCMofs document MOFs must be in one of the supported locations. In our example, we are going to use a S3 bucket.

For this demo, we created the following configuration file, which enforces the desired configuration on the target instances.

Configuration EC2DSCIISConfiguration
{ 
  Import-DscResource -Module xWebAdministration
  Import-DscResource -ModuleName xNetworking

  node ("localhost")
	{ 		
		$LogPath = "{tagssm:LogPath}"
        $WebSiteDestinationPath = "{tagssm:WebSiteDestinationPath}"
       	$WebSiteName = "{tagssm:WebSiteName}"

		WindowsFeature InstallWebServer 
		{ 
			Ensure          = "Present"
			Name            = "Web-Server" 
		} 		
		WindowsFeature ASP
		{
			Ensure          = "Present"
			Name            = "Web-Asp-Net45"
            DependsOn       = '[WindowsFeature]InstallWebServer'
		}
        xWebsite DefaultSite
        {
            Ensure          = "Present"
            Name            = "Default Web Site"
            State           = "Stopped"
            PhysicalPath    = "C:\inetpub\wwwroot" 
            DependsOn       = '[WindowsFeature]ASP'         
        }
        File CreateWebsiteFolder 
        {
            Type            = 'Directory'
            DestinationPath = $WebSiteDestinationPath
            Ensure          = "Present"
            DependsOn       = '[xWebsite]DefaultSite'
        }
        xWebsite NewWebsite
        {
            Ensure          = "Present"
            Name            = $WebSiteName
            State           = "Started"
            PhysicalPath    = $WebSiteDestinationPath
            BindingInfo     = @(
            MSFT_xWebBindingInformation
                {
                    Protocol              = "HTTP"
                    Port                  = 80
                })
            DependsOn       = '[File]CreateWebsiteFolder'
         }		
		xFirewall AllowManagementPort {
            Name            = "RDP xPort"
            DisplayName     = "RDP xPort"
            Ensure          = "Present"
            Protocol        = "TCP"
            Enabled         = "True"
            Direction       = "InBound"
            LocalPort       = 3389
		}
		 xIISLogging Logging
        {
            LogPath              = $LogPath 
            Logflags             = @('Date','Time','ClientIP','UserName','ServerIP')
            LoglocalTimeRollover = $True
            LogTruncateSize      = '2097152'
            LogFormat            = 'W3C'
            DependsOn            = '[xWebsite]NewWebsite'
        }
}} 
EC2DSCIISConfiguration

The configuration file dynamically obtains the following values from AWS Systems Manager Parameter Store.

$LogPath = A string value to configure a custom path for IIS server logging. In our example that is “C:\IISLogs”.

$WebSiteDestinationPath = A string value to configure the physical path for the new IIS website. In our example that is “C:\Reporting”.

$WebSiteName = A string value to configure the name for the new IIS website. In our example that is “ReportingWebSite”.

Before applying a DSC configuration to the target instances, it needs to be compiled into a MOF file. To do this, you can simply run your configuration file just like a PowerShell function.

Now that we have the MOF file, all we need is to upload this file to an S3 bucket in the same AWS Region and get the URL.

Since we will be using some dynamic tokens for the PowerShell DSC configuration, we need to add those values in SSM Parameter Store. Our target instances will retrieve those dynamic parameters each time they run the PowerShell DSC configuration.

You can either use the Systems Manager Run Command feature to enforce MOF file(s) manually or use the System Manager State Manager to create associations targeting various instances. State Manager will then ensure the desired state configuration is executed on the specified schedule. In our example, we will be using State Manager Associations.

In the Systems Manager console, choose State Manager, then choose Create Association, and provide a name. On the Command Document page, select AWS-ApplyDSCMofs, and configure related parameters as follows:

Mofs To Apply: Link to the MOF file stored in S3 bucket.

Report Bucket Name: The name of the S3 bucket to write compliance reports to.

Allow PS Gallery Module Source: We will keep default “True” value as we have some external modules to be used in the MOF file.

Enable Verbose Logging/Enable Debug Logging: We will also enable those values just to see how detailed output we can have after the execution.

Leave all other parameters as default and select your target instances. In our environment, we have a specific tag configured for all our IIS instances, so that querying this tag to select target instances is an easier choice.

As the last step, under Specify Schedule select a schedule type to run your configuration file across your target instances. In our case, we want to run the schedule every 2 hours to make sure that all our IIS EC2 instances have the right configuration and setup.

Choose Run Association.

If you choose your newly created association, under the Instances tab you can see initial compliance details. Now every 2 hours, our IISConf association runs across all the EC2 instances that have the ServiceType:IIS tag configured, and it makes sure that all the configurations are applied.

On the Insights page, choose Compliance, and you can see compliance reports:

If we connect to our instances, we see that all the required configurations have been applied as well.

Querying status logs with Amazon Athena

AWS Systems Manager stores logs generated in Amazon S3. The PowerShell DSC document is not an exception. In fact the AWS-ApplyDSCMofs document carries this functionality one step further and also stores status and reporting logs generated by PowerShell DSC in Amazon S3. This feature is particularly useful for aggregating compliance information from instances in multiple AWS Accounts and Regions, giving you a global compliance view. These logs are created in JSON format.

Amazon Athena is an interactive query service that makes it easy to analyze data directly in Amazon S3 using standard SQL. You can access Athena by using the AWS Management Console, through a JDBC connection, using the Athena API, or using the Athena CLI.

Amazon Athena has an interactive query editor on AWS Management Console. In this example we will use this query editor to query the DSC status logs created by the Systems Manager for PowerShell DSC document.

Step 1:  Create a database

Open the Amazon Athena console. In the Athena Query Editor, you see a query pane with an example query. Start typing your query anywhere in the query pane. To create a new database for our PowerShell DSC logs, enter the following CREATE DATABASE statement, and then choose Run Query:

## SQL
CREATE DATABASE dsc

Step 2: Create a table

Now that you have a database, you’re ready to create a table that’s based on the status data files generated by PowerShell DSC execution. You define columns that map to the data, specify how the data is delimited, and provide the location in Amazon S3 for the file.

In the query pane, enter the following CREATE TABLE statement, and then choose Run Query: Populate the highlighted area that follows with your DSC status logs bucket name before you run the statement.

## SQL
CREATE EXTERNAL TABLE IF NOT EXISTS dsc.status (
`ServicePath` string,
`InstanceId` string,
`ComputerName` string,
`AppliedMofs` string,
`Status` string,
`Compliant` string,
`NotCompliant` string,
`SetCount` string,
`LastRunId` string,
`LastRunUTC` string
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = '1'
) LOCATION 's3://<your DSC status bucket name>/default/'
TBLPROPERTIES ('has_encrypted_data'='false');

The table named status is created and appears in the Catalog dashboard for your database.

Step 3: Query data

Choose New Query, enter the following statement anywhere in the query pane, and then choose Run Query:

## SQL
SELECT computername, instanceid, appliedmofs from dsc.status WHERE status = 'Compliant'

Results are returned that look like the following:

Conclusion

By using the PowerShell DSC document in Systems Manager, you can focus on expressing what configuration you want rather than what changes are needed to get the configuration you want. The PowerShell DSC document is integrated with Parameter Store, so your security posture is enhanced because sensitive data is not embedded in the configuration. It is stored securely at all times.  Parameter Store integration also allows much greater reuse of the same MOF files as operational configurations change due to changing business requirements.

AWS Systems Manager provides a suite of services that help you manage cloud-based, on-premises-based, and hybrid Windows workloads with high reliability, availability and security at scale.

About the Authors

Anil Erduran is a Microsoft Partner Solutions Architect at Amazon Web Services based in London. He helps AWS partners in EMEA architect Microsoft solutions on AWS. Anil loves working with customers and he represents AWS at conferences and other events. You can find him on Twitter at @anil_erduran

Baris Furtinalar is a Senior Solutions Architect at AWS in Luxembourg. He is passionate about cloud computing and he believes this shift to the cloud helps companies transform their businesses, gain agility, and improve operational responsiveness. In his spare time, he loves spending time with his daughter, speeding on German Autobahns, and enjoying healthy food