Category: EC2 Systems Manager
Amazon EC2 Systems Manager Patch Manager now supports Linux
Hot on the heels of some other great Amazon EC2 Systems Manager (SSM) updates is another vital enhancement: the ability to use Patch Manager on Linux instances!
We launched Patch Manager with SSM at re:Invent in 2016 and Linux support was a commonly requested feature. Starting today we can support patch manager in:
- Amazon Linux 2014.03 and later (2015.03 and later for 64-bit)
- Ubuntu Server 16.04 LTS, 14.04 LTS, and 12.04 LTS
- RHEL 6.5 and later (7.x and later for 64-Bit)
When I think about patching a big group of heterogenous systems I get a little anxious. Years ago, I administered my school’s computer lab. This involved a modest group of machines running a small number of VMs with an immodest number of distinct Linux distros. When there was a critical security patch it was a lot of work to remember the constraints of each system. I remember having to switch back and forth between arcane invocations of various package managers – pinning and unpinning packages: sudo yum update -y, rpm -Uvh ..., apt-get, or even emerge (one of our professors loved Gentoo).
Even now, when I use configuration management systems like Chef or Puppet I still have to specify the package manager and remember a portion of the invocation – and I don’t always want to roll out a patch without some manual approval process. Based on these experiences I decided it was time for me to update my skillset and learn to use Patch Manager.
Patch Manager is a fully-managed service (provided at no additional cost) that helps you simplify your operating system patching process, including defining the patches you want to approve for deployment, the method of patch deployment, the timing for patch roll-outs, and determining patch compliance status across your entire fleet of instances. It’s extremely configurable with some sensible defaults and helps you easily deal with patching hetergenous clusters.
Since I’m not running that school computer lab anymore my fleet is a bit smaller these days:
As you can see above I only have a few instances in this region but if you look at the launch times they range from 2014 to a few minutes ago. I’d be willing to bet I’ve missed a patch or two somewhere (luckily most of these have strict security groups). To get started I installed the SSM agent on all of my machines by following the documentation here. I also made sure I had the appropriate role and IAM profile attached to the instances to talk to SSM – I just used this managed policy: AmazonEC2RoleforSSM.
Now I need to define a Patch Baseline. We’ll make security updates critical and all other updates informational and subject to my approval.
Next, I can run the AWS-RunPatchBaseline SSM Run Command in “Scan” mode to generate my patch baseline data.
Then, we can go to the Patch Compliance page in the EC2 console and check out how I’m doing.
Yikes, looks like I need some security updates! Now, I can use Maintenance Windows, Run Command, or State Manager in SSM to actually manage this patching process. One thing to note, when patching is completed, your machine reboots – so managing that roll out with Maintenance Windows or State Manager is a best practice. If I had a larger set of instances I could group them by creating a tag named “Patch Group”.
For now, I’ll just use the same AWS-RunPatchBaseline Run Command command from above with the “Install” operation to update these machines.
As always, the CLIs and APIs have been updated to support these new options. The documentation is here. I hope you’re all able to spend less time patching and more time coding!
– Randall
Catching Up On AWS Announcements from Early 2017
Even though we have published 123 posts so far this year, we simply don’t have the time to cover every significant AWS launch. Also, the newer services are often richer and take more space to describe, adding to our workload. This post (and others to follow each quarter) will outline some of the launches that we did not have time to address earlier.
So, here we go:
- Migration Support for NoSQL Databases
- Comments, Tagging, and Metadata APIs for WorkDocs.
- Email and SMS Integration for Pinpoint
- Usage Type Groups and Linked Account Access for AWS Budgets
- EC2 Systems Manager Support for Hierarchies, Tagging, and CloudWatch Events
These features have already launched and you may already be using them!
Migration Support for NoSQL Databases
With this launch, AWS Database Migration Service can migrate relational databases, NoSQL databases, and data warehouses. The launch adds support for MongoDB databases as a migration source and Amazon DynamoDB tables as a migration target. To get started, create a replication instance and database endpoints for MongoDB and DynamoDB:
Read MongoDB as a Migration Source and DynamoDB as a Migration Target for more information.
Comments, Tagging, and Metadata APIs for WorkDocs
This addition to the Amazon WorkDocs Administrative SDK provides APIs for creating and accessing metadata, tags, and comments:
Metadata – CreateCustomMetadata, DeleteCustomMetadata.
Tags – CreateLabels, DeleteLabels.
Comments – CreateComment, DeleteComment, DescribeComments.
The SDK is available for Java, Python, Go, JavaScript, .NET, PHP, and Ruby. It handles signing of API requests using Sigv4, and is integrated with IAM (roles and permissions), SNS (real-time notifications), and CloudTrail (monitoring).
Email and SMS Integration for Pinpoint
In addition to the existing Mobile Push Notifications, Amazon Pinpoint can now drive user engagement through email and SMS notifications. In order to use this feature you must first enable the desired channel or channels:
To learn more, read about Amazon Pinpoint Channels.
Usage Type Groups and Linked Account Access for AWS Budgets
AWS Budgets let you set cost and usage budgets and receive notification if they are breached (read Managing Your Costs with Budgets and AWS Budgets Update – Track Cloud Costs and Usage).
In order to make AWS Budgets even more useful, we added support for linked accounts and a new usage type filtering option. Organizations that make use of Consolidated Billing to consolidate payment for multiple AWS accounts will benefit from the support for linked accounts. The member accounts can now access their own budgets, while the payer account remains responsible for payment.
The usage type and usage type group filtering dimensions allow you to track your costs and usage from an aggregate level all the way down to the most basic unit of metering. For example, you can create a budget to track all EC2 usage (EC2-Running Hours):
Or a specific usage type, in this case three different sizes of T2 instances:
EC2 Systems Manager Support for Hierarchies, Tagging, and CloudWatch Events
This management service helps you to automatically collect software inventory, apply OS patches, create system images, and configure both Linux and Windows operating systems.
The Parameter Store (one of the service’s most popular features) stores configuration data such as database access strings and passwords in encrypted form. It is accessible from the CLI, APIs, and SDKs; this allows AWS Lambda functions and code running inside of Amazon ECS containers to access the same parameters.
We added support for storage of parameters in hierarchical form, giving you the ability to group them by organization, application, and so forth. You can also create parallel sets of parameters for use in development, testing, and production environments. To create a hierarchy of parameters, use names that include one or more “/” characters:
We also added support for tagging. You can query parameters based on tags and you can add IAM permissions to parameters via tags.
Finally, the Parameter Store is now a source of CloudWatch Events. You can now track changes to your parameters, perhaps making sure that they are not inadvertently changed in a way that could break an existing application:
Keeping Up
In addition to reading this blog on a regular basis, you can also follow me and AWS Cloud on Twitter. You can also check out the AWS What’s New and subscribe to the RSS Feed.
— Jeff;
Manage Instances at Scale without SSH Access Using EC2 Run Command
The guest post below, written by Ananth Vaidyanathan (Senior Product Manager for EC2 Systems Manager) and Rich Urmston (Senior Director of Cloud Architecture at Pegasystems) shows you how to use EC2 Run Command to manage a large collection of EC2 instances without having to resort to SSH.
— Jeff;
Enterprises often have several managed environments and thousands of Amazon EC2 instances. It’s important to manage systems securely, without the headaches of Secure Shell (SSH). Run Command, part of Amazon EC2 Systems Manager, allows you to run remote commands on instances (or groups of instances using tags) in a controlled and auditable manner. It’s been a nice added productivity boost for Pega Cloud operations, which rely daily on Run Command services.
You can control Run Command access through standard IAM roles and policies, define documents to take input parameters, control the S3 bucket used to return command output. You can also share your documents with other AWS accounts, or with the public. All in all, Run Command provides a nice set of remote management features.
Better than SSH
Here’s why Run Command is a better option than SSH and why Pegasystems has adopted it as their primary remote management tool:
Run Command Takes Less Time – Securely connecting to an instance requires a few steps e.g. jumpboxes to connect to or IP addresses to whitelist etc. With Run Command, cloud ops engineers can invoke commands directly from their laptop, and never have to find keys or even instance IDs. Instead, system security relies on AWS auth, IAM roles and policies.
Run Command Operations are Fully Audited – With SSH, there is no real control over what they can do, nor is there an audit trail. With Run Command, every invoked operation is audited in CloudTrail, including information on the invoking user, instances on which command was run, parameters, and operation status. You have full control and ability to restrict what functions engineers can perform on a system.
Run Command has no SSH keys to Manage – Run Command leverages standard AWS credentials, API keys, and IAM policies. Through integration with a corporate auth system, engineers can interact with systems based on their corporate credentials and identity.
Run Command can Manage Multiple Systems at the Same Time – Simple tasks such as looking at the status of a Linux service or retrieving a log file across a fleet of managed instances is cumbersome using SSH. Run Command allows you to specify a list of instances by IDs or tags, and invokes your command, in parallel, across the specified fleet. This provides great leverage when troubleshooting or managing more than the smallest Pega clusters.
Run Command Makes Automating Complex Tasks Easier – Standardizing operational tasks requires detailed procedure documents or scripts describing the exact commands. Managing or deploying these scripts across the fleet is cumbersome. Run Command documents provide an easy way to encapsulate complex functions, and handle document management and access controls. When combined with AWS Lambda, documents provide a powerful automation platform to handle any complex task.
Example – Restarting a Docker Container
Here is an example of a simple document used to restart a Docker container. It takes one parameter; the name of the Docker container to restart. It uses the AWS-RunShellScript method to invoke the command. The output is collected automatically by the service and returned to the caller. For an example of the latest document schema, see Creating Systems Manager Documents.
Putting Run Command into practice at Pegasystems
The Pegasystems provisioning system sits on AWS CloudFormation, which is used to deploy and update Pega Cloud resources. Layered on top of it is the Pega Provisioning Engine, a serverless, Lambda-based service that manages a library of CloudFormation templates and Ansible playbooks.
A Configuration Management Database (CMDB) tracks all the configurations details and history of every deployment and update, and lays out its data using a hierarchical directory naming convention. The following diagram shows how the various systems are integrated:
For cloud system management, Pega operations uses a command line version called cuttysh and a graphical version based on the Pega 7 platform, called the Pega Operations Portal. Both tools allow you to browse the CMDB of deployed environments, view configuration settings, and interact with deployed EC2 instances through Run Command.
CLI Walkthrough
Here is a CLI walkthrough for looking into a customer deployment and interacting with instances using Run Command.
Launching the cuttysh tool brings you to the root of the CMDB and a list of the provisioned customers:
You interact with the CMDB using standard Linux shell commands, such as cd, ls, cat, and grep. Items prefixed with s are services that have viewable properties. Items prefixed with d are navigable subdirectories in the CMDB hierarchy.
In this example, change directories into customer CUSTB’s portion of the CMDB hierarchy, and then further into a provisioned Pega environment called env1, under the Dev network. The tool displays the artifacts that are provisioned for that environment. These entries map to provisioned CloudFormation templates.
The ls –l command shows the version of the provisioned resources. These version numbers map back to source control–managed artifacts for the CloudFormation, Ansible, and other components that compose a version of the Pega Cloud.
Now, use Run Command to interact with the deployed environments. To do this, use the attach command and specify the service with which to interact. In the following example, you attach to the Pega Web Tier. Using the information in the CMDB and instance tags, the CLI finds the corresponding EC2 instances and displays some basic information about them. This deployment has three instances.
From here, you can use the run command to invoke Run Command documents. In the following example, you run the docker-ps document against instance 0 (the first one on the list). EC2 executes the command and returns the output to the CLI, which in turn shows it.
Using the same command and some of the other documents that have been defined, you can restart a Docker container or even pull back the contents of a file to your local system. When you get a file, Run Command also leaves a copy in an S3 bucket in case you want to pass the link along to a colleague.
Now, leverage the Run Command ability to do more than one thing at a time. In the following example, you attach to a deployment with three running instances and want to see the uptime for each instance. Using the par (parallel) option for run, the CLI tells Run Command to execute the uptime document on all instances in parallel.
Summary
Run Command improves productivity by giving you faster access to systems and the ability to run operations across a group of instances. Pega Cloud operations has integrated Run Command with other operational tools to provide a clean and secure method for managing systems. This greatly improves operational efficiency, and gives greater control over who can do what in managed deployments. The Pega continual improvement process regularly assesses why operators need access, and turns those operations into new Run Command documents to be added to the library. In fact, their long-term goal is to stop deploying cloud systems with SSH enabled.
If you have any questions or suggestions, please leave a comment for us!
— Ananth and Rich
EC2 Run Command is Now a CloudWatch Events Target
Ok, time for another peanut butter and chocolate post! Let’s combine EC2 Run Command (New EC2 Run Command – Remote Instance Management at Scale) and CloudWatch Events (New CloudWatch Events – Track and Respond to Changes to Your AWS Resources) and see what we get.
EC2 Run Command is part of EC2 Systems Manager. It allows you to operate on collections of EC2 instances and on-premises servers reliably and at scale, in a controlled and selective fashion. You can run scripts, install software, collect metrics and log files, manage patches, and much more, on both Windows and Linux.
CloudWatch Events gives you the ability to track changes to AWS resources in near real-time. You get a stream of system events that you can easily route to one or more targets including AWS Lambda functions, Amazon Kinesis streams, Amazon SNS topics, and built-in EC2 and EBS targets.
Better Together
Today we are bringing these two services together. You can now create CloudWatch Events rules that use EC2 Run Command to perform actions on EC2 instances or on-premises servers. This opens the door to all sorts of interesting ideas; here are a few that I came up with:
Final Log Collection – Collect application or system logs from instances that are being shut down (either manually or as a result of a scale-in operation initiated by Auto Scaling).
Error Log Condition – Collect logs after an application crash or a security incident.
Instance Setup – After an instance has started, download & install applications, set parameters and configurations, and launch processes.
Configuration Updates – When a config file is changed in S3, install it on applicable instances (perhaps determined by tags). For example, you could install an updated Apache web server config file on a set of properly tagged instances, and then restart the server so that it picks up the changes. Or, update an instance-level firewall each time the AWS IP Address Ranges are updated.
EBS Snapshot Testing and Tracking – After a fresh snapshot has been created, mount it on a test instance, check the filesystem for errors, and then index the files in the snapshot.
Instance Coordination – Every time an instance is launched or terminated, inform the others so that they can update internal tracking information or rebalance their workloads.
I’m sure that you have some more interesting ideas; please feel free to share them in the comments.
Time for Action!
Let’s set this up. Suppose I want to run a specific Linux shell command every time Auto Scaling adds another instance to an Auto Scaling Group.
I start by opening the CloudWatch Events Console and clicking on Create rule:
I configure my Event Source to be my Auto Scaling Group (AS-Main-1), and indicate that I want to take action when EC2 instances are launched successfully:
Then I set up the target. I choose SSM Run Command, pick the AWS-RunShellScript document, and indicate that I want the command to be run on the instances that are tagged as coming from my Auto Scaling group:
Then I click on Configure details, give my rule a name and a description, and click on Create rule:
With everything set up, the command service httpd start will be run on each instance launched as a result of a scale out operation.
Available Now
This new feature is available now and you can start using it today.
— Jeff;
EC2 Systems Manager – Configure & Manage EC2 and On-Premises Systems
Last year I introduced you to the EC2 Run Command and showed you how to use it to do remote instance management at scale, first for EC2 instances and then in hybrid and cross-cloud environments. Along the way we added support for Linux instances, making EC2 Run Command a widely applicable and incredibly useful administration tool.
Welcome to the Family
Werner announced the EC2 Systems Manager at AWS re:Invent and I’m finally getting around to telling you about it!
This a new management service that include an enhanced version of EC2 Run Command along with eight other equally useful functions. Like EC2 Run Command it supports hybrid and cross-cloud environments composed of instances and services running Windows and Linux. You simply open up the AWS Management Console, select the instances that you want to manage, and define the tasks that you want to perform (API and CLI access is also available).
Here’s an overview of the improvements and new features:
Run Command – Now allows you to control the velocity of command executions, and to stop issuing commands if the error rate grows too high.
State Manager – Maintains a defined system configuration via policies that are applied at regular intervals.
Parameter Store – Provides centralized (and optionally encrypted) storage for license keys, passwords, user lists, and other values.
Maintenance Window -Specify a time window for installation of updates and other system maintenance.
Software Inventory – Gathers a detailed software and configuration inventory (with user-defined additions) from each instance.
AWS Config Integration – In conjunction with the new software inventory feature, AWS Config can record software inventory changes to your instances.
Patch Management – Simplify and automate the patching process for your instances.
Automation – Simplify AMI building and other recurring AMI-related tasks.
Let’s take a look at each one…
Run Command Improvements
You can now control the number of concurrent command executions. This can be useful in situations where the command references a shared, limited resource such as an internal update or patch server and you want to avoid overloading it with too many requests.
This feature is currently accessible from the CLI and from the API. Here’s a CLI example that limits the number of concurrent executions to 2:
$ aws ssm send-command \
--instance-ids "i-023c301591e6651ea" "i-03cf0fc05ec82a30b" "i-09e4ed09e540caca0" "i-0f6d1fe27dc064099" \
--document-name "AWS-RunShellScript" \
--comment "Run a shell script or specify the commands to run." \
--parameters commands="date" \
--timeout-seconds 600 --output-s3-bucket-name "jbarr-data" \
--region us-east-1 --max-concurrency 2
Here’s a more interesting variant that is driven by tags and tag values by specifying --targets instead of --instance-ids:
$ aws ssm send-command \
--targets "Key=tag:Mode,Values=Production" ... You can also stop issuing commands if they are returning errors, with the option to specify either a maximum number of errors or a failure rate:
$ aws ssm send-command --max-errors 5 ...
$ aws ssm send-command --max-errors 5% ...
State Manager
State Manager helps to keep your instances in a defined state, as defined by a document. You create the document, associate it with a set of target instances, and then create an association to specify when and how often the document should be applied. Here’s a document that updates the message of the day file:
And here’s the association (this one uses tags so that it applies to current instances and to others that are launched later and are tagged in the same way):
Specifying targets using tags makes the association future-proof, and allows it to work as expected in dynamic, auto-scaled environments. I can see all of my associations, and I can run the new one by selecting it and clicking on Apply Association Now:
Parameter Store
This feature simplifies storage and management for license keys, passwords, and other data that you want to distribute to your instances. Each parameter has a type (string, string list, or secure string), and can be stored in encrypted form. Here’s how I create a parameter:
And here’s how I reference the parameter in a command:
Maintenance Window
This feature allows specification of a time window for installation of updates and other system maintenance. Here’s how I create a weekly time window that opens for four hours every Saturday:
After I create the window I need to assign a set of instances to it. I can do this by instance Id or by tag:
And then I need to register a task to perform during the maintenance window. For example, I can run a Linux shell script:
Software Inventory
This feature collects information about software and settings for a set of instances. To access it, I click on Managed Instances and Setup Inventory:
Setting up the inventory creates an association between an AWS-owned document and a set of instances. I simply choose the targets, set the schedule, and identify the types of items to be inventoried, then click on Setup Inventory:
After the inventory runs, I can select an instance and then click on the Inventory tab in order to inspect the results:
The results can be filtered for further analysis. For example, I can narrow down the list of AWS Components to show only development tools and libraries:
I can also run inventory-powered queries across all of the managed instances. Here’s how I can find Windows Server 2012 R2 instances that are running a version of .NET older than 4.6:
AWS Config Integration
The results of the inventory can be routed to AWS Config and allow you to track changes to the applications, AWS components, instance information, network configuration, and Windows Updates over time. To access this information, I click on Managed instance information above the Config timeline for the instance:
The three lines at the bottom lead to the inventory information. Here’s the network configuration:
Patch Management
This feature helps you to keep the operating system on your Windows instances up to date. Patches are applied during maintenance windows that you define, and are done with respect to a baseline. The baseline specifies rules for automatic approval of patches based on classification and severity, along with an explicit list of patches to approve or reject.
Here’s my baseline:
Each baseline can apply to one or more patch groups. Instances within a patch group have a Patch Group tag. I named my group Win2016:
Then I associated the value with the baseline:
The next step is to arrange to apply the patches during a maintenance window using the AWS-ApplyPatchBaseline document:
I can return to the list of Managed Instances and use a pair of filters to find out which instances are in need of patches:
Automation
Last but definitely not least, the Automation feature simplifies common AMI-building and updating tasks. For example, you can build a fresh Amazon Linux AMI each month using the AWS-UpdateLinuxAmi document:
Here’s what happens when this automation is run:
Available Now
All of the EC2 Systems Manager features and functions that I described above are available now and you can start using them today at no charge. You pay only for the resources that you manage.
— Jeff;
