AWS Cloud Operations Blog
How to Manage Credentials in AWS OpsWorks for Puppet Enterprise using Hiera-eyaml
For customers new to configuration management with AWS OpsWorks for Puppet Enterprise (OWPE), a frequently-asked question is “How do I store sensitive data, such as database passwords, for use in my manifests?” Hiera allows you to manage and access data from various backends (data storage locations). By default, Hiera 5 supports YAML, JSON, and HOCON backends and only plaintext data files and values.
With some additional configuration, Hiera 5 supports the eyaml backend, which allows administrators to define backend data files with encrypted data (without having to encrypt the file in its entirety). This provides security of data at rest while allowing fast lookup and ease of use/review by administrators. Additionally, plaintext values can be included in the same file. Hiera-eyaml supports encrypted arrays, hashes, and nested arrays/hashes.
Enabling Hiera-eyaml
The eyaml command line tool is made available for working with encrypted data files. To install and use hiera-eyaml, follow these instructions.
gem install hiera-eyaml
Import the following modules from the Puppet Forge to your Master.
- https://forge.puppet.com/puppet/hiera
- https://forge.puppet.com/puppetlabs/puppetserver_gem
- https://forge.puppet.com/puppetlabs/inifile
- https://forge.puppet.com/puppetlabs/stdlib
The following example Puppetfile includes the required modules.
forge "http://forge.puppetlabs.com" # Modules from the Puppet Forge mod "puppetlabs/concat" mod "puppetlabs/ntp" mod "puppetlabs/stdlib" mod "puppet/staging" mod "puppet-logrotate" mod "puppet/nginx" mod "puppetlabs/inifile" mod "puppetlabs/puppetserver_gem" mod "puppet/hiera"
Create a profile manifest to configure Hiera. This will enable three hierarchies (Virtual yaml, Nodes yaml, and Default yaml file). In the starter kit provided when creating an AWS OpsWorks for Puppet Enterprise instance, this would be located in [STARTER_KIT]/site/profile/manifests/hiera.pp.
class profile::hiera {
class { 'hiera':
hiera_version => '5',
hiera5_defaults => {"datadir" => "/etc/puppetlabs/code/environments/%{::environment}/hieradata", "data_hash" => "yaml_data"},
hierarchy => [
{"name" => "Virtual yaml", "path" => "virtual/%{::virtual}.yaml"},
{"name" => "Nodes yaml", "paths" => ['nodes/%{::trusted.certname}.yaml', 'nodes/%{::osfamily}.yaml']},
{"name" => "Default yaml file", "path" => "common.yaml"},
],
eyaml => true,
eyaml_gpg => true,
eyaml_gpg_recipients => 'mark@example.com,chris@example.com',
create_keys => false,
keysdir => '/etc/puppetlabs/code-staging/keys',
provider => puppetserver_gem,
}
}
Ensuring that you are following best practices, you can then add the newly created hiera profile to a role for our Puppet Master. This would be placed in [STARTER_KIT]/site/role/manifests/puppet_master.pp.
class role::puppet_master {
include profile::hiera
}
After steps 2-4 have been completed, and the control repository has been synchronized with the Puppet master (via webhook and puppet-code deploy), the role should be available for classification on the Puppet Master. This can be done in the Classification console under “All Nodes” -> “PE Infrastructure” -> “PE Master”. If for some reason the role is not appearing as a classification option, ensure that you have refreshed class definitions.
To quickly synchronize these changes, you can manually execute an Agent run on the Master using the PE console. At this point, the necessary configuration to support hiera-eyaml has been configured. The next steps would be to install encryption keys securely on the Puppet Master.
Secure Key Storage
After eyaml has been configured on the Master, you need to generate and securely store the asymmetric key pairs used for encryption and decryption of sensitive data. This can be done with the eyaml createkeys command from your workstation. The private key should be stored in a secure location that can only be accessed by the Puppet Master and any authorized users. In this case, you can leverage an Amazon S3 bucket with a strict access policy that allows only the OWPE instance’s IAM profile and administrators to our AWS account. The sample policy that follows demonstrates this. With this policy, it is important to ensure that the s3:GetObject and s3:ListBucket permissions are added to the aws-opsworks-cm-ec2 role. After the policy has been applied and the OWPE Amazon EC2 role has been updated, have an administrator upload the public and private keys to the bucket.
{
"Version": "2012-10-17",
"Id": "HieraKeysPolicy",
"Statement": [
{
"Sid": "AllowOpsWorksCMInstanceRole",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNT_ID:role/service-role/aws-opsworks-cm-ec2-role"
},
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::BUCKET_NAME/*",
"arn:aws:s3:::BUCKET_NAME"
]
},
{
"Sid": "AllowAdminAccess",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNT_ID:user/ADMIN_USER"
},
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::BUCKET_NAME/*",
"arn:aws:s3:::BUCKET_NAME"
]
}
]
}
After this policy has been applied, public and private keys can be generated with the eyaml createkeys command. This will automatically place the key files in a ./keys directory on your workstation. The public/private keys can then be uploaded to the S3 bucket, and you can modify the hiera.pp manifest to include importing these files to /etc/puppetlabs/code-staging/keys. This directory is configured as part of the Hiera profile created previously, and is included in the backup/restore procedures.
Since these files are not created with in-line content, or from files/templates stored within the module itself, you need to make use of the exec Puppet resource. Since the AWS CLI is installed by default on OWPE Masters, no additional configuration/installation is required. The benefit of this approach is that the AWS CLI will make use of instance profiles to determine access permissions to S3. You need to confirm that the role assigned to nodes in your environment has appropriate S3 permissions (specifically s3:ListBucket on the bucket itself, and s3:GetObject for any key files).
First, update the Hiera profile to include importing the public/private key files.
class profile::hiera {
class { 'hiera':
hiera_version => '5',
hiera5_defaults => {"datadir" => "/etc/puppetlabs/code/environments/%{::environment}/hieradata", "data_hash" => "yaml_data"},
hierarchy => [
{"name" => "Virtual yaml", "path" => "virtual/%{::virtual}.yaml"},
{"name" => "Nodes yaml", "paths" => ['nodes/%{::trusted.certname}.yaml', 'nodes/%{::osfamily}.yaml']},
{"name" => "Default yaml file", "path" => "common.yaml"},
],
eyaml => true,
eyaml_gpg => true,
eyaml_gpg_recipients => 'mark@example.com,chris@example.com',
create_keys => false,
keysdir => '/etc/puppetlabs/code-staging/keys',
provider => puppetserver_gem,
}
exec { 'get-private-key':
path => ['/usr/bin','/usr/sbin','/bin'],
cwd => '/etc/puppetlabs/code-staging',
command => 'aws s3api get-object --bucket BUCKET_NAME --key path/to/private/key.pem /etc/puppetlabs/code-staging/keys/private_key.pkcs7.pem',
creates => '/etc/puppetlabs/code-staging/keys/private_key.pkcs7.pem',
}
exec { 'get-public-key':
path => ['/usr/bin','/usr/sbin','/bin'],
cwd => '/etc/puppetlabs/code-staging',
command => 'aws s3api get-object --bucket BUCKET_NAME --key path/to/public/key.pem /etc/puppetlabs/code-staging/keys/public_key.pkcs7.pem',
creates => '/etc/puppetlabs/code-staging/keys/public_key.pkcs7.pem',
}
exec { 'update-key-perms':
path => ['/usr/bin','/usr/sbin','/bin'],
cwd => '/etc/puppetlabs/code-staging',
command => 'chown -R pe-puppet:pe-puppet /etc/puppetlabs/code-staging/keys && chmod -R 0500 /etc/puppetlabs/code-staging/keys && chmod 0400 /etc/puppetlabs/code-staging/keys/*.pem',
require => [
Exec['get-private-key'],
Exec['get-public-key'],
],
}
}
After this is complete and deployed to your control repository, run Puppet Agent on the master and deploy the changes with puppet-code deploy --wait --all --config-file .config/puppet-code.conf --token-file .config/puppetlabs/token (executed from the root of your starter kit). To verify the modifications work as intended, test the puppet lookup tool to verify you are able to retrieve data from common.yaml.
$ puppet lookup message --explain
Searching for "message"
Global Data Provider (hiera configuration version 5)
Using configuration "/etc/puppetlabs/puppet/hiera.yaml"
Hierarchy entry "Virtual yaml"
Path "/etc/puppetlabs/code/environments/production/hieradata/virtual/xenhvm.yaml"
Original path: "virtual/%{::virtual}.yaml"
Path not found
Hierarchy entry "Nodes yaml"
Path "/etc/puppetlabs/code/environments/production/hieradata/nodes/opsworks-1703-ohq1bhd0cetefkis.us-east-1.opsworks-cm.io.yaml"
Original path: "nodes/%{::trusted.certname}.yaml"
Path not found
Path "/etc/puppetlabs/code/environments/production/hieradata/nodes/RedHat.yaml"
Original path: "nodes/%{::osfamily}.yaml"
Path not found
Hierarchy entry "Default yaml file"
Path "/etc/puppetlabs/code/environments/production/hieradata/common.yaml"
Original path: "common.yaml"
Found key: "message" value: "This node is using common data"
Now, you can update the Hiera data within your control repository to include encrypted values. Note that this will require separate additions to the hierarchy specified in the Hiera class. Thus, you must first update hiera.pp to include the additional hierarchy layer.
class profile::hiera {
class { 'hiera':
hiera_version => '5',
hiera5_defaults => {"datadir" => "/etc/puppetlabs/code/environments/%{::environment}/hieradata", "data_hash" => "yaml_data"},
hierarchy => [
{"name" => "Secret Data", "lookup_key" => "eyaml_lookup_key", "paths" => ['common.eyaml'], "options" => { "pkcs7_private_key" => "/etc/puppetlabs/code/keys/private_key.pkcs7.pem", "pkcs7_public_key" => "/etc/puppetlabs/code/keys/public_key.pkcs7.pem" } },
{"name" => "Virtual yaml", "path" => "virtual/%{::virtual}.yaml"},
{"name" => "Nodes yaml", "paths" => ['nodes/%{::trusted.certname}.yaml', 'nodes/%{::osfamily}.yaml']},
{"name" => "Default yaml file", "path" => "common.yaml"},
],
eyaml => true,
eyaml_gpg => true,
eyaml_gpg_recipients => 'mark@example.com,chris@example.com',
create_keys => false,
keysdir => '/etc/puppetlabs/code-staging/keys',
provider => puppetserver_gem,
}
exec { 'get-private-key':
path => ['/usr/bin','/usr/sbin','/bin'],
cwd => '/etc/puppetlabs/code-staging',
command => 'aws s3api get-object --bucket BUCKET_NAME --key path/to/private/key.pem /etc/puppetlabs/code-staging/keys/private_key.pkcs7.pem',
creates => '/etc/puppetlabs/code-staging/keys/private_key.pkcs7.pem',
}
exec { 'get-public-key':
path => ['/usr/bin','/usr/sbin','/bin'],
cwd => '/etc/puppetlabs/code-staging',
command => 'aws s3api get-object --bucket BUCKET_NAME --key path/to/public/key.pem /etc/puppetlabs/code-staging/keys/public_key.pkcs7.pem',
creates => '/etc/puppetlabs/code-staging/keys/public_key.pkcs7.pem',
}
exec { 'update-key-perms':
path => ['/usr/bin','/usr/sbin','/bin'],
cwd => '/etc/puppetlabs/code-staging',
command => 'chown -R pe-puppet:pe-puppet /etc/puppetlabs/code-staging/keys && chmod -R 0500 /etc/puppetlabs/code-staging/keys && chmod 0400 /etc/puppetlabs/code-staging/keys/*.pem',
require => [
Exec['get-private-key'],
Exec['get-public-key'],
],
}
}
After this is complete, add an encrypted YAML file with some test data. Note that there is a specific format required when adding encrypted values, as seen in the following snippet.
# From the same directory as ./keys/ eyaml edit CONTROL_REPO/hieradata/common.eyaml
For testing purposes, we created a new value.
--- encryptedmessage: DEC::PKCS7['Hello, World!']!
After this is complete, you can commit and push the changes to your control repository, sync them with the Puppet Master, and run Puppet Agent on the master to implement the updated Hiera configuration. To validate that this works as expected, output the file contents, and compare them to the output of puppet lookup.
$ cat /etc/puppetlabs/code/environments/production/hieradata/common.eyaml --- encryptedmessage: ENC[PKCS7,ENCRYPTED_STRING] $ puppet lookup encryptedmessage Searching for "encryptedmessage" Global Data Provider (hiera configuration version 5) Using configuration "/etc/puppetlabs/puppet/hiera.yaml" Hierarchy entry "Secret Data" Path "/etc/puppetlabs/code/environments/production/hieradata/common.eyaml" Original path: "common.eyaml" Found key: "encryptedmessage" value: "'Hello, World!'"
At this point, you can add additional hierarchies for environment or fact-based data, and manage sensitive information within. From your manifest code, you can simply query Hiera with the lookup() function as normal. You need to create and maintain encryption keys, and distribute them to administrators who need to manage Hiera data.
About the Author
Nick Alteen is a Lab Development Engineer at Amazon Web Services, previously a Cloud Support Engineer. In both roles, he enjoys developing training and providing guidance to customers for usage of AWS services to fit their needs.