Best Practices for Local File Parameters

aws service command --parameter file://path_to_file

aws service command --parameter fileb://path_to_file

Parameter value is a long text body

One of the most common use cases for file:// is when the input is a long text body. For example, if I had a shell script named myshellscript that I wanted to run when I launch an Amazon EC2 instance, I could pass the shell script in when I launch my instance from the CLI:

$ aws ec2 run-instances --image-id ami-b66ed3de 
    --instance-type m3.medium 
    --key-name mykey 
    --security-groups my-security-group 
    --user-data file://myshellscript

This command will take the contents of myshellscript and pass it to the instance as user data such that once the instance starts running, it will run my shell script. You can read more about the different ways to provide user data in the Amazon EC2 User Guide.

Parameter requires JSON input

Oftentimes parameters require a JSON structure as input, and sometimes this JSON structure can be large. For example, let’s look at launching an EC2 instance with an additional Amazon EBS volume attached using the CLI:

$ aws ec2 run-instances --image-id ami-b66ed3de 
   --instance-type m3.medium 
    --key-name mykey 
    --security-groups my-security-group 
   --block-device-mappings '[{"DeviceName":"/dev/sdf","Ebs":{"VolumeSize":20,"DeleteOnTermination":false,"VolumeType":"standard"}}]'

Notice that the --block-device-mappings parameter requires JSON input, which can be somewhat lengthy on the command line. So, it would be convenient if you could specify the JSON input in a format that is easier to read and edit, such as in the form of a text file:

[
  {
    "DeviceName": "/dev/sdf",
    "Ebs": {
      "VolumeSize": 20,
      "DeleteOnTermination": false,
      "VolumeType": "standard"
    }
  }
]

By writing the JSON to a text file, it becomes easier to determine if the JSON is formatted correctly, and you can work with it in your favorite text editor. If the JSON above is written to some local file named myinput.json, you can run the same command as before using the myinput.json file as input to the --block-device-mappings parameter:

$ aws ec2 run-instances --image-id ami-b66ed3de 
   --instance-type m3.medium 
    --key-name mykey 
    --security-groups my-security-group 
   --block-device-mappings file://myinput.json

This becomes especially useful if you plan to reuse the input.json file for future ec2 run-instances commands since you will not have to retype the entire JSON input.

AWS Key Management Service (KMS) decryption

KMS is an AWS service that makes it easy for you to create and control the encryption keys used to encrypt your data. You can read more about KMS in the AWS Key Management Service Developer Guide. One service that KMS provides is the ability to encrypt and decrypt data using your KMS keys. This is really useful if you want to encrypt arbitrary data such as a password or RSA key. Here is how you can use KMS to encrypt data using the CLI:

$ aws kms encrypt --key-id my-key-id --plaintext mypassword
   --query CipherTextBlob --output text

CiAxWxaLB2LyTobc/ppFeNcSLW/abxdFuvBdD3IBtHBTYBKRAQEBAgB4MVsWiwdi8k6G3P6aRX
jXEi1v2m8XRbrwXQ9yAbRwU2AAAABoMGYGCSqGSIb3DQEHBqBZMFcCAQAwUgYJKoZIhvcNAQcBM
B4GCWCGSAFlAwQBLjARBAyE/taUnrxXzSqa1+8CARCAJSi8/E819toVhfxm2A+T9mFdOfnjGuJI
zGynaCB3FsPXnrwl7vQ=

This command uses the KMS key my-key-id to encrypt the data mypassword. However, in order for the CLI to properly display content, the encrypted data output from this command is base64 encoded. So by base64-decoding the output, you can store the data as a binary file:

$ aws kms encrypt --key-id my-key-id --plaintext mypassword
   --query CipherTextBlob 
   --output text | base64 --decode > my-encrypted-password

Then if I want to decrypt the data in my file, I can use KMS to decrypt my encrypted binary:

$ echo "$(aws kms decrypt  --ciphertext-blob fileb://my-encrypted-file 
   --query Plaintext --output text | base64 --decode)"
mypassword

Since the file is binary, I use fileb:// as opposed to file:// to read in the contents of the file. If I were to read the file in as text via file://, the CLI would try to decode the binary file using my set system encoding. However since the binary file has no encoding, decoding errors would be thrown:

$ echo "$(aws kms decrypt  --ciphertext-blob file://my-encrypted-file 
   --query Plaintext --output text | base64 --decode)"

'utf8' codec can't decode byte 0x8b in position 5: invalid start byte

EC2 User Data

Looking back at the EC2 user data example from the Parameter value is a long text body section, file:// was used to pass the shell script as text to --user-data. However in some cases, the value passed to --user-data is a binary file.

One limitation of passing user data when launching an EC2 instance is that the user data is limited to 16 KB. Fortunately, there is a way to help avoid reaching this limit. By utilizing the cloud-init package on EC2 instances, you can gzip-compress your cloud-init directives because the cloud-init package will decompress the user data for you when the instance is being launched:

$ aws ec2 run-instances --image-id ami-b66ed3de 
    --instance-type m3.medium 
    --key-name mykey 
    --security-groups my-security-group 
    --user-data fileb://mycloudinit.gz

By gzip-compressing the file, the cloud-init directive becomes a binary file. Subsequentially, the gzip-compressed file must be passed to the --user-data using fileb:// in order to read in the contents of the file as binary.