Customization and field maintenance with AWS Snowball Edge – sideloading AMIs

When the AWS Snowball Edge launched in 2016, if you wanted to run Amazon Elastic Compute Cloud (Amazon EC2) instances on the device, you were required to specify an Amazon Machine Image (AMI) to be installed on it at time of ordering. You were then able to launch Amazon EC2 instances on the device based on that AMI. However, if you needed to update an AMI or use a different AMI to address a new workload, fix an issue, or enable new functionality, you needed to return the device to AWS to have the new or updated AMI loaded and then wait for it to ship back to you. That is no longer the case. In this blog, I show you how you can sideload your own customized AMIs, quickly and easily, onto the Snowball Edge.

AWS Snowball Edge

AWS Snowball Edge, a member of the AWS Snow Family, is an edge computing, data migration, and edge storage device that comes in two options:

• Snowball Edge Storage Optimized devices provide both block storage and Amazon S3-compatible object storage, and 40 vCPUs. They are well suited for local storage and large scale-data transfer
• Snowball Edge Compute Optimized devices provide 52 vCPUs, block and object storage, and an optional GPU for use cases like advanced machine learning and full motion video analysis in disconnected environments

You can use these devices for data collection, machine learning and processing, and storage in environments with intermittent connectivity (like manufacturing, industrial, and transportation) or in extremely remote locations (like military or maritime operations). Devices may also be clustered together to build larger temporary installations.

Both options support running of Amazon EC2 instances and AWS Lambda functions, so you can develop and test in the AWS Cloud, then deploy applications on devices in remote locations to collect, pre-process, and ship the data to AWS. Common use cases include data migration, data transport, image collation, IoT sensor stream capture, and machine learning.

Overview

1. Create your Virtual Machine (VM), customize, and export in raw format
2. Upload the raw image to Snowball Edge
3. Create IAM permissions for image import
4. Import the raw image as a snapshot
5. Register an AMI from the snapshot
6. Launch an EC2 instance from your AMI
7. Get access keys from Snowball (optional)

1. Create your Virtual Machine (VM), customize, and export in raw format

In this blog, I will be creating a CentOS 7 AMI, but you are free to use any operating system supported by Amazon EC2. In addition, I will be using Oracle VM VirtualBox to create the image to be uploaded to the Snowball. You are free to use any virtualization solution that has the ability to export an image in raw format.

Once you have your VM up and running in VirtualBox, carry out whatever customization you require.

Some customizations I needed to make included the following:

• Needed to set ‘onboot=yes’ in ifcfg-enp0s3, this defaulted to ‘no’
• Ran ‘yum update’
• Set hostname
• Ensured that network time protocol (ntp) was up and running
• Installed any required SSH public keys
• Installed required software and carried out relevant configuration steps

There may be many more depending on your own requirements.

Once you are happy that you have your VM fully configured, shut it down.

Identify the location of the virtual disk image for your VM. In VirtualBox, select Machine, Settings, Storage, then click on the .vdi entry shown under Storage Devices. The fully qualified disk image path is then shown on the right, as Location.

Make a note of the location, and use it to export the image as follows:

$ VBoxManage clonehd /run/media/jeff/0a6aadd8-05e9-4e14-a64e-c81403a16b86/CentOS/CentOS7/CentOS7.vdi /mnt/stage/CentOS7.raw --format raw
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone medium created in format 'raw'. UUID: b35dc0d7-ab01-4e00-801f-2476dcafe546
[jeff@heisenberg ~]$ ls -ltr /mnt/stage/
total 17853620
-rw-------. 1 jeff jeff 18282102784 Sep 29 10:21 CentOS7.raw
$

2. Upload the raw image to Snowball Edge

Use the AWS OpsHub for Snow Family client to upload the raw image file to an Amazon S3 bucket on the Snowball Edge.

Start up the OpsHub client and connect to the Snowball Edge, supplying the required credentials:

If this is your first time connecting to your Snowball, click on Snowball Edge, and enter the Unlock code and Manifest file, both of which are available in the AWS Management Console to the account owner. Next, enter the Device IP address, as shown on the Snowball Edge display panel. Once login is successful, an OpsHub profile is created and saved. For subsequent logins, you can simply click on Existing device profile, and select the desired profile from the drop-down menu.

Once you are logged in, select the Snowball’s Device id.

On the Snowball dashboard page, select Get started on the Manage storage panel:

Select the Bucket name for your raw image file:

Select Upload, browse for your raw image file, and select it. Your CentOS image is then uploaded to the Snowball’s local S3 storage.

3. Create IAM permissions for image import

Create an IAM role and associated policy for use by the VM Import/Export process.

3.1. Create IAM policy

Create an IAM policy that grants the appropriate permissions for the local VM Import/Export service to download the snapshot from Amazon S3 on the device.

Select << Back twice, to get back to the Snowball dashboard page. Select Services, Users & Permissions (IAM), Policies, Create Policy.

On the Create policy page, provide a Name and Description for the policy.

For the Policy content box, go to: Importing an Image into Your Device as an Amazon EC2 AMI.

Scroll down to Step 2. Set Up Required Permissions and paste in the example policy shown. Ensure that you modify the Resource ARNs to correctly reflect your objects.

Note: It is possible that the precise details of the policy you need to use may change over time. Always use the details shown in the documentation at the preceding link.

Your screen should now look something like this:

Finally, select Create to create the policy.

3.2. Create IAM role and attach policy

Create an IAM role with a trust policy that allows Snowball VM Import/Export to assume the role. In addition, attach the policy you created in the previous step, to allow VM Import/Export to access the image stored in the S3 bucket on the device.

Select << Back twice, to get back to the Users & Permissions (IAM) page, select Roles, Create role. Provide a Name and Description for the role.

For Policies, click on the drop-down and select the policy you created in the previous step.

For the Assume role policy document box, go to: Importing an Image into Your Device as an Amazon EC2 AM

Scroll down to Step 2. Set Up Required Permissions and paste in the example trust policy shown.

Note: It is possible that the precise details of the trust policy you need to use may change over time. Always use the details shown in the documentation at the preceding link.

Your screen should now look something like this:

Finally, select Create role to create the role.

4. Import the raw image as a snapshot

Select << Back twice, to get back to the Snowball dashboard page.

Select Get started on the Start computing panel, and select Snapshots, Import snapshot:

On the Import snapshot page, provide the Import Description and the Snapshot description. For Role, specify the IAM role you created in step 3.2. Then select Browse S3:

In the Choose raw file in S3 panel, locate and select your raw image file, followed by Choose:

Back on the Import snapshot page, select Submit:

The snapshot import will then proceed, taking a few minutes to complete, depending on your image size:

Once complete, the State shows Completed:

5. Register an AMI from the snapshot

Select the snapshot id you just created, followed by Register image to register the snapshot as an Amazon Machine Image (AMI).

Enter a Name and Description for the AMI. Leave the Root volume device as /dev/sda1, and select Submit:

Your snapshot has now been registered as an AMI, from which you are able to launch EC2 instances.

6. Launch an EC2 instance from your AMI

• Select << Back twice, to get back to the Snowball dashboard page, and select Instances:

Select Launch instance and enter your AMI name and the instance type you require.

For assignment of a public IP address, select Create public IP address (VNI) if this is the first time you are launching an EC2 instance on this Snowball. As public IP addresses remain assigned on the Snowball until explicitly deleted, you have the choice with subsequent EC2 launches of using a previously assigned public IP, by selecting Use existing IP address (VNI), or of creating another new one (for example, if you want to run multiple EC2 instances with public IP addresses). You also obviously have the choice of not attaching a public IP address at all.

With regards to the key pair, as mentioned earlier, I tend to add any required public keys into the image itself, so I would select Do not attach key pair here. You have the options of doing the same, of creating and using a new key pair, or of using an existing key pair.

Finally, select Launch and your EC2 instance will be initialized.

Once your EC2 instance is up and running, indicated by the State, it can be accessed in the same way as any other EC2 instance running in AWS.

7. Get access keys from Snowball (optional)

Your Snowball’s access key and secret access key are needed if you want to use the AWS CLI with your Snowball. If you do not need, or want, to use the AWS CLI, then this step can be skipped.

7.1. Install the Snowball client

You need to use the Snowball client to gain access to the Snowball’s access key and secret access key. Download the client and follow the instructions to install it on your local machine.

7.2. Get the Snowball access key

Run the Snowball client list-access-keys command, to obtain your Snowball’s access key, supplying the following inputs (without the square brackets):

• snowball_ip_address
• manifest_file_location
• unlock_code

$ snowball-client-linux-1.1.2-228/bin/snowballEdge list-access-keys \
--endpoint https://[snowball_ip_address] \
--manifest-file [manifest_file_location] \
--unlock-code [unlock_code]
{
    "AccessKeyIds" : [ "WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW" ] 
}

Make a note of the access key returned.

7.3 Get the Snowball secret access key

Using the access key you obtained from the previous step, run the Snowball client get-secret-access-key command, to get your Snowball’s secret access key, supplying the following inputs (without the square brackets):

• snowball_ip_address
• manifest_file_location
• unlock_code
• access_key_id

Make a note of the secret access key returned.

$ snowball-client-linux-1.1.2-228/bin/snowballEdge get-secret-access-key \
--endpoint https://[snowball_ip_address] \
--manifest-file [manifest_file_location] \
--unlock-code [unlock_code] \
--access-key-id WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
[snowballEdge]
aws_access_key_id = WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

7.4. Configure your AWS CLI environment

Configure your CLI environment by running aws configure. When prompted for the access key and secret access key, enter the values you obtained in steps 7.2 and 7.3. Enter snowball for the default Region.

$ aws configure
AWS Access Key ID [None]: WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
AWS Secret Access Key [None]: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Default region name [None]: snowball
Default output format [None]:

Finally, you can verify that your CLI environment is configured correctly by running aws ec2 describe-instances to display the EC2 instance(s) currently configured on your Snowball:

$ aws ec2 describe-instances --endpoint http://10.55.32.52:8008
{
    "Reservations": [
        {
            "Instances": [
                {
                    "AmiLaunchIndex": 0,
                    "ImageId": "s.ami-8558c7fcf1879bf14",
                    "InstanceId": "s.i-8ffd519099219ceca",
                    "InstanceType": "sbe-c.small",
                    "LaunchTime": "2021-09-30T18:02:31.915000+00:00",
                    "PrivateIpAddress": "34.223.14.193",
                    "PublicIpAddress": "10.55.32.51",
                    "State": {
                        "Code": 16,
                        "Name": "running"
                    },
                    "BlockDeviceMappings": [
                        {
                            "DeviceName": "/dev/sda1",
                            "Ebs": {
                                "AttachTime": "2021-09-30T18:03:03.590000+00:00",
                                "DeleteOnTermination": true,
                                "Status": "attached",
                                "VolumeId": "s.vol-869750ed44b4d1001"
                            }
                        }
                    ],
                    "EbsOptimized": false,
                    "EnaSupport": false,
                    "RootDeviceName": "/dev/sda1",
                    "SecurityGroups": [
                        {
                            "GroupName": "default",
                            "GroupId": "s.sg-8ba774385987e4ccc"
                        }
                    ],
                    "SourceDestCheck": false,
                    "CpuOptions": {
                        "CoreCount": 1,
                        "ThreadsPerCore": 1
                    }
                }
            ],
            "ReservationId": "s.r-889aac57dee08da77"
        }
    ]
}

Conclusion

Congratulations, you have just sideloaded a new, customized AMI onto your Snowball Edge and, from it, launched an EC2 instance. This enables you to respond quickly to new and changing requirements for your Snowball Edge in the field.

For next steps, take a look at AWS Snow Family, AWS Snowball resources, along with the AWS Snowball FAQs. For full details around configuration options and pricing, see AWS Snowball Pricing.

I hope this exercise proved useful, and I would be happy to answer any questions in the comments section.