Batch Upload Files to Amazon S3 Using the AWS CLI
Introduction
Implementation
Create an AWS IAM User
In this step, you will use the IAM service to create a user account with administrative permission. In later steps, you will use this user account to securely access AWS services using the AWS CLI.
1. Sign in to the console
Click on the AWS Management Console home to open the console in a new browser window, so you can keep this step-by-step guide open. When this screen loads, enter your user name and password to get started. Then type IAM in the search bar and select IAM to open the Identity and Access Management dashboard.
2. Choose Users
From the AWS Identity and Access Management dashboard, click on Users on the left side.
3. Create a user
Click the Add user button.
4. Specify user details
Enter a user name in the textbox next to User name: (we’ll use AWS_Admin for this example) and select Programmatic access in the Select AWS Access Type section. Click the Next: Permissions button.
5. Add permissions
Click on Attach existing policies directly option. Select AdministratorAccess then click Next: Tags.
6. Add tags
IAM tags are key-value pairs you can add to your user. We’ll skip this step for this example. Click the Next: Review button.
7. Review and create
Take this opportunity to review that all settings are correct. When you are ready, click on Create user.
8. Review and create
Click the Download Credentials button and save the credentials.csv file in a safe location (you’ll need this later in step 3) and then click the Close button.
Install and Configure the AWS CLI
Now that you have your IAM user, you need to install the AWS CLI. Below are instructions based on the kind of operating system you are using; please select the tab that corresponds to your operating system.Select PC from the tabs below if you are using a Windows-based computer.Select Mac/Linux from the tabs below if you are using a machine running MacOS or Linux.
Install and configure the AWS CLI for PCs
Download and run the Windows installer (64-bit, 32-bit).
Note: Users of Windows Server 2008 v6.0.6002 will need to use a different install method, listed in the AWS Command Line Interface User Guide.
Open a command prompt by pressing the Windows Key + r to open the run box and enter cmd and press the OK button.
Type aws configure and press enter. When prompted, enter the following:
AWS Access Key ID [None]: Enter the Access Key Id from the credentials.csv file you downloaded earlier
Note: This should look something like AKIAPWINCOKAO3U4FWTN
AWS Secret Access Key [None]: Enter the Secret Access Key from the credentials.csv file you downloaded earlier
Note: This should look something like 5dqQFBaGuPNf5z7NhFrgou4V5JJNaWPy1XFzBfX3
Default region name [None]: Enter us-east-1
Default output format [None]: Enter json
Install and configure the AWS CLI for Mac / Linux
Follow these directions for installing the AWS CLI bundled installer.
MacOS users: Open a terminal window by pressing Command + Space and typing terminal in the search window. Then press enter to open the terminal window.
Linux users: Open a terminal window.
Type aws configure and press enter. Enter the following when prompted:
AWS Access Key ID [None]: Enter the Access Key Id from the credentials.csv file you downloaded earlier
Note: This should look something like AKIAPWINCOKAO3U4FWTN
AWS Secret Access Key [None]: Enter the Secret Access Key from the credentials.csv file you downloaded earlier
Note: This should look something like 5dqQFBaGuPNf5z7NhFrgou4V5JJNaWPy1XFzBfX3
Default region name [None]: Enter us-east-1
Default output format [None]: Enter json
Using the AWS CLI with Amazon S3
In this step, you will use the AWS CLI to create a bucket in Amazon S3 and copy a file to the bucket.
1. Create an S3 bucket
Creating a bucket is optional if you already have a bucket created that you want to use. To create a new bucket named my-first-backup-bucket type:
aws s3 mb s3://my-first-backup-bucketNote: Bucket naming has some restrictions; one of those restrictions is that bucket names must be globally unique (e.g., two different AWS users can not have the same bucket name); because of this, if you try the command above you will get a BucketAlreadyExists error.
2. Upload files to S3
To upload the file my first backup.bak located in the local directory (C:\users) to the S3 bucket my-first-backup-bucket, you would use the following command:
aws s3 cp “C:\users\my first backup.bak” s3://my-first-backup-bucket/Or, use the original syntax if the filename contains no spaces.
3. Download files from S3
To download my-first-backup.bak from S3 to the local directory we would reverse the order of the commands as follows:
aws s3 cp s3://my-first-backup-bucket/my-first-backup.bak ./4. Delete files from S3
To delete my-first-backup.bak from your my-first-backup-bucket bucket, use the following command:
aws s3 rm s3://my-first-backup-bucket/my-first-backup.bak
Conclusion
Congratulations! You have set up an IAM user, configured your machine for use with the AWS command line interface, and learned how to create, copy, retrieve, and delete files from the cloud.