AWS Storage Blog

Automatic tape deletion with AWS Storage Gateway

Virtual tapes offer a great replacement for cumbersome and expensive physical tapes and are ideal for hybrid workloads looking to take advantage of cloud scale and availability. Virtual tapes can already be less costly than physical tapes and offer more flexibility, less overhead, and better security, but you can save even more money through proper storage procedures throughout virtual tape lifecycles.

With virtual tapes on AWS Storage Gateway’s Tape Gateway, when your backup application ejects a virtual tape no longer needed for frequent access, the gateway moves the tape from the virtual tape library (VTL) on Amazon S3 Standard to the virtual tape shelf (VTS) on the Amazon S3 Glacier storage classes, where the tape is archived for long-term storage. Based on policies configured within the backup application, data on the archived virtual tape expires when the retention period is met. Although the customer backup application marks the archived tape as expired, the corresponding tape stored in AWS isn’t deleted by the backup application—the delete must be done manually.

In this post, as an example, I walk you through an automated process to identify virtual tapes that have expired in the Veeam Backup & Replication backup application but still exist in the VTS and delete them using a Python script. I also present how to configure an AWS Lambda function that runs the Python script at a scheduled frequency. This helps customers to reduce there archived storage cost.

Comparing components of Tape Gateway and a physical tape library

In the backup and disaster recovery world, a significant number of customers are using physical tapes to safeguard their data. However, there are many challenges in maintaining a large physical tape library, such as:

  • Purchasing new tapes, renewing hardware warranties
  • Managing physical tape inventory
  • Physical tape destruction for expired legacy media
  • Physical logistics of managing resources to swap out tapes and ship them offsite
  • Dealing with cleaning cartridges
  • Data center space limitations
  • Large amounts of power and cooling consumption

AWS Storage Gateway’s Tape Gateway enables you to replace physical tapes on premises without changing existing backup workflows. Tape Gateway presents a virtual tape library (VTL) consisting of virtual tape drives and a virtual media changer to your backup application using storage industry-standard iSCSI protocol. Tape Gateway supports all leading backup applications and caches virtual tapes on premises for low-latency data access. Tape Gateway encrypts data between the gateway and AWS for secure data transfer. It also compresses data and transitions virtual tapes between Amazon S3 and Amazon S3 Glacier Flexible Retrieval, or Amazon S3 Glacier Deep Archive, to minimize storage costs.

Customers can continue to use their existing backup applications and workflows while writing to a nearly limitless collection of virtual tapes. When the tapes are no longer required for frequent access, customers can transition them to an archive tier that resides in Amazon S3 Glacier Flexible Retrieval or Amazon S3 Glacier Deep Archive by ejecting and exporting them from their backup application, further reducing storage costs. Tape Gateway stores the virtual tapes in service-managed S3 storage and creates new virtual tapes automatically, simplifying management and easing customers’ transition to cloud storage.

Here I compare and contrast the components of Tape Gateway and a physical tape library infrastructure, including the entire lifecycle of a virtual tape, to demonstrate where the archives present themselves.

When a Tape Gateway is created and activated, the following components will be created by default.

  • 1 media changer: Similar to a robotic arm that moves tapes in a physical tape library.
  • 10 tape drives: Similar to physical tape drives where the actual tape cartridge will be inserted to read/write to the tape.
  • 1,500 tape slots: These are similar to physical slots within a physical tape library where the tapes are placed temporarily when it is not being loaded to tape drives.
  • 1,500 import/export slots: These are similar to physical slots where the newly created tapes are placed before the library imports/inventories them.
  • Virtual tape shelf (VTS): This is like an offsite tape archive location, where all the virtual tapes will be archived for long-term storage. This VTS comes with virtually unlimited storage capacity backed by Amazon S3 Glacier Flexible Retrieval or Amazon S3 Glacier Deep Archive.

When the backup application starts a backup job that is configured to use the Tape Gateway, the media changer will loads the required number of tapes from the tape slot to individual tape drives. The backup application will then begins to write to those tapes. Upon completion, they may be ejected or exported from the VTL, and tapes will be marked as archived and sent to the VTS.

Note that eject or export is a function the customer can configure in their backup job within their backup application, and these names may differ depending on the backup application.

When a customer wants to retrieve the data from an archived tape, they would first select Retrieve tape in the Storage Gateway console, which moves the tape from VTS to VTL before placing it into the import/export slots. The backup application will then import the tape into a tape slot and load it into the tape drive for reading the data.

Solution overview

In this blog, I show how you can use a Python script to automatically delete all of the archived tapes in an AWS account that have exceeded their retention period and don’t need to be stored any longer.

The following steps are performed in the script.

  1. You configure the input value specified in days, which will be used as a retention period for the tapes.
  2. The script performs DescribeTapeArchives API call and builds a list of tapes that are marked as archived. It then calculates the time difference between the current UTC and the tape’s archival date. Later, it builds a final list of virtual tapes that are marked as expired in the backup application but still exist in the VTS.
  3. Finally, it deletes those tapes from your AWS account with DeleteTapeArchive API call.

In order to run the script periodically, I also configure it as a Lambda function that kicks off on a predefined schedule. This periodically runs a python script against your specified AWS account, which accepts a cutoff date (tape retention days) as an input.

Solution walkthrough

In the following steps, I walk you through configuring a Lambda function to automatically delete archived tapes on a predefined schedule. First, I create a lambda function using the AWS Management Console. Then I configure permissions to the lambda function, and schedule it to run at my desired frequency.

Create a Lambda function using the AWS Management Console

  1. Open the AWS Lambda console and choose Create Function.

  1. In the Create Function dialog box, configure the following settings:
    • For Function name, enter StorageGatewayTapeDeletion.
    • For Runtime, choose Python 3.8.
    • To grant basic execution permission to the Lambda function, choose Create a new role with basic Lambda permissions.
    • Choose Create function.

  1. The console creates a Lambda function with a single source file named lambda_function.py.
  2. From the code editor section, select the lambda_function.py. For information on using the code editor, refer to the AWS Lambda Developer Guide.

  1. From the Editor pane in the code editor, replace the default code with the following code.
"""
********** README **********
THIS SCRIPT HAS TWO MAIN SECTIONS.
    1. "if" SECTION RUNS IF TOTAL NUMBER OF ARCHIVES ARE LESS THAN OR EQUAL TO 1000
    2. "else" SECTION RUNS IF TOTAL NUMBER OF ARCHIVES ARE MORE THAN 1000
IF YOU WANT TO DRY RUN THIS SCRIPT WITHOUT DELETING THE ACTUAL TAPES,
PLEASE COMMENT OUT THE LINE OF CODE CONTAINING "response = sg.delete_tape_archive(TapeARN=tape[0])" FROM BOTH SECTIONS BY COMMENTING THEM USING "#" AT THE START OF THE LINE.
"""

import boto3
import datetime
import csv
import time
import json

def lambda_handler(event, context):
    Retention = 150  # ==> Update the Cutoff date to your Tape Retention days. Archived Tapes will be deleted after 'n' days depending on this value.
    cutoff = datetime.datetime.utcnow() - datetime.timedelta(days=Retention)
    tapeCount = 0
    print("Establishing connection to AWS\n")
    sg = boto3.client('storagegateway')
    print("Gathering Tape Archive Information\n")
    tapeDict = sg.describe_tape_archives()
    marker = tapeDict.get('Marker')
    if marker == None:
        oldTapes = [(i['TapeARN'], i['CompletionTime'].replace(tzinfo=None)) for i in tapeDict['TapeArchives'] if i['CompletionTime'].replace(tzinfo=None) < cutoff]
        newTapes = [(i['TapeARN'], i['CompletionTime'].replace(tzinfo=None)) for i in tapeDict['TapeArchives'] if i['CompletionTime'].replace(tzinfo=None) > cutoff]
        print("\n" + "=" * 100)
        print("Following is the list of TapeARN's with expiry dates that are removed from Tape Archives (VTL) : ")
        print("=" * 100 + "\n")
        for tape in oldTapes:
            convertedDate = tape[1].date()
            print("%s, %s" % (tape[0] , convertedDate))
        tapeCount += len(oldTapes) + len(newTapes)
        print("\n" + "=" * 100)
        print("Total number of tapes considered for Deleting: ", len(oldTapes))
        print("=" * 100 + "\n")
        print("=" * 100)
        print("Reponses for each deletion:")
        print("=" * 100 + "\n")
        for tape in oldTapes:
            try:
                response = tape[0]
                #response = sg.delete_tape_archive(TapeARN=tape[0])       #  ====================> *****Comment this line by adding '#' to the start of the line, if you want to dry run the script without deleting the actual tapes.
                print(response)
            except:
                print("Exception Error")
                continue
    else:
        while marker != None:
            oldTapes = [(i['TapeARN'], i['CompletionTime'].replace(tzinfo=None)) for i in tapeDict['TapeArchives'] if i['CompletionTime'].replace(tzinfo=None) < cutoff]
            newTapes = [(i['TapeARN'], i['CompletionTime'].replace(tzinfo=None)) for i in tapeDict['TapeArchives'] if i['CompletionTime'].replace(tzinfo=None) > cutoff]
            print("\n" + "=" * 100)
            print("\nFollowing is the list of TapeARN's with expiry dates that are removed from Tape Archives (VTL) : ")
            print("=" * 100 + "\n")
            for tape in oldTapes:
                convertedDate = tape[1].date()
                print("%s, %s" % (tape[0] , convertedDate))
            tapeCount += len(oldTapes) + len(newTapes)
            print("\n\nTotal number of tapes considered for Deleting: ", len(oldTapes))
            print("\nReponses for each deletion:")
            for tape in oldTapes:
                try:
                    response = tape[0]
                    #response = sg.delete_tape_archive(TapeARN=tape[0])    #  ====================> *****Comment this line by adding '#' to the start of the line, if you want to dry run the script without deleting the actual tapes.
                    print(response)
                except:
                    print("Exception Error")
                    continue
            if marker == 'lastrun':
                break
            try:
                tapeDict = sg.describe_tape_archives(Marker=marker)
                doesmarkerexist = tapeDict.get('Marker','yes')
                if doesmarkerexist != 'yes':
                    newmarker = tapeDict['Marker']
                    if marker != newmarker:
                        marker = newmarker
                else:
                    marker = 'lastrun'
            except Exception as er:
                break
    print("\n" + "=" * 100)
    print("\n Script Executed Successfully. Total tapes processed: %s" % tapeCount)
    print("\n Total Tapes Deleted: %s" % len(oldTapes))
    print("\n" + "=" * 100)
    return{
        "statusCode": 200,
        "Total Tapes Processed": tapeCount,
        "Total Tapes Deleted": len(oldTapes)
    }
  1. To save the changes, in the Environment window under the left-hand menu bar, choose File and select Save. To deploy the changes, choose Deploy from the top menu bar.

Configure permission to the Lambda function

  1. Navigate to the Configuration tab and select Permissions.
  2. Under Execution role, select the Role name, which then opens up the IAM console with the respective role. Make a note of the Lambda execution Role ARN.

  1. Create a new IAM policy with the following permissions and attach it to the Lambda execution role along with the default policy that is attached to the role by the Lambda function when the function is created.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "storagegateway:DeleteTapeArchive",
                "storagegateway:DescribeTapeArchives"
            ],
            "Resource": "*"
        }
    ]
}

Schedule the Lambda function to run at the desired frequency

AWS supports the use of Amazon EventBridge event rules to initiate Lambda functions on a schedule. You can specify up to minute-level granularity when it comes to scheduling Lambda functions to be run.

To create a rule from the Amazon EventBridge console:

  1. Open the Amazon EventBridge console.
  2. In the navigation pane, expand Events and select rules, and then select Create rule.
  3. Enter a name and description for the rule. For Event bus, select default event bus.
  4. For Rule type, choose Schedule and choose Next.

  1. For Schedule pattern, choose A schedule that runs at a regular rate, such as every 10 minutes, and enter the rate expression of your choice and select Next.

  1. For Targets types, choose AWS service and select Lambda function from the dropdown.

  1. For Function, select the Lambda function that you created and select Next and then Create rule.

Example results

In this section, I show the outputs of the Lambda function with an example of Veeam Backup & Replication backup application.

  1. The following screenshot shows the backup job details in the Veeam Backup & Replication application console. For example, I have configured Daily job, which writes backup data directly to tapes.

  1. The following screenshot shows how I have configured Veeam Backup & Replication backup job settings to export the tape as soon as the backup application is done writing to the tape.

  1. After a couple of days, the Veeam Backup & Replication application is showing the tapes that are marked expired from the backup application’s perspective.

  1. In the Storage Gateway console, the tapes are showing as Archived.

  1. Once the Lambda function is executed, it deletes those expired tapes from the VTS. You can validate the same in the CloudWatch Logs output of the Lambda function, as shown in the following screenshot. Once those are removed from VTS, you can remove those tape barcodes from your Veeam Backup & Replication application configuration safely.

  1. Reviewing the Storage Gateway console, the expired tapes have been deleted.

Cleaning up

To avoid incurring unwanted AWS costs after performing these steps, delete the AWS resources created for this demonstration, which include the Amazon EventBridge rule that was scheduled and the AWS Lambda function.

Conclusion

AWS Storage Gateway’s Tape Gateway allows customers to store a nearly limitless capacity of virtual tapes. However, as the number of expired tapes grows, customers are looking for ways to automatically delete them, which helps customers to reduce there cost. The script covered in this article empowers customers to automatically delete expired virtual tapes based on their custom retention period. In addition, this blog covers how to configure and use AWS Lambda and Amazon EventBridge rules to run the script on a predefined schedule.

Thank you for taking the time to read my blog. Kindly leave us with your feedback in the comment section.

TAGS: , , , ,
Madhukumar Adavi

Madhukumar Adavi

Madhukumar Adavi is a Storage Specialized Solution Architect with Amazon Web Services. He works directly with customers to help, facilitate, and accelerate their transition to the cloud while resolving migration issues, removing obstacles, making suggestions, and teaching on best practices, particularly with regard to AWS DataSync. He enjoys watching movies with his family and playing games with his son.