AWS Database Blog
Automate Amazon RDS backups using the Oracle RMAN utility and upload backup files to Amazon S3
Amazon Relational Database Service (Amazon RDS) for Oracle creates automated snapshots and allows creation of user-initiated manual snapshots in a Region. In addition, Amazon RDS for Oracle supports Oracle native backup tools like Oracle Recovery Manager (RMAN) and Oracle Data Pump.
Many organizations spend considerable time and resources to manage and maintain database backups to meet business and regulatory requirements for data protection and long-term backup retention. Based on your backup compliance policies, you can choose Oracle native tools to backup the RDS for Oracle database during the following scenarios:
- The backup retention time required is greater than maximum retention period (35 days) of RDS automated snapshots and
- The number of RDS manual snapshots exceeds the limits per Region (100 snapshots per Region) for long term retention we can increase the limit as this limit is modifiable as long as the snapshots are needed.
For example, organizations with backup compliance requirements to retain Oracle database backups for seven or ten years with requirement of Point in time recovery (PITR) options can also choose Oracle RMAN to backup RDS for Oracle database and store the backup files on Amazon S3, using S3 for durable, scalable, and cost-effective storage.
You can restore the RMAN database backups in the same or different Region on Amazon EC2 or on premises for disaster recovery purposes, or to demonstrate backup compliance to auditors and regulators to meet business and regulatory requirements.
In this post, we show you how to automate creation of Oracle RMAN backups on an RDS database and upload RMAN backup files to Amazon S3, providing a cost-effective solution to store database backups for longer periods of time. We use AWS Lambda and Amazon SNS to notify database administrators on the status (completion or failure) of the automation task.
Understanding RDS snapshots and Oracle RMAN backups
RDS creates a storage volume snapshot of your DB instance, backing up the entire DB instance. These snapshots are stored in Amazon S3. RDS snapshots include the following:
- RDS automated snapshots creates the first snapshot of a DB instance containing the data for the full DB instance. Subsequent snapshots of the same DB instance are incremental for automated snapshot. The maximum backup retention period of automated snapshots on Amazon RDS is 35 days.
- RDS manual snapshots creates the snapshot of a DB instance containing the data for the full DB instance and cannot be used for point in time recovery (PITR). Manual snapshots are not deleted automatically; they must be explicitly deleted. Visit Quotas and constraints for Amazon RDS for the number of manual snapshots that can be created per Region (the limit does not apply to automated snapshots).
Automated backups and manual snapshots are stored in an S3 bucket that is owned and managed by the Amazon RDS service. Hence, you are not able to see them from your Amazon S3 console.
Oracle Recovery Manager (RMAN) is an Oracle native tool to backup and restore Oracle database at DB level. The RMAN BACKUP command generates a backup set, which is a logical object containing one or more backup pieces. Each backup piece is a physical file in a binary format. Amazon RDS for Oracle uses the rdsadmin.rdsadmin_rman_util package to perform RMAN backups of an Oracle database on the RDS host.
Solution overview
We present a solution to automate creation of RMAN backups on the RDS for Oracle database and upload RMAN backup files to Amazon S3 for longer retention. The following architecture diagram presents an overview of the solution in this post:
The process includes the following steps:
- The Oracle DBMS job scheduler schedules the creation of Oracle RMAN backups using an Oracle stored procedure on the RDS for Oracle database.
- The stored procedure copies the RMAN backup files generated on the RDS host to an S3 bucket using Amazon S3 integration.
- After completion of RMAN backup, the Oracle stored procedure uploads a status file with the success or failure of the task to an S3 bucket.
- Uploading the status file to the S3 bucket invokes an AWS Lambda function.
- The Lambda function uses Amazon SNS to send an email notification to database administrators (customers or subscribers) of the success or failure of the task.
The following high level diagram depicts AWS Identity and Access Management (IAM) role and policies created to grant access to AWS resources.
The solution copies Oracle RMAN backups created on RDS host to Amazon S3. We use an Amazon S3 bucket accessible through the IAM role from an RDS instance. The Lambda function uses an IAM role that has an AWS Lambda execution role and IAM policies attached granting access to read file in S3 bucket and publish SNS notification to the subscriber.
Prerequisites
To follow along with this post, you should have the following prerequisites:
- Familiarity with the following AWS services:
- AWS Command Line Interface (AWS CLI)
- AWS Lambda
- Amazon RDS for Oracle
- Amazon SNS
- Amazon S3
- Amazon EC2
- Amazon Virtual Private Cloud (Amazon VPC)
- An RDS for Oracle database (or a restore from a snapshot).
- A client host to connect to the database, such as an Amazon EC2 instance or bastion host in the same Amazon VPC, with an Oracle client (for example, SQL*Plus or SQLcl).
- A dedicated S3 bucket for the backups. For more information, see Creating a bucket. Use a dedicated bucket: if it contains unrelated objects, the stored procedure’s upload status check may not complete as expected.
- Storage for backup staging. RMAN backup pieces are written to a directory on the DB instance host (for example,
DATA_PUMP_DIR) before they’re uploaded to Amazon S3. You can use a default directory or create a new directory. To avoid oversizing your primary volume, you can instead stage backups on an additional storage volume and remove it afterward.Additional storage volumes require an instance with at least 64 GiB of memory and a primary volume of 200 GiB or larger, and each additional volume has a 200 GiB minimum. For details, see Working with storage in RDS for Oracle.
To implement this solution, you must create the following resources:
- Amazon RDS for Oracle integration with Amazon S3 (an option group with the
S3_INTEGRATIONoption and an IAM role attached to the DB instance), described in the next section - An SNS topic and subscription
- An IAM role and policy for the Lambda function
- A Lambda function
- An S3 event trigger
- An Oracle PL/SQL stored procedure on the RDS for Oracle database
- An Oracle Scheduler (
DBMS_SCHEDULER) job to run the procedure on a schedule
Configure Amazon RDS for Oracle integration with Amazon S3
Amazon RDS for Oracle supports integration with Amazon S3 to transfer data between your RDS for Oracle DB instances and Amazon S3. This integration provides a secure way to use your S3 bucket to copy RMAN backup files from Amazon RDS for Oracle and share them for compliance or retention requirements, which you can access from both Amazon RDS for Oracle and other database hosts. Amazon S3’s lifecycle rules help you save costs by automating object transition from one storage class to another.
Configuring the integration requires an option group with the S3_INTEGRATION option attached to your DB instance, and an IAM role attached to the instance that grants access to your S3 bucket. The following steps use the AWS CLI.
- Create an option group, add the
S3_INTEGRATIONoption, and associate the option group with your DB instance. Match--engine-nameand--major-engine-versionto your DB instance (for example,oracle-se2or a different version): - Create an IAM role that Amazon RDS can assume, and attach a policy granting access to your S3 bucket:
- Attach the role to your DB instance for the
S3_INTEGRATIONfeature: - Confirm the role is associated and its status is
ACTIVEbefore running the procedure:
ACTIVE. If the role is missing, the upload task fails with “The DB instance doesn’t have credentials to access the specified Amazon S3 bucket.”
Configure an SNS topic
We use Amazon SNS to notify database administrators of the status (completion or failure) of the RMAN task on the Amazon RDS for Oracle database, including the upload of RMAN backup files from the RDS host to Amazon S3.
- Create a standard SNS topic:
- Using the topic ARN returned by the previous command, subscribe an email endpoint:
Amazon SNS sends a confirmation email to the endpoint. Choose Confirm subscription in that email before testing; the endpoint won’t receive notifications until the subscription is confirmed.
Configure an IAM role and policies
In this step, we configure the IAM role and policies for the Lambda function that sends notifications. (This is separate from the S3 integration role you attached to the DB instance in the previous section.)
- Create an IAM role using the AWS CLI (for this post, called
RMAN-backup-automate-S3-role). The following trust policy allows Lambda to assume the role: - Create an IAM policy (
lambda-sns-policy) to allow the function to publish to the SNS topic (RMAN_S3_BACKUP_NOTIFY): - Create an IAM policy (
RMAN-backup-automate1-S3) to allow the function to read the status file from your S3 bucket: - Attach the
lambda-sns-policypolicy to the roleRMAN-backup-automate-S3-role: - Attach the
RMAN-backup-automate1-S3policy to the roleRMAN-backup-automate-S3-role: - Attach the AWS managed policy
AWSLambdaBasicExecutionRoleto the role. The Lambda execution role grants the function permission to write logs: - Confirm the policies are attached to the role:
Configure a Lambda function
We use a Lambda function to send email notifications using Amazon SNS about the success or failure of RMAN backups on Amazon RDS for Oracle and the upload of RMAN backup files to Amazon S3. The function reads the status file the stored procedure uploads to Status/job_status.txt and publishes the result to the SNS topic. To create the function, complete the following steps:
- Save the following code as
rman_notify.py: - Package the file into a deployment
.zip: - Create the function with the AWS CLI, using the execution role from the previous section and setting the environment variables. We use Python 3.13; we recommend the latest available runtime:
Configure an S3 bucket event trigger
In this section, you configure a trigger on your S3 bucket so that when the stored procedure uploads the status file, it invokes the Lambda function RMAN-backup-automate-S3. Complete the following steps:
- Grant Amazon S3 permission to invoke the function:
- Configure the bucket to invoke the function when the status file is created. The filter matches the
Status/prefix and thejob_status.txtsuffix, so only the status file triggers the function: - Confirm the notification configuration was applied:
Create a stored procedure on the RDS for Oracle DB instance
We create a stored procedure on the RDS for Oracle database that uses RDS packages to set the configurations and create backups using the Oracle RMAN utility.
- From your client host, connect to the RDS for Oracle DB instance as the master user (such as
admin). - Create the directory used for the status file:
- Retain archive logs long enough for the Oracle RMAN tool to use them. This example uses 48 hours (see Retaining archived redo logs):
- Create the stored procedure. It removes any leftover backup pieces, runs a full RMAN backup, uploads the backup pieces to Amazon S3, and writes a status file to
Status/job_status.txt(which triggers the notification). Run the following in your SQL client:
Schedule the backup
The stored procedure runs on demand, but you’ll typically want backups to run automatically. Create an Oracle Scheduler (DBMS_SCHEDULER) job that calls the procedure on a schedule. Adjust repeat_interval and the bucket argument to match your backup policy; you can customize it to your liking (for example, a different time of day or frequency):
Test the solution
Now you can test the stored procedure on your RDS for Oracle instance to create backups using the RMAN utility. RMAN backup files are copied from the RDS host to an S3 bucket using the Amazon S3 integration.
- Connect to the RDS for Oracle DB instance as the admin user and run the stored procedure. In this post,
amzn-s3-demo-bucketis the example S3 bucket: - Optionally, list the RMAN backup pieces created on the DB instance. The pieces are written to
DATA_PUMP_DIR: - Confirm the files were uploaded to Amazon S3. List the bucket to see the backup pieces at the root and the status file under
Status/:You should see the RMAN backup pieces (names beginning with
BACKUP) and the status fileStatus/job_status.txt, similar to the following (sizes and timestamps will vary): - Optionally, check the status file contents:
On a successful run, the file contains
File loaded to s3 success.
When the status file is written to amzn-s3-demo-bucket/Status/, the S3 event notification triggers the Lambda function, which uses Amazon SNS to email the subscriber the success or failure of the task. Check your inbox for the notification.
You can move these RMAN backups from Amazon S3 to Amazon S3 Glacier for long-term storage that complements your automated backup strategy. For more information, see Transitioning objects using Amazon S3 Lifecycle.
Clean up
On its next run, the stored procedure removes leftover RMAN backup pieces from the DB instance directory (DATA_PUMP_DIR); it does not delete objects from Amazon S3. Manage S3 retention with S3 Lifecycle rules, or delete objects manually. To remove the resources created in this post, delete the following:
- The Oracle Scheduler job and the stored procedure on the RDS for Oracle DB instance:
- The Lambda function and its execution role and policies —
RMAN-backup-automate-S3,RMAN-backup-automate-S3-role,lambda-sns-policy, andRMAN-backup-automate1-S3(To delete a Lambda function) - The S3 event notification on the bucket
- The SNS topic and associated subscription (Deleting an Amazon SNS subscription and topic)
- The Amazon S3 integration you configured for the DB instance — the IAM role (
rds-oracle-s3-integration) and the option group (oracle-s3-integration). Detach the role and reset the instance to its previous option group before deleting the option group. - The S3 bucket and its contents (Deleting a bucket)
- The RDS for Oracle database instance (Deleting a DB instance)
Conclusion
In this post, we showed you how to automate RMAN backups on Amazon RDS for Oracle to Amazon S3 and send email notifications on the completion or failure of the backups, which reduces operational overhead. With the approach described in this post and using Amazon S3 lifecycle policies, you can achieve a cost-effective and scalable solution with unlimited storage maintaining database backups for higher retention periods.
Try out the solution for your real-world use cases, and leave your feedback, thoughts, and ideas in the comments.
About the Authors
Zeeshan (Zee) Mirza is a Database Consultant with the Professional Services team at AWS. He works with customers in their journey to the cloud with a focus on complex database migration programs. In his spare time, Zee enjoys traveling to new places with his wife and riding his bicycle whenever weather permits.
Harish Lingegowda is a Database Consultant with the Professional Services team at AWS. He works as a database migration specialist to help Amazon customers migrate their on-premises database environment to AWS Cloud database solutions.
Nethravathi Muddarajaiah is a Senior Database Specialist Solutions Architect at AWS. She works with our customers to provide guidance and technical assistance on database projects, helping them improve the value of their solutions when using AWS.
Vijaya Kumar Mallela is a Database Consultant with the Professional Services team at AWS. He works as a database migration specialist to help Amazon customers migrate their on-premises database environment to AWS Cloud database solutions.
