AWS Storage Blog

Migrate Oracle databases to or within AWS using Amazon S3 integration with Oracle RMAN backups

When migrating to the cloud, customers generally take the opportunity to re-analyze their application and database workloads to optimize their move. Oracle databases are a popular database engine for on-premise infrastructures, but can often lead to increased total cost of ownership, with high upfront hardware costs and locked-in licensing periods. While planning their migration, customers are looking to reduce these costs and adopt efficiencies that are cloud nactive.

One Oracle database customers can modernize their architecture is by migrating to Amazon Elastic Compute Cloud (EC2) in a lift and shift method. This method reduces infrastructure costs, while enhances scalability, availability and efficiency. And, while migrating Oracle databases to Amazon EC2, many organizations prefer using Oracle’s in-built backup and recovery utilities like Oracle Recovery Manager (Oracle RMAN). Customers’ Database operation teams are familiar with this in-built oracle utility because it doesn’t involve a complex learning curve. Oracle RMAN provides encryption, which secures the backups for migration. However, customers encounter significant challenges, leading to increased recovery and migration timelines. Some of the challenges are manually creating, maintaining and migrating backups along with disk space shortage on both on-premises and Amazon EC2 servers.

Once the database is migrated to Amazon EC2, customers must store daily and weekly backups for the point in time recovery (PITR). They also need to have sustainable backup and retention policies to meet their long-term compliance requirements. Storing and maintaining these backups on a disk can be an expensive and inefficient strategy. Amazon Simple Storage Service (Amazon S3) can provide durable, scalable, and cost-effective storage option for Oracle RMAN backups.

In this post, we focus on resolving these significant storage issues during migration of Oracle RMAN database backups using Amazon S3. We showcase the integration of Amazon S3 with Oracle Recovery Manager (RMAN) utility to facilitate migrations of Oracle databases and storage of their backups.

Benefits of using Amazon S3 integration for Oracle RMAN backups

Amazon S3 integration with Oracle RMAN on Amazon EC2 is a perfect combination of getting the best of two worlds (AWS and Oracle) in AWS:

  1. For Oracle databases on Amazon EC2, customers can use the RMAN utility to take backups to Amazon S3, uses the latter for its durability, scalability, resilience, and availability. Additionally, Amazon S3 provides fast, easy, secure, and low-cost storage method to directly store these backups.
  2. These backups can be readily available to restore and create secondary production databases or to build lower or other adjacent database environments, in various availability zones, Regions, or even across AWS accounts.
  3. Using Amazon S3’s tiered storage options and lifecycle rules, customers can transition their backups from short-term to long-term retention to meet their compliance requirements.
  4. Amazon S3’s encryption is additional security for data at rest for these database backups.

Oracle RMAN backups:

Oracle RMAN is an Oracle-native tool to backup and restore Oracle databases at database level. The RMAN BACKUP command generates a backup set, which is a logical object containing one or more backup pieces. Each backup piece is a physical file in a binary format.

Solution overview

This post shows how you can integrate Amazon S3 with Oracle RMAN and migrate Oracle databases in two scenarios: one scenario is on-premises to AWS and second scenario is within AWS.

The process includes the following steps:

I. Configure and integrate Amazon S3 with Oracle RMAN: This section of the post has common steps for both the scenarios.

a. Download and configure Oracle Secure Backup (OSB) cloud module.

b. Integration Amazon S3 with Oracle RMAN using OSB.

II. Scenario 1: Migrate Oracle databases from on-premises to an Amazon EC2 using Amazon S3.

III. Scenario 2: Migrate Oracle databases within AWS from one Amazon EC2 instance to another using Amazon S3.

Prerequisites

To follow along with this post, you should have the following prerequisites:

To implement this solution, you must create the following resources:

Walkthrough

In this section, we walk through configuring and integrating Amazon S3 with Oracle RMAN, and then through two different migration scenarios.

Configure and integrate Amazon S3 with Oracle RMAN

In this section, we illustrate steps how to integrate Oracle RMAN with Amazon S3 using OSB.

1. Installing Java

In this step, we confirm that Java version 1.7 or higher is installed for OSB

  • Verify if Java is present.
  • If Java is present with an older version, then upgrade to the latest version.
  • If Java is absent, then download the latest version and install.
/* Command to check Java installation */

# which java  

/* Download Java */

# yum list java* 

# sudo yum install java-devel (Example: java-1.8.0-openjdk-devel)

/* Verify Java version post installation */

# which java 

# java -version

2. Create a wallet directory

In this step:

  • Create a directory in Oracle Home.
  • OSB wallet is stored here
# mkdir $ORACLE_HOME/osbws_wallet

3. Download OSB cloud module

In this step:

  • Download OSB from OTN site, which requires an active OTN enterprise account.
  • Upload the file to premises server or Amazon EC2 machine where the Oracle database is hosted.
  • Extract the OSB jar files in directory created in previous upload step.
/* Extract the OSB files into directory created in previous step*/

# cd $ORACLE_HOME/osbws_wallet

# ls osbws*.zip
  osbws_installer.zip

# unzip osbws_installer.zip

4. Install and configure OSB for Oracle home

To install OSB, the following input parameters are needed during this step.

These input parameter details are from the prerequisites:

  • Customer OTN username
  • Customer OTN Password
  • AWS Security ID of the IAM user
  • AWS Secret key of the IAM user
  • OTN user id
  • OTN user password

This directory and location of the OSB wallet were created from the previous step (Step 2):

/* Install OSB java installer */

# java -jar osbws_install.jar -AWSID <aws_user_id> -AWSKey <aws_user_key> -walletDir $ORACLE_HOME/dbs/osbws_wallet -libDir $ORACLE_HOME/lib/ -proxyHost <www-proxy.example.com>

# java -jar osbws_install.jar -AWSID <aws_user_id> -AWSKey <aws_user_key> -walletDir $ORACLE_HOME/osbws_wallet -libDir $ORACLE_HOME/lib -location us-east-1 -awsEndPoint s3.us-east-1.amazonaws.com -otnUser <username@company.com> -otnPass <OTN_passwd>

# java -jar osbws_install.jar  -walletDir $ORACLE_HOME/osbws_wallet -libDir $ORACLE_HOME/lib -IAMRole oracledb-role-ssm

5. Rename the OSB configuration file

In this step, the OSB file name is renamed to maintain the standard naming conventions throughout this post.

/* Rename OSB config file */

# cd $ORACLE_HOME/dbs

# cat osbwsorcl1.ora 

# mv osbwsorcl1.ora osbwsCONFIG.ora

Scenario 1: Migrate Oracle databases from on premises to an Amazon EC2 using Amazon S3

Description: This part of the post shows how to migrate an Oracle database from on-premises to an Amazon EC2 instance using Amazon S3.

These steps include:

  1. Take an Oracle RMAN backup of Source database on-premises to local storage disks.
  2. Copy the backups from local storage to Amazon S3 bucket.
  3. Restore the backups directly from an Amazon S3 bucket to the destination Amazon EC2 database instance.

The direct restoration of backups from Amazon S3 (Scenario 1, step 3) to a target database on Amazon EC2 helps avoid an additional intermediate step of downloading the backups from Amazon S3 to Amazon EC2 on an Amazon EBS or Amazon EFS volume. This practice makes the overall experience of database migrations faster, easier, and more cost effective.

The following architecture diagram presents an overview of the solution for scenario 1 in this post:

Scenario 1 Migrate Oracle databases from on premises to an Amazon EC2 using Amazon S3

Configuration steps for scenario 1

Amazon S3 is integrated to the on-premises Oracle RMAN. If there is a consent from customer security and compliance team to create database backups directly to Amazon S3, then follow the “Steps to integrate S3 with Oracle RMAN” from the previous section.

If there is no consent from the customer security and compliance team to create database backups directly to Amazon S3, then continue with the following steps.

In this post, we manually upload the database Oracle RMAN backups from an on-premises to Amazon S3.

  1. Create an Oracle RMAN backup of database on-premises

In this step, we complete the following:

  • Backup the source on-premise Oracle database using RMAN.
  • Verify the backup files after backup is completed.
/* Connect as SYSDBA to on premise database which is being migrated */

# rman target /

/* following RMAN command creates database backup with archive logs */

RMAN> run
{
ALLOCATE CHANNEL c1 DEVICE type disk;
BACKUP DATABASE PLUS ARCHIVELOG;
}

/* Create controlfile backup */

RMAN> backup current controlfile format ‘/location_of_backup_files/control_file.bkp';

/* Connect Source on premises Oracle database on SQL*PLUS */

SQL> create pfile='/location_of_backup_files/init.ora' from spfile;

/* List the backup files from the OS prompt */

[oracle@source ~]$ ls -lrt <location_of_backup_files>

-rw-r----- 1 oracle dba  1395593728 Oct 21 18:30 o1_mf_annnn_TAG20211021T183038_jq3dggx4_.bkp
-rw-r----- 1 oracle dba 10492379136 Oct 21 18:32 o1_mf_nnndf_TAG20211021T183053_jq3dgy8l_.bkp
-rw-r----- 1 oracle dba    11141120 Oct 21 18:32 o1_mf_ncsnf_TAG20211021T183053_jq3dky9k_.bkp
-rw-r----- 1 oracle dba        4608 Oct 21 18:32 o1_mf_annnn_TAG20211021T183231_jq3dkzjc_.bkp
  1. Upload the backups from on-premises database server to Amazon S3 bucket

In this step, we use a manual upload operation of files, which is automated.

/* Connect from on premises AWS CLI and list the S3 bucket */

aws s3 ls s3://<S3bucketName>

/* Copy files from on premises to S3 */

aws s3 cp <location_of_backup_files> s3://<S3BucketName> --recursive
  1. Startup the Destination EC2 Oracle database using Oracle DB parameter file (pfile)

In this step:

  • Copy database pfile from Amazon S3 to local Destination Amazon EC2 instance to a temporary directory.
  • Copy database controlfile from Amazon S3 to local Destination EC2 instance to a temporary directory.
  • Start the database in nomount state using the pfile.
/* Copy files from S3 to Destination EC2 instance on AWS */

# aws s3 cp s3://<S3BucketName>/init.ora /tmp/init.ora
# aws s3 cp s3://control_file.bkp /tmp/controlfile.bkp

/* Connect to Destination EC2 Oracle database using SQLPLUS */

#sqlplus / nolog

SQL> conn / as sysdba

SQL> startup nomount pfile=’/tmp/init.ora’;
  1. Restore the Oracle RMAN backup from Amazon S3 directly to destination database

In this step:

  • Connect to the target database which was started in nomount state in previous step using Oracle RMAN.
  • Restore the controlfile from the backup.
  • Start the target database in mount state using controlfile.
  • Restore the database.
/* Connect to EC2 database on which Recovery to be performed using RMAN utility – as SYSDBA */

# rman target /

/* Restore the control file which was restored from S3 in previous step */

RMAN> restore controlfile from ‘/tmp/control_file.bkp’;

RMAN> alter database mount;

RMAN>run
{
ALLOCATE CHANNEL c1 DEVICE type ‘SBT_TAPE’ PARMS ‘SBT_LIBRARY=libosbws.so ENV=(OSB_WS_PFILE=<$ORACLE_HOME>/dbs/osbws_restore_from_UAT.ora)’;
RESTORE DATABASE;
}
  1. Recover and open the database

In this step:

  • Recover the database.
  • Open the database with resetlogs.
/* Stay Connected to RMAN prompt from previous step */

RMAN> RECOVER DATABASE;

/* Once DB is completely recovered run below command */

RMAN> ALTER DATABASE OPEN RESETLOGS;
  1. Verify the database connectivity on destination Amazon EC2

In the final step of scenario 1, we verify the connection to the Oracle database and data that was restored on destination Amazon EC2.

/* Output: Database status should be active */

SQL> SELECT INSTANCE_NAME, DATABASE_STATUS, INSTANCE_ROLE from v$instance;

Once verified, the Oracle database restores on the destination EC2 instance and is now ready for users and regular operations.

Scenario 2: Migrate Oracle databases within AWS from one Amazon EC2 database instance to another Amazon EC2 using Amazon S3

Description: This part of the post shows the steps to migrate the Oracle database within AWS from one Amazon EC2 instance to another. Scenario 2 is optimal for Oracle databases migrated between Amazon EC2 instances across Regions, availability zones within same Region, and across AWS accounts.

The steps include:

  1. Take an Oracle RMAN backup of Source database on Source Amazon EC2 instance to Amazon S3 bucket.
  2. Restore the backups directly from an Amazon S3 bucket to the destination Amazon EC2 database instance.

The direct database backup and restore using Amazon S3 between source and destination Amazon EC2 instance makes the overall experience of database migrations fast, easy, and cost effective with minimal manual effort.

The following architecture diagram presents an overview of the solution for scenario 2 in this post:

Migrate Oracle databases within AWS from one Amazon EC2 database instance to another Amazon EC2 using Amazon S3

Configuration steps for scenario 2

Ensure the Amazon S3 integration with Oracle RMAN step is completed before moving with the following steps.

  1. RMAN backup of Oracle database on source Amazon EC2 instance

In this step:

  • Create RMAN backups of the Oracle database directly to Amazon S3 on Source Amazon EC2 instance.
  • Verify the backup files after the backup is completed.
/* Connect to the EC2 Oracle source database on which Backup & Recovery to be performed */

# as sysdba. $rman target /

/* This step takes backup files directly to S3 bucket using OSB */

RMAN> run
 {
 allocate channel c1 device type sbt parms='SBT_LIBRARY=libosbws.so, SBT_PARMS=(OSB_WS_PFILE=<$ORACLE_HOME>/dbs/osbwsCONFIG.ora)';
 BACKUP DATABASE PLUS ARCHIVELOG;}
  1. Backup control file and pfile copy

In this step:

  • Backup controlfile of the source Oracle database.
  • Create pfile from the source Oracle database.
/* Connect to RMAN as SYSDBA and take backup EC2 Source Oracle database with SQL*PLUS */

# rman target /

RMAN> backup current controlfile format ‘/location_of_backup_files/control_file.bkp';

/* Connect EC2 Source Oracle database with SQL*PLUS */

SQL> create pfile='/location_of_backup_files/init.ora' from spfile;
  1. Upload the controlfile and pfile from Amazon EC2 database instance to Amazon S3 bucket

In this step:

  • Copy the pfile to backup Amazon S3 bucket.
  • Copy the controlfile to backup Amazon S3 bucket.
  • If the database backups are huge, then see the best practices uploading large files.
/* List and copy files from EC2 to S3 */

#aws s3 ls s3://<S3bucketName>

#aws s3 cp <location_of_backup_files>/init.ora s3://<S3BucketName>

#aws s3 cp location_of_backup_files>/control_file.bkp s3://<S3BucketName>
  1. Startup the database on destination Amazon EC2 instance

In this step:

  • Download following files from Amazon S3 copied previously.
  • Oracle database pfile.
  • Oracle database controlfile backup.
  • Startup the database in nomount using the pfile.
/* List and copy files from S3 to destination EC2 */

#aws s3 cp s3://<S3BucketName>/init.ora /tmp/init.ora
#aws s3 cp s3://<S3BucketName>/control_file.bkp /tmp/controlfile.bkp

/* Connect to destination EC2 Oracle database using SQLPLUS & startup*/

#sqlplus / nolog

SQL> conn / as sysdba

SQL> startup nomount pfile=’/tmp/init.ora’;
  1. Restore the controlfile and mount the database

This step is done on destination Amazon EC2 instance

  • Restore the controlfile from the backup file that was downloaded previously.
  • Start the destination database in mount state.
/* Connect to RMAN using SYSDBA and restore the control file and mount the database */

#rman target /

RMAN> restore controlfile from ‘/tmp/control_file.bkp’;

RMAN> alter database mount;
  1. Restore the database

In this step, we restore the database from Oracle RMAN backups directly from Amazon S3 on to the Destination Amazon EC2 instance.

/* do not disconnect RMAN prompt from previous steps */

RMAN> run
   {
    ALLOCATE CHANNEL c1 DEVICE type ‘SBT_TAPE’ PARMS   ‘SBT_LIBRARY=libosbws.so ENV=(OSB_WS_PFILE=<$ORACLE_HOME>/dbs/osbws_restore_from_UAT.ora)’;
    RESTORE DATABASE;
   }
  1. Recover and open the database

In this step:

  • Recover the database
  • Open the database with resetlogs
/* Stay Connected to RMAN prompt from previous step */

RMAN> RECOVER DATABASE;

/* Once the Recover is complete then run the below step */

RMAN> ALTER DATABASE OPEN RESETLOGS;
  1. Verify the database connectivity on Destination Amazon EC2

This is the final step of scenario 2. During this step, we verify the connection to the Oracle database and data which was recovered on destination Amazon EC2.

/* Output: Database status should be active */

SQL> SELECT INSTANCE_NAME, DATABASE_STATUS, INSTANCE_ROLE from v$instance;

Once verified, the Oracle database is restored on the destination Amazon EC2 and is ready for users and regular operations.

Cleaning up

The following checklist cleans up any additional components that were created in the above steps:

  1. Drop the Amazon S3 bucket if it is not needed.
  2. Drop the OSB wallet and its directory.
  3. Remove the RMAN OSB SBT_TAPE configuration.
  4. Test the normal RMAN backup on the databases without Amazon S3.

Conclusion

In this post, we gave a walkthrough of the issues that customers faced while migrating Oracle databases to AWS and within AWS. We demonstrated how you can effectively integrate Amazon S3 with Oracle RMAN to store and migrate Oracle database backups. We also described migration options, which included  on-premises (i.e., from a corporate data center) to AWS or within the AWS.

With the walkthrough provided in this post, you should have a better understanding of the overall migration process and its benefits of using Amazon S3 to reduce overall time, effort, and costs of migration.

To get started on AWS, or to learn more about building a well-architected AWS environment, visit the AWS database migrations getting started page for guidance. Thank you for reading this post.

TAGS: , , ,
Zeeshan Mirza

Zeeshan Mirza

Zeeshan (Zee) Mirza is a Database Consultant with the Professional Services team at AWS. He works with customers with their journey to the cloud with a focus on complex database migration programs. In his spare time, Zee enjoys traveling to new places with his wife and riding his bicycle whenever weather permits.

Harish Lingegowda

Harish Lingegowda

Harish Lingegowda is a Database Consultant with the Professional Services team at AWS. He works as a database migration specialist to help Amazon customers migrate their on-premises database environment to AWS Cloud database solutions.

Mansi Suratwala

Mansi Suratwala

Mansi Suratwala is a Database Consultant working at Amazon Web Services. She closely works with companies from different domains to provide scalable and secure database solutions in AWS cloud. She is passionate to collaborate with customers to achieve their cloud adoption goals.

Vijaya Kumar Mallela

Vijaya Kumar Mallela

VijayaKumar Mallela is a Database Consultant with the Professional Services team at AWS. He works as a database migration specialist to help Amazon customers migrate their on-premises database environment to AWS Cloud database solutions.