AWS Storage Blog

Accelerate development refresh cycles and optimize cost with Amazon FSx for NetApp ONTAP cloning

Organizations perform refreshes of database environments to ensure they can perform development, testing, and QA on copies of the most up-to-date, production-like datasets. In addition, IT teams are tasked with managing increasing data footprints, while maintaining the highest levels of availability, resiliency, and performance. Many customer development cycles maintain DTAP (Development, Test, Acceptance, and Production) data environments, all of which require infrastructure and resources. As more versions and refreshes are required, more storage is required.

With Amazon Elastic Compute Cloud (Amazon EC2) and Amazon FSx for NetApp ONTAP, you can clone large databases in seconds. And, you can do this while not consuming any additional storage capacity, accelerating times to refresh, and reducing resource consumption across your testing environments. To accelerate refreshes of database environments (from production to a development environment), three key data management features of FSx for ONTAP can be leveraged: Snapshots, Flexclones, and SnapMirror replication.

In this post, we walk through the process of refreshing a database environment from production into development using Amazon FSx for NetApp ONTAP and its native features. You can reduce operational complexity, improve agility, and reduce costs by using FSx for ONTAP to snapshot and clone your database and application data sets across multiple environments within minutes. In this blog, we demonstrate the end-to-end refresh process with Oracle Database Server 19c.

Solution overview

In this example, we show you how to leverage FSx for ONTAP with Oracle Database Server running on Amazon EC2 to accelerate your data environment refreshes and deliver a cost-optimized data layer for your database/application data sets. The same concepts can be applied to other databases, such as MySQL, PostgreSQL, MariaDB, Oracle, SAP HANA, and/or applications where multiple data sets are required.

FSx for ONTAP provides fully managed, shared storage in the AWS Cloud with the popular data access and management capabilities of ONTAP.

In using FSx for ONTAP’s combined features of Snapshots, FlexClones, and SnapMirror, you can swiftly and cost efficiently perform environment refreshes. For example, if you’re running a database workload and want to test a database operation prior to executing it against your production database, you can test the operation by creating clones of your database and then making it available to your various testing environments

There are several ways you can use Amazon FSx file systems in your Oracle Database deployments. In this scenario, we present NFS shares to be mapped as data, log, and Oracle home drives to the Oracle Database 19c instance running on Amazon EC2.

These drives (presented via NFS shares) are the subject of the data set and form the basis for the process of snap, mirror, clone, and present to a development Oracle Database Server. This architecture includes a production FSx for ONTAP File system and a development FSx for ONTAP filesystem. Even though this architecture assumes isolated production and development environments, the same process can also be performed on the same file system (without needing to SnapMirror to a separate file system).

FSx for ONTAP file system replicated to secondary FSx for ONTAP file system

Image 1: FSx for ONTAP file system replicated to secondary FSx for ONTAP file system

Prerequisites

  • This post assumes that the reader understands Oracle Databases, Linux mounting of NFS shares, and has familiarity with FSx for ONTAP. To create a similar deployment in your environment, you must meet the following prerequisites.An AWS account with access to Amazon EC2 and FSx for ONTAP
  • EC2 instances deployed with Oracle Database installed (or another Database Application), with data and log volumes presented as per diagram (or similar)
  • FSx for ONTAP (Single or Multi-Availability Zone (AZ)) deployed with cluster peered and SnapMirror set up for Oracle volumes (you can also run this without mirroring to a development environment and create clones locally on the production file system)
  • FSx for ONTAP volumes created and presented as NFS shares to the Oracle Database server as per diagram
  • Access to both the Oracle Database Servers and FSx for ONTAP file systems via AWS Command Line Interface (AWS CLI) in both production and development environments
  • Intermediate Oracle skills as SYSDBA

Walkthrough: Refresh database environment from production to development

The following is a detailed walkthrough of the steps required to perform the end-to-end process of cloning the Oracle 19c Database environment in the previous architecture diagram. By stepping through this process, you can take an application consistent snapshot of the Oracle Database volume, replicate across to the development environment, create the clones, and then mount the read/write volumes into the development Oracle Database server. Then, you can perform testing and QA as per your requirements. We also include the optional steps of either 1, deleting and/or 2, splitting the clones off into independent datasets.

FSx for ONTAP clones created from replicated volumes and mounted to Oracle Dev Server

Image 2: FSx for ONTAP clones created from replicated volumes and mounted to Oracle Dev Server

In this example, we walk you through the process of performing the database refresh into development. We cover the full refresh to development environment process, along with the post process of either cleaning up the cloned databases or maintaining them independently.

At a high level, the process is as follows:

  1. Put the Oracle Database into backup mode (to make sure of application consistent snapshot)
  2. Snap the Oracle Database Volumes in FSx for ONTAP
  3. Take the Oracle Database out of backup mode
  4. Perform the SnapMirror update to take across the Snapshots just performed, and create the clones from the snapshots on Dev FSx for ONTAP filesystem
  5. Mount the clone volumes into the development Oracle Server
  6. Start the database and immediately shut it down
  7. Finalize Oracle operations; Update the development Oracle Database Server configuration files, mount the cloned Database into the development server, and validate the Database is up and open for read write operations

Implementation

  1. Put the Oracle Database into backup mode (to make sure of application consistent snapshot).

In SQL Plus, put your database into backup mode.

 SQLPLUS> ALTER DATABASE BEGIN BACKUP;
  1. Snap the Oracle Database Volumes in FSx for ONTAP.

In the FSx for ONTAP Filesystem CLI, snap the volumes which your database files are on (label the snapshot a suitable name).

FsxId01234567890abc:> vol snapshot create -vserver fsx -volume prddb_orahome -snapshot prddb_orahome_snapshot_28March2023
FsxId01234567890abc:> vol snapshot create -vserver fsx -volume prddb_oradb -snapshot prddb_oradb_snapshot_28March2023
FsxId01234567890abc:> vol snapshot create -vserver fsx -volume prddb_oraarch -snapshot prddb_oraarch_snapshot_28March2023
  1. Take the Oracle Database out of backup mode, and backup the database control file to a trace file to be re-used later in the development environment.
SQLPLUS> ALTER DATABASE END BACKUP;

SQLPLUS>
alter database backup controlfile to trace as '/home/oraebs/controfilescript.sql';

With the application consistent snapshots now available, we can proceed to mirror these across to the development environment (note that you could also perform cloning and mounting locally).

  1. Perform the SnapMirror update to take across the Snapshots just performed.

In the development FSx for ONTAP Filesystem CLI, update the SnapMirror relationship to make sure that the snapshots are transferred across (if you aren’t using a development environment and want to use clones locally, you can skip this step).

FsxId01234567890abc:> snapmirror update -destination-path <SVM name>:devdb_orahome_DP -source-snapshot prddb_orahome_snapshot_28April2023
FsxId01234567890abc:> snapmirror update -destination-path <SVM name>:devdb_oradb_DP -source-snapshot prddb_oradb_snapshot_28April2023
FsxId01234567890abc:> snapmirror update -destination-path <SVM name>:devdb_oraarch_DP -source-snapshot prddb_oraarch_snapshot_28April2023

Once updated, you see the snapshots available. Run the following commands to validate the snapshot names as listed previously in step 2.

FsxId01234567890abc:> snap list devdb_orahome_DP

FsxId01234567890abc:> snap list devdb_oradb_DP

FsxId01234567890abc:> snap list devdb_oraarch_DP

Create the clones from the snapshots on Dev FSx-N filesystem.

In the development FSx for ONTAP file system CLI, create the volume clones referencing the snapshots that were mirrored across.

FsxId01234567890abc:> vol clone create -vserver fsx_dev -flexclone devdb_orahome_clone_28March2023 -type RW -parent-volume devdb_orahome_DP -parent-snapshot prddb_orahome_snapshot_28March2023 -junction-path /devdb_orahome -junction-active true

FsxId01234567890abc:> vol clone create -vserver fsx_dev -flexclone devdb_oradb_clone_28March2023 -type RW -parent-volume devdb_oradb_DP -parent-snapshot prddb_oradb_snapshot_28March2023 -junction-path /devdb_oradb -junction-active true

FsxId01234567890abc:> vol clone create -vserver fsx_dev -flexclone devdb_oraarch_clone_28March2023 -type RW -parent-volume devdb_oraarch_DP -parent-snapshot prddb_oraarch_snapshot_28March2023 -junction-path /devdb_oraarch -junction-active true
  1. Mount the Clone volumes into the development Oracle Server.

In the Oracle development server create directories for your mount points (or skip this point if you already have mount points created on your Development server/s).

# mkdir -p /devdb_orahome
# mkdir -p /devdb_oradb
# mkdir -p /devdb_oraarch

Mount the created volume clones to the mount points required.

# mount -t nfs -o nconnect=16 <SVM NFS DNS name or IP address>/devdb_orahome_clone_28March2023 /devdb_orahome

# mount -t nfs -o nconnect=16 <SVM NFS DNS name or IP address>/devdb_oradb_clone_28March2023 /devdb_oradb

# mount -t nfs -o nconnect=16 <SVM NFS DNS name or IP address>/devdb_oraarch_clone_28March2023 /devdb_oraarch
  1. Start the database and immediately shut it down.
SQLPLUS> startup;
SQLPLUS>shut immediate;
  1. Update the development Oracle Database Server configuration files.

On the development Oracle Database server, update the control file trace SQL script which was backed up in the earlier step (controlfilescript.sql).

# vi controlfilescript.sql

Update the file contents to match the new DB name and locations of log file and datafile directories

Mount the cloned Database into the development server. In SQL Plus, mount the database via the control file SQL script and put reset the database logs.

SQLPLUS> @controlfilescript.sql;
SQLPLUS> ALTER DATABASE OPEN RESETLOGS;

In Oracle, validate that the Database is operational.

SQLPLUS> SELECT STATUS FROM V$INSTANCE;
SQLPLUS> SELECT NAME FROM V$DATABASE;

With the process above completed, the database from production was refreshed across to development. This enables you to do read write testing on the database. This process is fast and efficient, doesn’t require extra storage, and only consumes the storage of changed blocks.

Validate the usage of the volume clones via the following command and viewing the final column, which displays the percentage of physical used storage for the volume/s.

FsxId01234567890abc:> vol show -vserver <vserver name> -volume <volume clone name> -fields size,used,available,percent-used,physical-used,physical-used-percent

Clean up or maintain new copy of DBs process

The preceding steps placed the volume clones and development database into read write for testing, QA, and other activities. With that process complete, now you must decide whether to remove the test datasets or split them into an independent dataset.

In the following tasks, we walk you through the process of doing either a delete of the clone Database or a split process.

Delete development database

Dismount the database and delete the clones.

  1. Dismount the database in Oracle and unmount the NFS mount points from the Oracle Development Server.
  2. Offline and delete the volume clones from the development FSx for ONTAP filesystem, using the following commands.
FsxId01234567890abc:> volume offline -vserver <vserver name> -volume <volume clone name>

FsxId01234567890abc:> volume delete -vserver <vserver name> -volume <volume clone name>

Maintain the development database

Split the clones and continue normal development operations.

This process forms an independent set of volumes on the current FSx Filesystem and consumes the full storage usage of the dataset.

  1. Split the volume clones into independent volumes.
FsxId01234567890abc:> volume clone split start -vserver <vserver name> -volume <volume clone name>

2. (Optional) As the volumes maintain the original clone name, rename the volumes to suit.

FsxId01234567890abc:> volume rename -vserver <vserver name> -volume <volume> -newname <new name of volume>

Conclusion

In this post, we walked through the process of refreshing a database environment from production into development using FSx for ONTAP features and capabilities like Snapshots, FlexClones, and SnapMirror. By leveraging these features, you can efficiently and swiftly perform refreshes while cost-optimizing storage consumption. This can help ensure that development, testing, and QA are done on copies of the most up-to-date, production-like datasets.

This post showed the example of an Oracle 19c database refresh, though the same principles of data refreshes can be applied to other databases (SAP HANA, MySQL, PostGres, Microsoft SQL, etc.), and applications which could benefit in managing multiple read/write copies of the same dataset.

For further information about FSx for ONTAP, visit the product page. To learn more about Oracle with AWS, visit the Oracle FAQ page on AWS.

TAGS: , ,
Naim Mucaj

Naim Mucaj

Naim Mucaj is Senior Solutions Architect specializing in Data and Storage Management services at AWS. Naim has over 20 years of experience in designing and building data and infrastructure solutions, solely focused on customer-centric outcomes. Aside helping customers, when not at AWS, Naim enjoys travelling and learning about new cultures.

Ramam Kallakuri

Ramam Kallakuri

Ramam Kallakuri is a Solutions Architect specializing in Oracle E-Business Suite, Oracle Hyperion at AWS. He works with customers to provide guidance and technical assistance for various E-Business Suite & Hyperion migrations, helping them improving the value of their solutions when using AWS.

Saadelden Abdelkreem

Saadelden Abdelkreem

Saadelden Abdelkreem is a solutions architect at Amazon Web Services (AWS) on the Australian public sector team. He assists public sector customers in achieving their transformation, automation, and business goals. Saadelden has worked for large telecom firms, assisting them with database and solution architecture. In his spare time, he enjoys exploring new activities and reading a book.