Best practices for upgrading Amazon RDS for Oracle DB instances from 11.2.0.4 to 19c

Amazon Relational Database Service (Amazon RDS) for Oracle provides newer versions of databases so you can keep your DB instances up-to-date. These versions can include bug fixes, security enhancements, and other improvements. When Amazon RDS for Oracle supports a new version, you can choose how and when to upgrade your DB instances. As you may be already aware, Amazon RDS for Oracle supports two types of upgrades: major version and minor version. In general, a major engine version upgrade can introduce changes that aren’t compatible with existing applications. In contrast, a minor version upgrade in Oracle generally only includes changes that are backward-compatible with existing applications. In other words, a minor version upgrade applies an Oracle Database Patch Set Update (PSU) or Release Update (RU) to a major version. For example, upgrading from 11.2.0.4 to 19c (19.0.0.0) is a major version upgrade, whereas going from 12.2.0.1.ru-2019-04.rur-2019-04.r1 to 12.2.0.1.ru-2019-07.rur-2019-07.r1 is considered a minor version upgrade.

Although Amazon RDS for Oracle allows the option to automate some of the patching decisions, such as minor version patching in the form of automatic minor version upgrade, the majority of patching decisions—including the major version upgrades— are up to you. Therefore, it’s critical to be aware of common issues, steps involved, and best practices to upgrade with the least impact on your business.

In this post, we a look at the 11.2.0.4 End of Support timeline in Amazon RDS for Oracle, study the version upgrade choices available to you, and dive deep into the best practices to follow during the upgrade process.

This post is relevant to database administrators running their DB instances on Amazon RDS for Oracle. Most pre- and post-upgrade steps in this post are general guidelines from Oracle support and upgrade documentation that are relevant to Amazon RDS for Oracle. This post is written considering the various workloads on Amazon RDS for Oracle that could be impacted by 11.2.0.4 DB upgrade. Not all guidelines mentioned here are relevant to all customers. We strongly recommend that you use your judgment to see which one is suitable for your database based on your options and parameters used. Prior knowledge of Oracle database administration and the environment should be used for performing the upgrade.

Amazon RDS for Oracle: 11.2.0.4 version End of Support timeline

Oracle has announced the end date of support for Oracle Database version 11.2.0.4 as December 31, 2020, after which Oracle Support will no longer release Critical Patch Updates for this database version. Amazon RDS for Oracle will end support for Oracle Database version 11.2.0.4 Standard Edition 1 (SE1) for License Included (LI) model on October 31, 2020. For the Bring Your Own License (BYOL) model, Amazon RDS for Oracle will end the support for Oracle Database version 11.2.0.4 for all editions on December 31, 2020.

All 11.2.0.4 SE1 LI instances will be automatically upgraded to 19c starting on November 1, 2020. Likewise, the 11.2.0.4 BYOL instances will be automatically upgraded to 19c starting on January 1, 2021. We highly recommend you upgrade your existing Amazon RDS for Oracle 11.2.0.4 DB instances and validate your applications before the automatic upgrades begin.

The following table summarizes the timeline for Oracle 11.2.0.4 with SE1 with LI.

The following table summarizes the timeline for Oracle 11.2.0.4 on SE, SE1, and EE with BYOL.

For more information about the EOS of the 11.2.0.4 version by Amazon RDS for Oracle, see Announcement: Amazon RDS for Oracle – End of Support Timeline for 12.2.0.1 and 11.2.0.4 Major Versions.

Version upgrade choices in Amazon RDS for Oracle

Amazon RDS for Oracle supports the following major version upgrade paths (as of this writing).

As part of deciding which version to upgrade to, you have the following options:

• 12.1.0.2 –Extended Support for Oracle Database 12.1.0.2 (Terminal Patch Set) ends on July 31, 2022. Amazon RDS for Oracle plans to support Oracle Database 12.1.0.2 until July 31, 2022.
• 12.2.0.1 –Limited Error Correction Support for Oracle Database 12.2.0.1 ends on March 31, 2022. Amazon RDS for Oracle plans to support Oracle Database 12.2.0.1 until March 31, 2022.
• 18c – Support for Oracle Database 18c ends on June 8, 2021. Amazon RDS for Oracle plans to support Oracle Database 18c until June 8, 2021. (There is no Extended Support for Oracle Database 18c.)
• 19c – Premier Support for Oracle Database 19c ends on April 30, 2024, and the Extended Support ends on April 30, 2027. Amazon RDS for Oracle plans to support Oracle Database 19c until April 30, 2027.

We recommend you upgrade your 11.2.0.4 DB instances to 19c because it’s the long-term support release.

When upgrading any software, checking for compatibility of the new version and its features plays a crucial role in the upgrade’s overall success. Oracle Database versions and releases can have differences in how they work and interact with applications, which may result in compatibility issues. Although the way you interact with Amazon RDS for Oracle remains the same from the major version upgrade perspective, Oracle Database specific features across major versions may change or even become obsolete. For more information about major version upgrades, see Oracle Database Engine Release Notes.

Keep in mind the following:

• Major version downgrades aren’t supported.
• A major version upgrade from 11g to 12c must upgrade to an Oracle Patch Set Update (PSU) released in the same month or later.

For example, a major version upgrade from Oracle version 11.2.0.4.v14 to Oracle version 12.1.0.2.v11 is supported. However, a major version upgrade from Oracle version 11.2.0.4.v14 to Oracle version 12.1.0.2.v9 isn’t supported. This is because Oracle version 11.2.0.4.v14 was released in October 2017, and Oracle version 12.1.0.2.v9 was released in July 2017. For information about the release date for each Oracle PSU, see Oracle Database Engine Release Notes.

Downtime considerations for a major version upgrade

Amazon RDS lets you manually initiate a major version upgrade by modifying the DB instance—either via the AWS Management Console, AWS Command Line Interface (AWS CLI), or Amazon RDS API. This is an in-place upgrade and requires downtime for your applications during the upgrade process. In the event of any issues with the upgrade phase, you can restore the latest backup. The duration of the outage varies based on your engine version and the DB instance class. You can determine the exact amount of time it takes by exercising a test upgrade using a snapshot restore of the production database in a pre-production environment.

If performing the major version upgrade using the modify DB instance method isn’t desirable for your application, an alternative approach is using logical bi-directional replication with either AWS Database Migration Service (AWS DMS) or Oracle GoldenGate. This method can provide the least downtime for the upgrade. This method involves setting up logical replication between the source and target DB instances (both running on the same version). You then upgrade the target DB instance to 19c, while leaving the source to run on 11.2.0.4. When the upgrade of the target DB instance is complete, you can point your application to the upgraded target DB instance. This method, which uses bi-directional replication between the source and target DB instances, can also be used as a fallback plan, should the upgrade break due to incompatibility.

This post covers the upgrade process using the modify DB instance method. The logical replication method of upgrading is out of scope of this post.

How Amazon RDS for Oracle performs a major version upgrade

When a major version upgrade is invoked on the console, AWS CLI, or Amazon RDS API, the automation completes the following steps:

1. Takes a pre-upgrade snapshot (if configured for backups). You can use this snapshot to roll back to if needed.
2. Shuts down the instance and prepares it for the upgrade.
3. Runs Oracle upgrade scripts.
4. Takes a post-upgrade snapshot.

When Amazon RDS initiates Step 1, the instance’s status changes from Available to Upgrading. After Step 4, it returns to Available. The following diagram illustrates the high-level steps when the modify-db-instance AWS CLI call is invoked.

Best practices for major version upgrades

In this section, we dive deep into the recommended best practices during various phases of the upgrade process: pre-upgrade, upgrade, and post-upgrade.

What we discuss in the following sections are the common steps typically completed in most Oracle Database upgrades. For a complete list, see the Oracle Database Upgrade Guide.

Pre-upgrade phase

Oracle uses the following terminology when upgrading to a higher release (or major version upgrade in Amazon RDS terminology):

• Deprecated features – Features that are no longer being enhanced, but are still supported for the full life of this release of Oracle Database.
• Desupported features – Those that are no longer supported by fixing bugs related to that feature. Oracle can choose to remove the code required to use the feature.

Where indicated, a deprecated feature can be desupported in a future major release. When moving across multiple major releases during the upgrade process, it’s highly recommended to consult Oracle documentation for deprecated and desupported features, options, and parameters in the intermediary releases as well. For more information, see Behavior Changes, Deprecated and Desupported Features for Oracle Database.

You should consider the following factors in the pre-upgrade phase.

Downtime considerations

A typical upgrade with all possible options in the option group might take 1–2 hours. To reduce downtime, consider the following:

• Take a manual snapshot using the create-db-snapshot AWS CLI prior to starting the upgrade phase. This speeds up the time taken for a pre-upgrade snapshot (which is automatically invoked during the upgrade phase).
• If you’re upgrading to 19c, it’s recommended to convert all DBMS_JOB to DBMS_SCHEDULER before the upgrade. During the upgrade, Oracle tries to convert the DBMS_JOB to DBMS_SCHEDULER. If you have a large number of DBMS_JOB entries, the upgrade takes longer.
• Make sure the audit trails aren’t very large. Pre-upgrade checks and upgrades can take longer with large audit trails.
• Gather optimizer statistics before the upgrade using DBMS_STATS.gather_dictionary_stats. Also gather fixed and dictionary stats.
• Remove options that aren’t used. It’s a good practice to remove unused options to speed up the upgrades and result in fewer issues and conflicts when moving from one major version to another.
• Remove or reset parameters that aren’t used for the same preceding reasons.
• If necessary, you might also have to upgrade the instance class based on the target instance choice. For more information, see DB Instance Class Support for Oracle.

Multi-AZ considerations

If your DB instance is in a Multi-AZ deployment, Amazon RDS for Oracle upgrades both the primary and Multi-AZ standby simultaneously.

Option groups considerations

Consider the following for option groups:

• By default, when one engine is upgraded to the next higher major version, Amazon RDS for Oracle chooses the default option group if the new corresponding custom option group for the target version isn’t chosen. For example, when upgrading from 11.2.0.4 to 19c, you should create a new option group that’s compatible with the new version 19c and matches closest to the 11.2.0.4 options, and use it during upgrade process. You should also consider factors such as APEX upgrade, which is recommended to be handled as a separate preparatory process, to speed up the upgrade. This also applies to OEM agents. In some cases, options are uninstalled and reinstalled as you move from one version to another. For example, SQLT is freshly installed, which deletes any old metadata stored by the option.
• When choosing the version of the OEM agent, consider the compatibility with the OMS.
• Note the option group and the VPC. If a DB instance is in a VPC, the option group associated with the instance is linked to that VPC. This means that you can’t use the option group assigned to a DB instance if you try to restore the instance to a different VPC or a different platform (for example, when you use the OEM option).
• When you upgrade a major version, understand the changes Oracle makes to the options in that change. For example, Oracle has desupported the Multimedia option in 19c. However, it separated the MDSYS functionality into a separate option called Locator. Oracle also recommends you move from Locator to Spatial and Graph prior to the upgrade. All MDSYS functionality is available with Spatial and Graph. So, when you upgrade to 19c and install Spatial and Graph, all stored procedures work fine. For more information, see “My Oracle support” article 2347372.1 for more details.
• The PL/SQL package DBMS_XMLQUERY is deprecated in Oracle Database 18c. Oracle recommends using DBMS_XMLGEN instead.

Parameter groups considerations

If you associate a new parameter group with a DB instance, reboot the database after the upgrade completes. If you need to reboot the instance to apply the parameter group changes, the instance’s parameter group status shows pending-reboot. You can view an instance’s parameter group status on the console or by using a describe command, such as describe-db-instances.

The continuous_mine functionality of the LogMiner package DBMS_LOGMNR.START_LOGMNR is obsolete. It was deprecated in Oracle Database 12c Release 2 (12.2). There is no replacement functionality. Oracle didn’t provide any alternative to this. If you use this, you need to address it in a different way, such as AWS DMS or Oracle GoldenGate.

Security considerations

The following are some important security considerations:

• By default, Oracle accounts that have not had their passwords reset before upgrade (that are set to EXPIRED status), and that are also set to LOCKED status, are set to NO AUTHENTICATION after the upgrade is complete. Therefore, post-upgrade, any account with a password set to default, locked, and expired loses the authentication method. Although it could be reverted back, it’s recommended to validate the password for its strength before the upgrade and lock it.
• Check the accounts that use the case-insensitive password version. Log in to SQL*Plus as an administrative user, and enter the following SQL query. If there are any 10g versions, you should refer to the Oracle documentation to fix 10g versions, or user accounts with LOCKED after the upgrade is complete:

  SELECT USERNAME, PASSWORD_VERSIONS FROM DBA_USERS;

• Make sure that you don’t have the deprecated parameter SEC_CASE_SENSITIVE_LOGON set to FALSE.
• Starting with Oracle 12c, Oracle uses Oracle Real application Security (ORAS) to store Oracle security policies. Before you upgrade an 11g database to Oracle Database 19c, you must delete any data security roles defined in 11g. After the upgrade, you may have to define the data security roles again. If it’s upgraded without taking this action, any data security policy that includes a data security role becomes invalid in the new 19c database.

General considerations

Finally, you should consider the following general points:

• When upgrading to 19c, we recommend you look at the provisioned capacity and see if it would be met. Depending on the instance class your 11g database is running on, you may need to scale the compute to a newer generation of the instance class. For more information, see DB Instance Class Support for Oracle. This could also be an opportunity to evaluate Amazon RDS reserved instances and you may want to purchase new leases for your version before performing the upgrade.
• Make sure that no objects are invalid before upgrading. It’s a good practice to keep a list of all objects with their respective count, type, and the status prior to the upgrade and compare it after the upgrade.
• Take a backup of optimizer statistics for all application schemas using the DBMS_STATS package.
• Collect pre-upgrade AWR or Statspack snapshots prior to the upgrade. These are used during the post-upgrade performance validation. Even though you would have done this in the pre-production environment, it’s a good practice to do this comparison in the production.
• If you also have a DB maintenance task, such as adding partitions, it’s better to run them before upgrading.
• Disable scheduled database custom jobs or cron jobs.
• It’s recommended to disable any custom DDL triggers before upgrade and re-enable them after upgrade.
• If you’re using materialized views (MV), check the status of all MVs prior to the upgrade. Check the size of the MV logs. If it’s non-zero, address them to make sure they are all in sync. It’s recommended to wait until all MVs have completed refreshing. Make sure that any objects referencing remote databases across database links are accessible to reduce time spent on network timeouts. For more information, see “My Oracle support” Doc ID 1406586.1. For License Included customers, please create a support case for this document.
• Keep a list of clients and its drivers used along with the version numbers. Identify the corresponding version that works with 19c.
• Review all hidden parameters and make sure they’re modified or removed to meet the target version. Include all features affected by these parameters as part of the functional and performance testing in the pre-production environment.

Upgrade phase

After completing the pre-upgrade checks successfully, you can move on to the upgrade phase. If your instance has been using custom option group or parameter group configurations, you must specify new option and parameter groups for the upgrade to go through along with any other additional attributes.

When you upgrade your 11.2.0.4 SE1 DB instance to 19c, by default it takes you to SE2 on 19c.

In this section, we discuss how to perform the upgrade on the console or via the AWS CLI.

To upgrade the engine version of a DB instance on the console, complete the following steps:

1. On the Amazon RDS console, choose Databases.
2. Choose the DB instance that you want to upgrade.
3. Choose Modify.
   The Modify DB Instance page appears.
4. For License model, choose the desired license type. For DB engine version, choose the new version.
   
5. Choose Continue and check the summary of modifications.
6. To apply the changes immediately, choose Apply immediately.
   Choosing this option can cause an outage in some cases. For more information, see Using the Apply Immediately Setting. Otherwise, you can have the upgrade take place in your next weekly maintenance window.
7. On the confirmation page, review your changes and choose Modify DB Instance to save your changes.

To perform a major version upgrade in your next maintenance window using the AWS CLI modify-db-instance command, enter the following code:

aws rds modify-db-instance \
    --db-instance-identifier mydbinstance \
    --engine-version new_version \
    --allow-major-version-upgrade \
    --no-apply-immediately

For information about valid engine versions, use the AWS CLI describe-db-engine-versions command.

If you have multiple instances, you could use the AWS CLI combined with a shell script to upgrade multiple instances. For more information about upgrading the database engine for Amazon RDS for Oracle, see Upgrading the Oracle DB Engine.

Post-upgrade phase

After the upgrade phase, you can complete the following checks and post-upgrade steps:

1. Recompile the INVALID As mentioned in the pre-upgrade phase section, checking for objects with INVALID status and fixing them is required before testing further. This is also a good time to compare the report/list (of object count with status) gathered during the pre-upgrade phase.
2. If Amazon RDS for Oracle is used as an RMAN catalog, upgrade the RMAN recovery catalog. Use the UPGRADE CATALOG command to upgrade the RMAN recovery catalog schema from an older version to the version required by the RMAN client.
3. Complete the following client connectivity steps:
   • A) Make sure to set the ORACLE_HOME, PATH, LD_LIBRARY_PATH, LIBPATH, and more to 19.x in the client home.
   • B) Make appropriate changes to the tnsnames.ora and sqlnet.ora on the client side and incorporate changes related to the 19c or the new endpoint if testing on a new RDS instance restored from the production.
   • C) Test the client connectivity. Based on the identified target version of the driver, perform a regression test. Keep in mind there are cases where the choice of JDBC version might impact performance and change the process optimizer plan.
4. If you used the DBMS_STATS package to export the optimizer stats as mentioned in the pre-upgrade tasks, upgrade the stats table (using DBMS_STATS.UPGRADE_STAT_TABLE) where the optimizer statistics are backed up prior to the upgrade. Then restore statistics and run performance testing.
5. If you have database links to other Amazon RDS for Oracle DB instances or other Oracle instances on Amazon Elastic Compute Cloud (Amazon EC2), it’s recommended to test them from both a functionality and performance standpoint after the upgrade.
6. If you have any external scheduler jobs scheduled in Amazon CloudWatch or cron, ensure those are enabled and can still connect to the database after upgrade and run with no issues.
7. Also make sure to enable any DDL or system triggers disabled as part of the pre-upgrade check.
8. Make sure to have enough free space in SYSTEM and SYSAUX table space. Having an additional 1–2 GB of space in each tablespace is recommended.
9. Validate your applications against the upgraded database. It’s vital that you look at the top SQL statements (either via AWR or Statspack) and verify that they are using the desired plans.
10. After the upgrade, if the SQL statements perform poorly due to the change in the plans by the 19c optimizer, you can use the OPTIMIZER_FEATURES_ENABLE This parameter is alterable at the session level and system level. For example, if you upgrade your database from release 11.2 to release 19c, but you want to keep the release 11.2 optimizer behavior, you can do so by setting this parameter to 11.2.0.4. At a later time, you can try the enhancements introduced in releases up to and including release 19c by setting the parameter to 19.0.0.
11. If you have an automated process of backup and restore, disaster recovery, or production to non-production refresh procedures, we strongly recommend you test them. You need to test any process that interacts with other databases that may or may not be at the same level.

Common issues

We highly recommend you upgrade the manual snapshots you intend to retain longer, including the snapshots taken using AWS Backup or any third-party partner products. For example, if you used AWS Backup, when the backup is restored, the first thing the RDS does is force upgrade it to one of the supported versions and then complete the restore. If you want to keep it as a copy of the old version (for example, if the database is supporting an old version of a third-party application) you should plan on making a backup using native Oracle tools such as export data pump or RMAN. Also keep the version incompatibilities in mind.

After upgrading the database, if you run into connectivity issues such as "ORA-28040: No matching authentication protocol", you might have to upgrade the client version or the JDBC driver. In general, it’s best to match the client and server versions. The best client version for the database version 19c would be client 19c. However, the 11.2.0.4 clients are also supported on 19c. We highly recommend testing for this combination. Please refer to “My Oracle support” document 207303.1.

Testing the upgrade of your Amazon RDS for Oracle DB instance

Before upgrading the production database to a new major version, you must validate your applications against the new version in a pre-production environment. This rehearsal of the upgrade process also allows you to gauge the time it takes to perform this major version upgrade.

The following diagram illustrates the steps to test your upgrade.

The process includes the following steps:

1. Take a snapshot of the existing DB instance using the create-db-snapshot AWS CLI call.
2. Restore the DB snapshot using the restore-db-instance-from-db-snapshot AWS CLI call to create a test DB instance on the same version.
3. Run the modify-db-instance AWS CLI call on the test DB instance to upgrade it to the new major version.
4. When the upgrade is complete, you can validate your application against the test DB instance.
5. After the testing is complete, you can schedule and perform the production database upgrade.

Additional considerations

Amazon RDS for Oracle supports the read replicas feature for the versions 12.1.0.2 and above. If you upgrade your database from 11.2.0.4 to a higher version, you may want to evaluate the read replicas capability and determine if your application requires cross-Region disaster recovery and read scalability. For more information, see Managed disaster recovery and managed reader farm with Amazon RDS for Oracle using Oracle Active Data Guard.

As part of upgrading to 19c, you can also use options such as OLAP and Oracle Label security.

Conclusion

In this post, we reviewed the 11.2.0.4 End of Support timeline in Amazon RDS for Oracle, studied the version upgrade choices available to you, and covered the best practices during the phases of the upgrade process.

If you need additional assistance in planning and upgrading your Amazon RDS for Oracle DB instances running on 11.2.0.4, you can reach out to the following resources:

• AWS Professional Services
• AWS Partners

For more information about the upgrade, see the following:

• Upgrading the Oracle DB Engine
• Oracle SE2 Upgrade Paths
• Deprecation of Oracle 11.2.0.4

About the Authors

Sundar Raghavan is a Senior Specialist Solutions Architect at Amazon Web Services.

Srinagesh Battula is a Principal Product Manager at Amazon Web Services