Best practices for upgrading Amazon RDS for Oracle DB instances from 18c to 19c

Amazon Relational Database Service (Amazon RDS) for Oracle provides newer versions of databases so you can keep your DB instances up to date. These versions can include bug fixes, security enhancements, and other improvements. When Amazon RDS for Oracle supports a new version, you can choose how and when to upgrade your DB instances. Amazon RDS for Oracle supports two types of upgrades: major version and minor version. In general, a major engine version upgrade can introduce changes that aren’t compatible with existing applications. In contrast, a minor version upgrade in Oracle generally only includes changes that are backward-compatible with existing applications. In other words, a minor version upgrade applies an Oracle Database Patch Set Update (PSU) or Release Update (RU) to a major version. For example, upgrading from 18c to 19c (19.0.0.0) is a major version upgrade, whereas going from 12.2.0.1.ru-2019-04.rur-2019-04.r1 to 12.2.0.1.ru-2019-07.rur-2019-07.r1 is considered a minor version upgrade.

Although Amazon RDS for Oracle allows the option to automate some of the patching decisions, such as minor version patching in the form of automatic minor version upgrade, the majority of patching decisions—including the major version upgrades— are up to you. Therefore, it’s critical to be aware of common issues, steps involved, and best practices to upgrade with the least impact on your business.

In this post, we look at 18c End of Support timeline in Amazon RDS for Oracle, study the version upgrade choices available to you, and dive deep into the best practices to follow during the upgrade process.

This post is relevant to database administrators running their DB instances on Amazon RDS for Oracle. Most pre-upgrade and post-upgrade steps in this post are general guidelines from Oracle support and upgrade documentation relevant to Amazon RDS for Oracle. This post is written considering the various workloads on Amazon RDS for Oracle that could be impacted by 18c to 19c DB upgrade. Not all guidelines mentioned here are relevant to all customers. We strongly recommend that you use your judgment to see which one is suitable for your database based on your options and parameters used. Prior knowledge of Oracle database administration and the environment should be used for performing the upgrade.

Amazon RDS for Oracle: 18c version End of Support timeline

Oracle has announced the end date of support for Oracle Database version 18c as June 30, 2021, after which Oracle Support will no longer release Critical Patch Updates for this database version. 18c is not eligible for Extended Support. Amazon RDS for Oracle will deprecate Oracle Database version 18c on June 30, 2021.

All 18c instances will be automatically upgraded to 19c starting on July 1, 2021. We highly recommend you upgrade your existing Amazon RDS for Oracle 18c DB instances and validate your applications before the automatic upgrades begin.

The following table summarizes the timeline for Oracle 18c.

For up-to-date information about the End of Support of the 18c version by Amazon RDS for Oracle, see Amazon RDS for Oracle – End of Support Timeline for 18c Major Version.

Refer to MOS note #742060.1 for the latest status of Oracle Database releases and support coverage. For License Included customers, please create a support case for this document with AWS.

Version upgrade choices in Amazon RDS for Oracle

Amazon RDS for Oracle supports the following major version upgrade paths (as of this writing).

As part of deciding which version to upgrade to, you have the following options:

• 12.1.0.2 – Extended Support for Oracle Database 12.1.0.2 (Terminal Patch Set) ends on July 31, 2022. Amazon RDS for Oracle plans to support Oracle Database 12.1.0.2 until July 31, 2022.
• 12.2.0.1 – Limited Error Correction Support for Oracle Database 12.2.0.1 ends on March 31, 2022. Amazon RDS for Oracle plans to support Oracle Database 12.2.0.1 until March 31, 2022.
• 19c – Premier Support for Oracle Database 19c ends on April 30, 2024, and the Extended Support ends on April 30, 2027. Amazon RDS for Oracle plans to support Oracle Database 19c until April 30, 2027.

We recommend you upgrade your 18c DB instances to 19c because it’s the long-term support release. When upgrading any software, checking for compatibility of the new version and its features plays a crucial role in the upgrade’s overall success. Oracle Database versions and releases can differ in how they work and interact with applications, which may result in compatibility issues. Although the way you interact with Amazon RDS for Oracle remains the same from the major version upgrade perspective, Oracle Database specific features across major versions may change or even become obsolete. For more information about major version upgrades, see Oracle database engine release notes.

Keep in mind the following:

• Major version downgrades aren’t supported
• A major version upgrade from 18c to 19c must upgrade to an Oracle PSU released in the same month or later

For example, a major version upgrade from Oracle version 18.0.0.0.ru-2019-10.rur-2019-10.r1 to Oracle version 19.0.0.0.ru-2020-01.rur-2021-01.r1 is supported. However, a major version upgrade from Oracle version 18.0.0.0.ru-2019-10.rur-2019-10.r1 to Oracle version 19.0.0.0.ru-2019-07.rur-2019-07.r1 isn’t supported. This is because Oracle version 18.0.0.0.ru-2019-10.rur-2019-10.r1 was released in October 2019, and Oracle version 19.0.0.0.ru-2019-07.rur-2019-07.r1 was released in July 2019. For information about the release date for each Oracle PSU, see Oracle database engine release notes.

Downtime considerations for a major version upgrade

Amazon RDS lets you manually initiate a major version upgrade by modifying the DB instance—either via the AWS Management Console, AWS Command Line Interface (AWS CLI), or Amazon RDS API. This is an in-place upgrade and requires downtime for your applications during the upgrade process. In the event of any issues with the upgrade phase, you can restore the latest backup. The duration of the outage varies based on your engine version and the DB instance class. You can determine the exact amount of time it takes by exercising a test upgrade using a snapshot restore of the production database in a pre-production environment.

If performing the major version upgrade using the modify DB instance method isn’t desirable for your application, an alternative approach is using logical bi-directional replication with either AWS Database Migration Service (AWS DMS) or Oracle GoldenGate. This method can provide the least downtime for the upgrade. This method involves setting up logical replication between the source and target DB instances (both running on the same version). You then upgrade the target DB instance to 19c, while leaving the source to run on 18c. When the upgrade of the target DB instance is complete, you can point your application to the upgraded target DB instance. This method, which uses bi-directional replication between the source and target DB instances, can also be used as a fallback plan, should the upgrade break due to incompatibility.

This post covers the upgrade process using the modify DB instance method. The logical replication method of upgrading is out of scope of this post.

How Amazon RDS for Oracle performs a major version upgrade

When a major version upgrade is invoked on the console, AWS CLI, or Amazon RDS API, the automation completes the following steps:

1. Takes a pre-upgrade snapshot (if configured for backups). You can use this snapshot to roll back to if needed.
2. Shuts down the instance and prepares it for the upgrade.
3. Runs Oracle upgrade scripts.
4. Takes a post-upgrade snapshot.

When Amazon RDS initiates Step 1, the instance’s status changes from Available to Upgrading. After Step 4, it returns to Available. The following diagram illustrates the high-level steps when the modify-db-instance AWS CLI call is invoked.

Best practices for major version upgrades

In this section, we dive deep into the recommended best practices during various phases of the upgrade process: pre-upgrade, upgrade, and post-upgrade. What we discuss in the following sections are the common steps typically completed in most Oracle Database upgrades. For a complete list, see the Oracle Database Upgrade Guide.

Pre-upgrade phase

Oracle uses the following terminology when upgrading to a higher release (or major version upgrade in Amazon RDS terminology):

• Deprecated features – Features that are no longer being enhanced, but are still supported for the full life of this release of Oracle Database.
• Desupported features – Those that are no longer supported by fixing bugs related to that feature. Oracle can choose to remove the code required to use the feature.

Where indicated, a deprecated feature can be desupported in a future major release. When moving across multiple major releases during the upgrade process, it’s highly recommended to consult Oracle documentation for deprecated and desupported features, options, and parameters in the intermediary releases as well. For more information, see Behavior Changes, Deprecated and Desupported Features for Oracle Database.

You should consider the following factors in the pre-upgrade phase.

Downtime considerations

A typical upgrade with all possible options in the option group might take 1–2 hours. To reduce downtime, consider the following:

• Take a manual snapshot using the create-db-snapshot AWS CLI call prior to starting the upgrade phase. This speeds up the time taken for a pre-upgrade snapshot (which is automatically invoked during the upgrade phase).
• When upgrading to 19c, it’s recommended to convert all DBMS_JOB to DBMS_SCHEDULER before the upgrade. During the upgrade, Oracle tries to convert the DBMS_JOB to DBMS_SCHEDULER. If you have a large number of DBMS_JOB entries, the upgrade takes longer.
• Make sure the audit trails aren’t very large. Pre-upgrade checks and upgrades can take longer with large audit trails.
• Gather optimizer statistics before the upgrade using DBMS_STATS.gather_dictionary_stats. Also gather fixed and dictionary stats.
• Remove options that aren’t used. It’s a good practice to remove unused options to speed up the upgrades and result in fewer issues and conflicts when moving from one major version to another.
• Remove or reset parameters that aren’t used for the same preceding reasons.
• If necessary, you might also have to upgrade the instance class based on the target instance choice. For more information, see RDS for Oracle instance classes.

Multi-AZ considerations

If your DB instance is in a Multi-AZ deployment, Amazon RDS for Oracle upgrades both the primary and Multi-AZ standby simultaneously.

Option groups considerations

Consider the following for option groups:

• By default, when one engine is upgraded to the next higher major version, Amazon RDS for Oracle chooses the default option group if the new corresponding custom option group for the target version isn’t chosen. For example, when upgrading from 18c to 19c, you should create a new option group that’s compatible with the new version 19c and matches closest to the 18c options, and use it during upgrade process. You should also consider factors such as APEX upgrade, which is recommended to be handled as a separate preparatory process, to speed up the upgrade. This also applies to OEM agents. In some cases, options are uninstalled and reinstalled as you move from one version to another. For example, SQLT is freshly installed, which deletes any old metadata stored by the option.
• When choosing the version of the OEM agent, consider the compatibility with the OMS.
• Note the option group and the VPC. If a DB instance is in a VPC, the option group associated with the instance is linked to that VPC. This means that you can’t use the option group assigned to a DB instance if you try to restore the instance to a different VPC or a different platform (for example, when you use the OEM option). When you upgrade a major version, understand the changes Oracle makes to the options in that change. For example, Oracle Multimedia is deprecated as of the Oracle Database 18c release and is desupported in 19c. However, it separated the MDSYS functionality into a separate option called Locator. Oracle also recommends you move from Locator to Spatial and Graph prior to the upgrade. All MDSYS functionality is available with Spatial and Graph. So, when you upgrade to 19c and install Spatial and Graph, all stored procedures work fine. For more information, see “My Oracle support” article 2347372.1. For License Included customers, please create a support case for this document with AWS.
• Starting in Oracle Database 19c, the SQL*Plus table PRODUCT_USER_PROFILE (PUP table) is desupported. The SQL*Plus product-level security feature is unavailable in Oracle Database 19c. Oracle recommends that you protect data by using Oracle Database settings, so that you ensure consistent security across all client applications.

Parameter groups considerations

Consider the following regarding parameter groups:

• If you associate a new parameter group with a DB instance, reboot the database after the upgrade completes. If you need to reboot the instance to apply the parameter group changes, the instance’s parameter group status shows pending-reboot. You can view an instance’s parameter group status on the console or by using a describe command, such as describe-db-instances.
• The continuous_mine functionality of the LogMiner package DBMS_LOGMNR.START_LOGMNR is obsolete. It was deprecated in Oracle Database 12c Release 2 (12.2). There is no replacement functionality. Oracle didn’t provide any alternative to this. If you use this, you need to address it in a different way, such as AWS DMS or Oracle GoldenGate.
• The apply parameter OPTIMIZE_PROGRESS_TABLE for Oracle GoldenGate Integrated Replicat, XStream In, and Logical Standby, is desupported in Oracle Database 19c. Before you upgrade to Oracle Database 19, you must turn off this parameter. If OPTIMIZE_PROGRESS_TABLE is set to ON, then stop apply gracefully, turn off the parameter, and restart apply. For GoldenGate Apply and XStream, this parameter is set to OFF by default.

Security considerations

The following are some important security considerations:

• By default, Oracle accounts that have not had their passwords reset before upgrade (that are set to EXPIRED status), and that are also set to LOCKED status, are set to NO AUTHENTICATION after the upgrade is complete. Therefore, post-upgrade, any account with a password set to default, locked, and expired loses the authentication method. Although it could be reverted back, it’s recommended to validate the password for its strength before the upgrade and lock it.
• Check the accounts that use the case-insensitive password version. Log in to SQL*Plus as an administrative user, and enter the following SQL query. If there are any 10g versions, you should refer to the Oracle documentation to fix 10g versions, or user accounts with LOCKED after the upgrade is complete:

  SELECT USERNAME, PASSWORD_VERSIONS FROM DBA_USERS;

• Make sure that you don’t have the deprecated parameter SEC_CASE_SENSITIVE_LOGON set to FALSE.

General considerations

Finally, you should consider the following general points:

• When upgrading to 19c, we recommend you look at the provisioned capacity and see if it would be met. Depending on the instance class your 18c database is running on, you may need to scale the compute to a newer generation of the instance class. For more information, see RDS for Oracle instance classes. This could also be an opportunity to evaluate Amazon RDS reserved instances, and you may want to purchase new leases for your version before performing the upgrade.
• Make sure that no objects are invalid before upgrading. It’s a good practice to keep a list of all objects with their respective count, type, and status prior to the upgrade and compare it after the upgrade.
• Take a backup of optimizer statistics for all application schemas using the DBMS_STATS package.
• Collect pre-upgrade AWR or Statspack snapshots prior to the upgrade. These are used during the post-upgrade performance validation. Even though you would have done this in the pre-production environment, it’s a good practice to do this comparison in the production.
• If you also have a DB maintenance task, such as adding partitions, it’s better to run them before upgrading.
• Disable scheduled database custom jobs or cron jobs.
• It’s recommended to disable any custom DDL triggers before upgrade and re-enable them after upgrade.
• If you’re using materialized views (MV), check the status of all MVs prior to the upgrade. Check the size of the MV logs. If it’s non-zero, address them to make sure they are all in sync. It’s recommended to wait until all MVs have completed refreshing. Make sure that any objects referencing remote databases across database links are accessible to reduce time spent on network timeouts. For more information, see “My Oracle support” Doc ID 1406586.1. For License Included customers, please create a support case for this document with AWS.
• Keep a list of clients and its drivers used along with the version numbers. Identify the corresponding version that works with 19c.
• Review all hidden parameters and make sure they’re modified or removed to meet the target version. Include all features affected by these parameters as part of the functional and performance testing in the pre-production environment.

Upgrade phase

After you complete the pre-upgrade checks successfully, you can move on to the upgrade phase. If your instance has been using custom option group or parameter group configurations, you must specify new option and parameter groups for the upgrade to go through along with any other additional attributes.

In this section, we discuss how to perform the upgrade on the console or via the AWS CLI.

To upgrade the engine version of a DB instance on the console, complete the following steps:

1. On the Amazon RDS console, choose Databases.
2. Choose the DB instance that you want to upgrade.
3. Choose Modify.The Modify DB Instance page appears.
4. For License model, choose the desired license type.
5. For DB engine version, choose the new version.
   
6. Choose Continue and check the summary of modifications.
7. To apply the changes immediately, choose Apply immediately.Choosing this option can cause an outage in some cases. For more information, see Using the Apply Immediately setting. Otherwise, you can have the upgrade take place in your next weekly maintenance window.
8. On the confirmation page, review your changes and choose Modify DB Instance to save your changes.

To perform a major version upgrade in your next maintenance window using the AWS CLI modify-db-instance command, enter the following code:

aws rds modify-db-instance \
    --db-instance-identifier mydbinstance \
    --engine-version new_version \
    --allow-major-version-upgrade \
    --no-apply-immediately

For information about valid engine versions, use the AWS CLI describe-db-engine-versions command.

If you have multiple instances, you could use the AWS CLI combined with a shell script to upgrade multiple instances. For more information about upgrading the database engine for Amazon RDS for Oracle, see Upgrading the Oracle DB engine.

Post-upgrade phase

After the upgrade phase, you can complete the following checks and post-upgrade steps:

1. Recompile INVALID As mentioned in the pre-upgrade phase section, checking for objects with INVALID status and fixing them is required before testing further. This is also a good time to compare the report (list of object count with status) gathered during the pre-upgrade phase.
2. If Amazon RDS for Oracle is used as an RMAN catalog, upgrade the RMAN recovery catalog. Use the UPGRADE CATALOG command to upgrade the RMAN recovery catalog schema from an older version to the version required by the RMAN client.
3. Complete the following client connectivity steps:
   • Make sure to set the ORACLE_HOME, PATH, LD_LIBRARY_PATH, LIBPATH, and more to 19.x in the client home.
   • Make appropriate changes to tnsnames.ora and sqlnet.ora on the client side and incorporate changes related to 19c or the new endpoint if testing on a new RDS instance restored from the production.
   • Test the client connectivity. Based on the identified target version of the driver, perform a regression test. Keep in mind there are cases where the choice of JDBC version might impact performance and change the process optimizer plan.
4. If you used the DBSM_STATS package to export the optimizer stats as mentioned in the pre-upgrade tasks, upgrade the stats table (using DBMS_STATS.UPGRADE_STAT_TABLE) where the optimizer statistics are backed up prior to the upgrade. Then restore statistics and run performance testing.
5. If you have database links to other RDS for Oracle DB instances or other Oracle instances on Amazon Elastic Compute Cloud (Amazon EC2), it’s recommended to test them from both a functionality and performance standpoint after the upgrade.
6. If you have any external scheduler jobs scheduled in Amazon CloudWatch or cron, ensure those are enabled and can still connect to the database after upgrade and run with no issues.
7. Also make sure to enable any DDL or system triggers disabled as part of the pre-upgrade check.
8. Make sure to have enough free space in SYSTEM and SYSAUX tablespaces. Having an additional 1–2 GB of space in each tablespace is recommended.
9. Validate your applications against the upgraded database. It’s vital that you look at the top SQL statements (either via AWR or Statspack) and verify that they are using the desired plans.
10. After the upgrade, if the SQL statements perform poorly due to the change in the plans by the 19c optimizer, you can use OPTIMIZER_FEATURES_ENABLE. This parameter is alterable at the session level and system level. For example, if you upgrade your database from release 18c to release 19c, but you want to keep the release 18c optimizer behavior, you can do so by setting this parameter to 18c. At a later time, you can try the enhancements introduced in releases up to and including release 19c by setting the parameter to 19.0.0.
11. If you have an automated process of backup and restore, disaster recovery, or production to non-production refresh procedures, we strongly recommend you test them. You need to test any process that interacts with other databases that may or may not be at the same level.

Common issues

We highly recommend you upgrade the manual snapshots you intend to retain longer, including the snapshots taken using AWS Backup or any third-party partner products. For example, if you used AWS Backup and the backup was migrated to an Amazon Elastic File System (Amazon EFS) file system based on the lifecycle defined in AWS Backup, when the backup is restored, the first thing the RDS does is force upgrade it to one of the supported versions and then complete the restore. If you want to keep it as a copy of the old version (for example, if the database is supporting an old version of a third-party application) you should plan on making a backup using native Oracle tools such as export data pump or RMAN. Also keep the version incompatibilities in mind.

After you upgrade the database, if you run into connectivity issues such as "ORA-28040: No matching authentication protocol", you might have to upgrade the client version or the JDBC driver. In general, it’s best to match the client and server versions. The best client version for the database version 19c is client 19c. However, the 18c clients are also supported on 19c. We highly recommend testing for this combination. Refer to “My Oracle support” document 207303.1 for more information with AWS.

Test the upgrade of your RDS for Oracle DB instance

Before you upgrade the production database to a new major version, you must validate your applications against the new version in a pre-production environment. This rehearsal of the upgrade process also allows you to gauge the time it takes to perform this major version upgrade.

The following diagram illustrates the steps to test your upgrade.

The process includes the following steps:

1. Take a snapshot of the existing DB instance using the create-db-snapshot AWS CLI call.
2. Restore the DB snapshot using the restore-db-instance-from-db-snapshot AWS CLI call to create a test DB instance on the same version.
3. Run the modify-db-instance AWS CLI call on the test DB instance to upgrade it to the new major version.
4. When the upgrade is complete, you can validate your application against the test DB instance.
5. After the testing is complete, you can schedule and perform the production database upgrade.

Conclusion

In this post, we reviewed the 18c End of Support timeline in Amazon RDS for Oracle, studied the version upgrade choices available to you, and covered the best practices during the phases of the upgrade process.

If you need additional assistance in planning and upgrading your Amazon RDS for Oracle DB instances running on 18c, you can reach out to the following resources:

• AWS Professional Services
• AWS Partners

For more information about the upgrade, see the following:

• Upgrading the Oracle DB engine
• Oracle SE2 upgrade paths
• Deprecation of Oracle 18c

About the authors

Yamuna Palasamudram is a Senior Database Specialist Solutions Architect with Amazon Web Services. She works with AWS RDS team, focusing on commercial database engines like Oracle. She enjoys working with customers to help design, deploy, and optimize relational database workloads on AWS.

Sundar Raghavan is a Senior Specialist Solutions Architect at Amazon Web Services.