Implement high availability for Oracle Enterprise Manager in the AWS Cloud

As you migrate Oracle workloads to AWS, you may want to implement Oracle Enterprise Manager (OEM) Cloud Control in AWS. OEM is Oracle’s management platform, which provides a single pane of glass for managing Oracle environments.

In this post, we provide two solutions to implement OEM high availability in AWS. The first solution is Oracle’s OEM Level 3 high availability architecture, which provides high availability at the application and database tiers. To implement this solution, you need a F5 BIG-IP load balancer. OEM must be configured to integrate with F5 BIG-IP Local Traffic Manager (LTM). For this post, we refer to this solution as the traditional OEM Level 3 HA AWS architecture.

The second solution for OEM high availability includes many benefits in Oracle’s OEM Level 2 and Level 3 high availability architectures. These benefits include handling loss of network connectivity, loss of system power, software issues on the physical host, hardware issues on the physical host that impact network reachability, providing multiple OEM Oracle Management Servers (OMSs), and providing flashback database capabilities.

The second solution doesn’t meet all of Oracle’s OEM Level 3 requirements. Specifically, it doesn’t provide multiple OEM OMSs for OEM agent uploads. However, it does provide a dedicated OEM OMS for OEM agent uploads. The second solution uses native AWS services such as Amazon Elastic Compute Cloud (Amazon EC2) instances, Amazon Elastic Block Storage (Amazon EBS), Amazon Elastic File System (Amazon EFS), and Network Load Balancer (NLB). We refer to this solution as the AWS native OEM HA architecture.

This post includes architecture diagrams and considerations for implementing both solutions. Some components in the first solution (traditional OEM Level 3 HA AWS architecture), including Oracle Real Application Clusters (RAC) and F5 BIG-IP, aren’t covered in this post.

For more information about installing OEM, adding additional OEM OMSs, and deploying OEM management agents, see Cloud Control Advanced Installation and Configuration Guide.

This post explains why the AWS native OEM HA architecture is preferred in AWS. This solution is less complex, has a faster implementation time, and doesn’t rely on AWS third-party solutions to provide OEM high availability.

Integrate Amazon RDS for Oracle with OEM Cloud Control

You can monitor your Amazon Relational Database Service (Amazon RDS) for Oracle instances using OEM.

Amazon RDS for Oracle supports OEM version 13.5. For more information, see Oracle Management Agent for Enterprise Manager Cloud Control.

OEM high availability architecture overview

The outage of any OEM service impacts the monitoring of all OEM targets. These targets can include Oracle databases, Oracle middleware, and EC2 instances.

Customers can have more than 25 concurrent OEM user sessions, more than 100 OEM agents, or more than 1,000 OEM targets. For environments that meet any of these criteria, Oracle recommends a medium or large OEM configuration, which requires high availability.

Customers can easily exceed 1,000 OEM targets. As a result, you may undersize your OEM environment and experience performance problems. A single Oracle database, OEM environment, and Oracle E-Business Suite environment contains more targets than you might anticipate.

The following screenshot shows that a single Amazon RDS for Oracle instance has six OEM targets.

An OEM environment has at least 40 OEM targets. Additionally, a single Oracle E-Business Suite environment with two web tiers, two application tiers, one E-Business Suite Concurrent Manager tier, and two database hosts (primary and standby databases) has at least 100 OEM targets.

OEM licensing

Oracle provides a Restricted Use License for OEM to run on EC2 instances. Based on Oracle’s OEM sizing guide, you can establish an evaluation or small OEM installation and use Oracle’s Restricted Use Licenses.

However, if you use the traditional OEM Level 3 HA AWS architecture, the Restricted Use License for OEM may not be compliant. Based on Oracle’s recommended sizing, you may need to purchase Oracle Enterprise Edition Database and Oracle RAC licenses. Additional Oracle database licenses may be required due to the implementation of an Oracle Data Guard environment. Oracle RAC licenses may be required because the OEM database repository is deployed on Oracle RAC.

You should send database backups directly to Amazon Simple Storage Service (Amazon S3) via the Oracle Secure Backup (OSB) Module for Amazon S3. OSB should be implemented with both OEM solutions, which may require an Oracle license.

You are responsible for making sure you have the proper Oracle licenses to run your workloads.

Traditional OEM Level 3 HA AWS architecture

For OEM medium and large environments, Oracle recommends the following components:

• Two load balancers (F5 BIG-IP load balancer is recommended)
• Two Oracle Management Service (OMS) servers (application tier)
• One Oracle RAC two-node cluster Oracle Management Repository (OMR) database (primary database)
• One single-instance database (standby database)

The following diagram shows an active/active traditional OEM Level 3 HA AWS environment.

Oracle has published a whitepaper that explains the OEM Level 3 high availability architecture. The whitepaper also includes OEM and F5 BIG-IP integration instructions.

AWS native OEM HA architecture

The traditional OEM Level 3 HA AWS architecture is a proven, viable option. It’s recommended if you have available Oracle Database Enterprise Edition and Oracle RAC licenses. However, a simpler, much more cost-efficient option is available.

For OEM medium and large environments, the AWS native OEM HA architecture contains the following components:

• Network Load Balancer (Application Load Balancer is also an option)
• Three Oracle Management Service (OMS) servers (application tier)
• One Oracle Management Repository (OMR) database (primary database)
• Optionally, one OMR standby database

If you use the optional standby database, you’re responsible for making sure that the standby database has the proper Oracle licenses.

Unlike the traditional OEM Level 3 HA AWS architecture, this architecture doesn’t require an administrator to configure a Server Load Balancer (SLB) with the OEM OMS. Tasks such as creating SLB monitors, SLB pools, SLB TCP profiles, SLB persistent profiles, SLB rules, and SLB virtual servers aren’t required with the AWS native OEM HA architecture.

This solution provides an active/active OMS for the OEM concurrent users. The following diagram illustrates the AWS native OEM HA architecture.

The following diagram illustrates the AWS native OEM HA architecture with an optional Oracle Database standby database for disaster recovery (DR). The standby OEM database is kept in sync with the primary OEM database via Oracle Data Guard.

The AWS native OEM HA architecture has a dedicated OEM Administrative Server to upload agent activity from all the OEM targets. The OEM Administrative Server also exists to perform OEM administrative tasks; such as cloning a new OEM OMS and deploying OEM agents to targets.

The two OMSs that are used for OEM console users can function without the presence of the OEM Administration Server. When an OMS for OEM console usage starts, it contacts the Administration Server to retrieve its configuration information. If an OMS for OEM console usage is unable to connect to the Administration Server during startup, it can retrieve its configuration information directly by reading a copy of the config.xml file and other files located on the OMS’s own file system. Additionally, if necessary, you can relocate the OEM Administration Server to another EC2 instance.

A current generation EC2 instance type is recommended for the OEM EC2 instances. Current generation instance types, such as instances built on the AWS Nitro System, support hardware virtual machines (HVMs). HVM Amazon Machine Images (AMIs) are required to take advantage of enhanced networking, and they also offer increased security. The following virtualized instances are built on the Nitro System and are recommended for OEM OMS and OMR EC2 instances: C5, M5, R5, R5b, T3 and z1d. As a reminder, the EC2 instance types should be selected based on Oracle’s OEM sizing guide.

After you create your EC2 instances in a private subnet for OEM, restrict network access for inbound and outbound traffic to a least privilege model. The OEM OMS requires more than a dozen open ports on the OMS instances. You should have a security group for the EM OMSs and EM targets. The security group source should refer to a security group in the AWS account, prefix lists, or a specific set of IP addresses (using the x.x.x.x/32 format). The security group source shouldn’t use Classless Inter-Domain Routing (CIDR), nor should security groups be accessible from the public internet (0.0.0.0/0).

All the EC2 instances in this solution should enable Amazon EC2 instance termination. This prevents an instance from being accidentally terminated by requiring the user to disable the protection before terminating the instance.

Additionally, all the EC2 instances in this solution should enable the Amazon EC2 automatic recovery feature. This resolves issues if the hardware that hosts an EC2 instance becomes impaired. This feature recovers the instance on different underlying hardware and reduces the need for manual intervention.

The OEM OMR (database) EC2 instance should follow Oracle’s general best practices for high availability. This includes running the database in ARCHIVELOG mode and enabling flashback database. Database backups should be sent directly to Amazon S3 via the OSB module for Amazon S3. OSB should be implemented with both OEM solutions. Amazon S3 provides data protection while in transit (as it travels to and from Amazon S3) and at rest (while it’s stored on disks in Amazon S3). You can protect data in transit using Secure Socket Layer/Transport Layer Security (SSL/TLS) or client-side encryption. We recommend using server-side encryption (SSE) for this solution. When you use server-side encryption with Amazon S3-managed keys (SSE-S3), each object is encrypted with a unique key. As an additional safeguard, it encrypts the key with a primary key that it regularly rotates. Amazon S3 SSE uses one of the strongest block ciphers available, 256-bit Advanced Encryption Standard (AES-256), to encrypt your data.

Amazon Elastic File System (Amazon EFS) is required when multiple OMSs are deployed with OEM. Therefore, Amazon EFS is required for both OEM high availability solutions. You use Amazon EFS to store the OEM Software Library and the OEM BI Publisher components. The Software Library stores software entities such as software patches, virtual appliance images, reference gold images, application software, and their associated directive scripts. BI Publisher stores all configuration data and report definitions.

The following screenshot shows the three Amazon EFS file systems associated with an OEM highly available environment.

All production data should be encrypted at rest. Enable EBS volume encryption, Amazon EFS volume encryption, and the Oracle database on Amazon EC2 using SSE with AWS Key Management Service (AWS KMS).

Also encrypt data in transit to the Oracle EC2 instance. Oracle Database supports native network encryption and data integrity.

As previously mentioned, you can use a Network Load Balancer (NLB) or Application Load Balancer (ALB) for the AWS native OEM HA architecture. We use the NLB for this solution because the NLB is priced slightly lower and has better performance than the ALB. An ALB is a Layer 7 load balancer type. The NLB is a Layer 4 load balancer type. Both load balancers support IP and instance target types. For a detailed comparison, see Elastic Load Balancing features.

Configure the NLB to access the OEM OMSs designated for the OEM console users. The NLB should be set up using pass-through mode to terminate SSL on the OEM OMS EC2 instances. Additionally, use the NLB sticky session feature (also known as session affinity) to enable the load balancer to bind a user’s session to a specific target. This ensures that all requests from the user during the session are sent to the same target. This feature is useful for servers that maintain state information in order to provide a continuous experience to clients.

The following screenshot illustrates the NLB configuration.

The following screenshot illustrates the target group configuration.

Install the AWS native OEM HA solution

Creating the AWS native OEM HA solution requires familiarity with AWS services and Oracle software. To download Oracle software, you must use a valid Oracle My Oracle Support (MOS) account. To understand prerequisites, installation procedures, and post-installation steps, refer to the AWS, Oracle OEM, and Oracle Database documentation.

We can perform this Oracle installation using a Java UI (such as Oracle runInstaller or Oracle DBCA) or via command line (silent install). We recommend using the Java UI. XQuartz is recommended for macOS as a X Windows client. XQuartz is an open-source project.

This blog has been tested and validated using the following Oracle software versions and patches:

• Oracle Linux 7.6
• Oracle Enterprise Manager 13.4.0.0.0
• Database 19.3.0.0.0
• Database patch 32126828 (Jan 19, 2021—19.10.0.0.210119)
• OEM Database patch 30912308

After successfully installing the AWS native OEM HA architecture, you can access the OEM console via the NLB URL.

The OEM environment should undergo a stress test to ensure that it can easily scale up to twice the number of expected concurrent users. Additionally, monitor the OEM EC2 instances via CloudWatch, and publish notifications to Amazon SNS. If CPU utilization exceeds 80% or I/O throughput is throttled, consider changing the EC2 instance type, EBS volume type, or Provisioned IOPS setting.

Also set up CloudWatch alarms for AWS CloudTrail events. For example, a CloudWatch alarm should be triggered when configuration changes occur on security groups. This alerts the operations team when someone attempts to gain access to the OEM EC2 instances.

We recommend using CloudWatch Logs to store logs (including the OEM error logs). For security purposes, make sure the HOSM (execution) role used is following the least privilege design. We suggest only giving access to PutLogEvents to the EC2 instances.

Periodically (every 2 weeks at minimum), run a scan against your EC2 instances and verify compliance. You can accomplish this using Amazon Inspector. Amazon Inspector is an automated security assessment service that helps improve the security and compliance of applications deployed on AWS. Amazon Inspector automatically assesses applications for exposure, vulnerabilities, and deviations from best practices. After performing an assessment, Amazon Inspector produces a detailed list of security findings prioritized by level of severity. You can review these findings directly or as part of detailed assessment reports, which are available via the Amazon Inspector console or API.

Conclusion

In this post, we provided two high availability solutions to deploy OEM in AWS.

The AWS native OEM HA architecture, without the optional Oracle standby database, provides the following protection:

• Multiple OMS active/active environments to support a large number of OEM concurrent users
• Protection from most hardware failures for EC2 instances

The AWS native OEM HA architecture is less complex, has a faster implementation time, and doesn’t rely on AWS third-party solutions to provide OEM high availability when compared to the traditional OEM Level 3 HA AWS architecture.

To learn more about OEM and its integration with Amazon RDS for Oracle, refer to AWS documentation.

About the Author

Marvin Vinson is an Oracle Database Specialist for AWS. He works in the Worldwide Public Sector (WWPS) organization. He is focused on migrating Oracle workloads to AWS.