How do I resolve the error "Cache of region boundaries are out of date" when running an HBase read-replica Amazon EMR cluster with Phoenix?

Error: ERROR 1108 (XCL08): Cache of region boundaries are out of date. (state=XCL08,code=1108) org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108
 (XCL08): Cache of region boundaries are out of date.
      at org.apache.phoenix.exception.SQLExceptionCode$14.newException(SQLExceptionCode.java:365)
      at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150)
      at org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:189)
      at org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:169)
      at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:140)

Phoenix tries to connect to the hbase:meta table by default. However, because the hbase:meta table belongs to the primary cluster, Phoenix can't to connect the read-replica cluster. To resolve this problem, modify hbase-site.xml to point to the hbase:meta_cluster-id table that belongs to the HBase read-replica cluster.

Before you begin, confirm that Phoenix is installed on the primary cluster as well as the read-replica cluster. Phoenix can't connect to HBase from a read-replica cluster if Phoenix isn't installed on the primary cluster.

On a running cluster

1.    Add the following configurations to the HBase configuration file (/etc/phoenix/conf/hbase-site.xml) on the master node. Replace cluster-id with the ID of your read-replica cluster.

<property>
   <name>hbase.balancer.tablesOnMaster</name>
   <value>hbase:meta</value>
</property>
<property>
   <name>hbase.meta.table.suffix</name>
   <value>cluster-id</value>
</property>

2.    Restart the Phoenix service:

sudo stop phoenix-queryserver
sudo start phoenix-queryserver

In Amazon EMR release versions 5.21.0 and later, you can also make these configuration changes by overriding the cluster configuration for the master instance group:

1.    Open the Amazon EMR console.

2.    In the cluster list, choose the active read-replica cluster that you want to reconfigure.

3.    Open the cluster details page for the cluster and go to Configurations tab.

4.    In the Filter drop-down list, choose the master instance group.

5.    In Reconfigure drop-down menu, choose either Edit in table.

6.    Choose Add configuration, and then add the following two configurations:

Classification: phoenix-hbase-site
Property: hbase.balancer.tablesOnMaster
Value: hbase:meta

Classification: phoenix-hbase-site
Property: hbase.meta.table.suffix
Value: ${emr.clusterId}

7.    Choose Save changes.

For more information about this process, see Supplying a Configuration for an Instance Group in the Console.

On a new cluster

Add a configuration object similar to the following when you launch a cluster using Amazon EMR release version 4.6.0 or later:

[
  {
    "Classification": "phoenix-hbase-site", 
    "Configurations": [
      ], 
    "Properties": {
      "hbase.balancer.tablesOnMaster" : "hbase:meta",
      "hbase.meta.table.suffix" : "${emr.clusterId}"
      }
  },
  {
    "Classification": "hbase-site", 
    "Configurations": [
      ], 
    "Properties": {
      "hbase.meta.table.suffix" : "${emr.clusterId}"
      }
  }
]