AWS Database Blog
Use cases for query plan management in Amazon Aurora PostgreSQL
This blog post is the second in a series. The previous blog post talks about the need for the stable, consistent database performance amid changes that otherwise can cause regression on execution plans of the SQL statements. It also demonstrates how query plan management (QPM) for Amazon Aurora with PostgreSQL compatibility helps you overcome plan stability and plan adaptability issues.
In this post, we continue to discuss Aurora PostgreSQL QPM features. In particular, we discuss how these features help you achieve plan stability and adaptability for a variety of advanced use cases.
Use case #1: Helping ensure plan stability and adaptability with QPM manual capture
For the first use case, we walk through an example on how QPM helps ensure plan stability. Then, you put into place the changes that we discussed in the previous post, Introduction to Aurora PostgreSQL Query Plan Management. Without QPM, these changes can result in plan regression.
In most cases, you set up QPM to use automatic plan capture so that plans are captured for all statements that run two or more times. However, you can also capture plans for a specific set of statements that you specify manually. To do this, you set capture_plan_baselines = off
by default. At the session level, capture_plan_baselines = manual
at the session level. How to do it is described below.
- Enable manual plan capture to instruct QPM to capture the execution plan of the desired SQL statements manually.
- Run an explain plan for the query so that QPM can capture the plan of the query (the following output for the explain plan is truncated for brevity).
- Disable manual capture of the plan after you capture the execution plan for the desired SQL statement.
- View the captured query plan for the query that ran previously. The
plan_outline
column in the tableapg_plan_mgmt.dba_plans
shows the entire plan for the SQL. For brevity, theplan_outline
isn’t shown here. Instead,plan_hash_value
from the explain plan preceding is compared withplan_hash
from the output of theapg_plan_mgmt.dba_plans
query. - To instruct the query optimizer to use the approved or preferred captured plans for your managed statements, set the parameter
apg_plan_mgmt.use_plan_baselines
to true. - View the explain plan output to see that the QPM approved plan is used by the query optimizer.
- Create a new index on the
pgbench_history
table column “mtime” to change the planner configuration and force the query optimizer to generate a new plan. - View the explain plan output to see that QPM detects a new plan but still uses the approved plan and maintains the plan stability.
- Run the following SQL query to view the new plan and status of the plan. To ensure plan stability, QPM stores all the newly generated plans for a managed query in QPM as unapproved plans. The following output shows that there are two different execution plans stored for the same managed statement, as shown by the two different
plan_hash
values. Although the new execution plan has the minimum cost (lower than the approved plan), QPM continues to ignore the unapproved plans to maintain plan stability.Theplan_outline
column in the tableapg_plan_mgmt.dba_plans
shows the entire plan for the SQL. For the sake of brevity, theplan_outline
is not shown here. Instead,plan_hash_value
from the explain plan preceding is compared withplan_hash
from the output of theapg_plan_mgmt.dba_plans
query.The following is an example of plan adaptability with QPM. This example evaluates the unapproved plan based on the minimum speedup factor. It approves any captured unapproved plan that is faster by at least 10 percent than the best approved plan for the statement. For additional details, see Evaluating Plan Performance in the Aurora documentation.
After QPM evaluates the plan based on the speed factor, the plan status changes to approved. At this point, the optimizer can choose that plan for that managed statement now.
Use case #2. Fixing plans with QPM using pg_hint_plan
In some cases, the query optimizer doesn’t generate the best execution plan for the query. One approach to fixing this problem is to put query hints into your application code, but this approach is widely discouraged because it makes applications more brittle and harder to maintain, and in some cases, you can’t hint the SQL because it is generated by a 3rd party application. What we will show is how to use hints to control the query optimizer, but then to remove the hints and allow QPM to enforce the desired plan, without adding hints to the application code.
For this purpose, PostgreSQL users can use the pg_hint_plan
extension to provide directives such as “scan method,” “join method,” “join order,”, or “row number correction,” to the optimizer. The resulting plan will be saved by QPM, along with any GUC parameters you choose to override (such as work_mem
). QPM remembers any GUC parameter overrides and uses them when it needs to recreate the plan. To install and learn more about how to use the pg_hint_plan
extension, see the pg_hint_plan documentation.
QPM steps to fix the plan generated by using hints
Working with pg_hint_plan
is incredibly useful for cases where the query can’t be modified to add hints. In this example, use a sample query to generate the execution plan that you want by adding hints to the managed statement. Then associate this execution plan with the original unmodified statement.
Here are the detailed steps:
- Check if the plan capture is disabled.
- Run the query with the hint to use. In the following example, use the “HashJoin” hint, which is a directive for the optimizer to choose the join method as HashJoin. The original plan of the query without hints is as follows.
- Enable
pg_hint_plan
and manual plan capture. - EXPLAIN the query with the hints you want to use. In the following example, use the HashJoin (a, b) hint, which is a directive for the optimizer to use a hash join algorithm to join from table a to table b. The plan that you want with a hash join is as follows.
- Verify that plan 1139293728 was captured, and note the status of the plan. View the captured plan and the status of the plan.
- If necessary, Approve the plan. In this case this is the first and only plan saved for statement 356104612, so it was saved as an Approved plan. If this statement already had a baseline of approved plans, then this plan would have been saved as an Unapproved plan. In general, to Reject all existing plans for a statement and then Approve one specific plan, you could call apg_plan_mgmt.set_plan_status twice, like this.
- Remove the hint, turn off manual capture, turn on
use_plan_baselines
, and also verify that the desired plan is in use without the hint.
Use case 3: Deploying QPM-managed plans globally using export and import for QPM-managed plans
Large enterprise customers often have applications and databases deployed globally. They also often maintain several environments (Dev, QA, Staging, UAT, Preprod, and Prod) for each application database. However, managing the execution plans manually in each of the databases in specific AWS Regions and each of the database environments can be cumbersome and time-consuming.
QPM provides an option to export and import QPM-managed plans from one database to another database. With this option, you can manage the query execution centrally and deploy databases globally. This feature is useful for the scenarios where you investigate a set of plans on a preprod database, verify that they perform well, and then load them into a production database.
Here are the steps to migrate QPM-managed plans from one database to another. For additional details, see Exporting and Importing Plans in the Aurora documentation.
- Export the QPM-managed plan from the source system.
To do this, from the source database with the preferred execution plan, an authorized DB user can copy any subset of theapg_plan_mgmt.plans
table to another table. That user can then save it using thepg_dump
command. For additional details on thepg_dump
, see pg_dump in the PostgreSQL documentation. - Import the QPM-managed plan on the target system.
- On the target system, copy any subset of the
apg_plan_mgmt.plans
table to another table, and then save it using thepg_dump
command. This is an optional step to preserve any existing managed plans on the target system before importing new plans for the source system. - We have assumed that you have used
pg_dump
tar-format archive in step 1. Use thepg_restore
command to copy the .tar file into a new table (plan_copy). For additional details about thepg_restore
, see pg_restore in the PostgreSQL documentation. - Merge the new table with the
apg_plan_mgmt.plans
table. - Reload the managed plans into shared memory and remove the temporary plans table.
SELECT apg_plan_mgmt.reload();
— Refresh shared memory with new plans.DROP TABLE plans_copy; --
Drop the temporary plan table.
Summary
In this post, we have covered some of the advanced use cases for Aurora PostgreSQL QPM. One of these is how QPM integrates with pg_hint_plan
, which allows the optimizer to generate an optimal execution plan using hints. The optimizer can do this even for SQL statements that can’t be modified (for example, statements that are part of commercial off-the-shelf applications).
Other use cases include working with QPM export and import features, which enable you to manage your execution plans centrally and deploy them globally. Central management and global deployment enable quick deployment of the good known execution plans across multiple databases spread across multiple environments and AWS Regions.
We have covered a few of the important use cases for QPM. In addition, you can implement other use cases using QPM that are equally important to achieving sustainable and optimal database performance. For additional details about QPM, see Managing Query Execution Plans for Aurora PostgreSQL in the Aurora documentation.
About the Authors
Sameer Malik is a principal database solutions architect at Amazon Web Services.
Jim Finnerty is a principal software engineer for Aurora PostgreSQL at Amazon Web Services.