mirror of https://github.com/postgres/postgres
Before executing a cached generic plan, AcquireExecutorLocks() in plancache.c locks all relations in a plan's range table to ensure the plan is safe for execution. However, this locks runtime-prunable relations that will later be pruned during "initial" runtime pruning, introducing unnecessary overhead. This commit defers locking for such relations to executor startup and ensures that if the CachedPlan is invalidated due to concurrent DDL during this window, replanning is triggered. Deferring these locks avoids unnecessary locking overhead for pruned partitions, resulting in significant speedup, particularly when many partitions are pruned during initial runtime pruning. * Changes to locking when executing generic plans: AcquireExecutorLocks() now locks only unprunable relations, that is, those found in PlannedStmt.unprunableRelids (introduced in commitpull/200/headcbc127917e), to avoid locking runtime-prunable partitions unnecessarily. The remaining locks are taken by ExecDoInitialPruning(), which acquires them only for partitions that survive pruning. This deferral does not affect the locks required for permission checking in InitPlan(), which takes place before initial pruning. ExecCheckPermissions() now includes an Assert to verify that all relations undergoing permission checks, none of which can be in the set of runtime-prunable relations, are properly locked. * Plan invalidation handling: Deferring locks introduces a window where prunable relations may be altered by concurrent DDL, invalidating the plan. A new function, ExecutorStartCachedPlan(), wraps ExecutorStart() to detect and handle invalidation caused by deferred locking. If invalidation occurs, ExecutorStartCachedPlan() updates CachedPlan using the new UpdateCachedPlan() function and retries execution with the updated plan. To ensure all code paths that may be affected by this handle invalidation properly, all callers of ExecutorStart that may execute a PlannedStmt from a CachedPlan have been updated to use ExecutorStartCachedPlan() instead. UpdateCachedPlan() replaces stale plans in CachedPlan.stmt_list. A new CachedPlan.stmt_context, created as a child of CachedPlan.context, allows freeing old PlannedStmts while preserving the CachedPlan structure and its statement list. This ensures that loops over statements in upstream callers of ExecutorStartCachedPlan() remain intact. ExecutorStart() and ExecutorStart_hook implementations now return a boolean value indicating whether plan initialization succeeded with a valid PlanState tree in QueryDesc.planstate, or false otherwise, in which case QueryDesc.planstate is NULL. Hook implementations are required to call standard_ExecutorStart() at the beginning, and if it returns false, they should do the same without proceeding. * Testing: To verify these changes, the delay_execution module tests scenarios where cached plans become invalid due to changes in prunable relations after deferred locks. * Note to extension authors: ExecutorStart_hook implementations must verify plan validity after calling standard_ExecutorStart(), as explained earlier. For example: if (prev_ExecutorStart) plan_valid = prev_ExecutorStart(queryDesc, eflags); else plan_valid = standard_ExecutorStart(queryDesc, eflags); if (!plan_valid) return false; <extension-code> return true; Extensions accessing child relations, especially prunable partitions, via ExecGetRangeTableRelation() must now ensure their RT indexes are present in es_unpruned_relids (introduced in commitcbc127917e), or they will encounter an error. This is a strict requirement after this change, as only relations in that set are locked. The idea of deferring some locks to executor startup, allowing locks for prunable partitions to be skipped, was first proposed by Tom Lane. Reviewed-by: Robert Haas <robertmhaas@gmail.com> (earlier versions) Reviewed-by: David Rowley <dgrowleyml@gmail.com> (earlier versions) Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> (earlier versions) Reviewed-by: Tomas Vondra <tomas@vondra.me> Reviewed-by: Junwang Zhao <zhjwpku@gmail.com> Discussion: https://postgr.es/m/CA+HiwqFGkMSge6TgC9KQzde0ohpAycLQuV7ooitEEpbKB0O_mg@mail.gmail.com
Amit Langote
7 months ago
parent
4aa6fa3cd0
commit
525392d572
@ -0,0 +1,250 @@ |
|||||||
|
Parsed test spec with 2 sessions |
||||||
|
|
||||||
|
starting permutation: s1prep s2lock s1exec s2dropi s2unlock |
||||||
|
step s1prep: SET plan_cache_mode = force_generic_plan; |
||||||
|
PREPARE q AS SELECT * FROM foov WHERE a = $1 FOR UPDATE; |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q (1); |
||||||
|
QUERY PLAN |
||||||
|
----------------------------------------------------- |
||||||
|
LockRows |
||||||
|
-> Append |
||||||
|
Subplans Removed: 2 |
||||||
|
-> Index Scan using foo1_1_a on foo1_1 foo_1 |
||||||
|
Index Cond: (a = $1) |
||||||
|
(5 rows) |
||||||
|
|
||||||
|
step s2lock: SELECT pg_advisory_lock(12345); |
||||||
|
pg_advisory_lock |
||||||
|
---------------- |
||||||
|
|
||||||
|
(1 row) |
||||||
|
|
||||||
|
step s1exec: LOAD 'delay_execution'; |
||||||
|
SET delay_execution.executor_start_lock_id = 12345; |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q (1); <waiting ...> |
||||||
|
step s2dropi: DROP INDEX foo1_1_a; |
||||||
|
step s2unlock: SELECT pg_advisory_unlock(12345); |
||||||
|
pg_advisory_unlock |
||||||
|
------------------ |
||||||
|
t |
||||||
|
(1 row) |
||||||
|
|
||||||
|
step s1exec: <... completed> |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is not valid |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is valid |
||||||
|
QUERY PLAN |
||||||
|
------------------------------------ |
||||||
|
LockRows |
||||||
|
-> Append |
||||||
|
Subplans Removed: 2 |
||||||
|
-> Seq Scan on foo1_1 foo_1 |
||||||
|
Filter: (a = $1) |
||||||
|
(5 rows) |
||||||
|
|
||||||
|
|
||||||
|
starting permutation: s1prep2 s2lock s1exec2 s2dropi s2unlock |
||||||
|
step s1prep2: SET plan_cache_mode = force_generic_plan; |
||||||
|
PREPARE q2 AS SELECT * FROM foov WHERE a = one() or a = two(); |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q2; |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is valid |
||||||
|
QUERY PLAN |
||||||
|
--------------------------------------------------- |
||||||
|
Append |
||||||
|
Subplans Removed: 1 |
||||||
|
-> Index Scan using foo1_1_a on foo1_1 foo_1 |
||||||
|
Index Cond: (a = ANY (ARRAY[one(), two()])) |
||||||
|
-> Seq Scan on foo1_2 foo_2 |
||||||
|
Filter: ((a = one()) OR (a = two())) |
||||||
|
(6 rows) |
||||||
|
|
||||||
|
step s2lock: SELECT pg_advisory_lock(12345); |
||||||
|
pg_advisory_lock |
||||||
|
---------------- |
||||||
|
|
||||||
|
(1 row) |
||||||
|
|
||||||
|
step s1exec2: LOAD 'delay_execution'; |
||||||
|
SET delay_execution.executor_start_lock_id = 12345; |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q2; <waiting ...> |
||||||
|
step s2dropi: DROP INDEX foo1_1_a; |
||||||
|
step s2unlock: SELECT pg_advisory_unlock(12345); |
||||||
|
pg_advisory_unlock |
||||||
|
------------------ |
||||||
|
t |
||||||
|
(1 row) |
||||||
|
|
||||||
|
step s1exec2: <... completed> |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is not valid |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is valid |
||||||
|
QUERY PLAN |
||||||
|
-------------------------------------------- |
||||||
|
Append |
||||||
|
Subplans Removed: 1 |
||||||
|
-> Seq Scan on foo1_1 foo_1 |
||||||
|
Filter: ((a = one()) OR (a = two())) |
||||||
|
-> Seq Scan on foo1_2 foo_2 |
||||||
|
Filter: ((a = one()) OR (a = two())) |
||||||
|
(6 rows) |
||||||
|
|
||||||
|
|
||||||
|
starting permutation: s1prep3 s2lock s1exec3 s2dropi s2unlock |
||||||
|
step s1prep3: SET plan_cache_mode = force_generic_plan; |
||||||
|
PREPARE q3 AS UPDATE foov SET a = a WHERE a = one() or a = two(); |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q3; |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is valid |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is valid |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is valid |
||||||
|
QUERY PLAN |
||||||
|
--------------------------------------------------------------- |
||||||
|
Nested Loop |
||||||
|
-> Append |
||||||
|
Subplans Removed: 1 |
||||||
|
-> Index Only Scan using foo1_1_a on foo1_1 foo_1 |
||||||
|
Index Cond: (a = ANY (ARRAY[one(), two()])) |
||||||
|
-> Seq Scan on foo1_2 foo_2 |
||||||
|
Filter: ((a = one()) OR (a = two())) |
||||||
|
-> Materialize |
||||||
|
-> Append |
||||||
|
Subplans Removed: 1 |
||||||
|
-> Seq Scan on bar1 bar_1 |
||||||
|
Filter: (a = one()) |
||||||
|
|
||||||
|
Update on bar |
||||||
|
Update on bar1 bar_1 |
||||||
|
-> Nested Loop |
||||||
|
-> Append |
||||||
|
Subplans Removed: 1 |
||||||
|
-> Index Scan using foo1_1_a on foo1_1 foo_1 |
||||||
|
Index Cond: (a = ANY (ARRAY[one(), two()])) |
||||||
|
-> Seq Scan on foo1_2 foo_2 |
||||||
|
Filter: ((a = one()) OR (a = two())) |
||||||
|
-> Materialize |
||||||
|
-> Append |
||||||
|
Subplans Removed: 1 |
||||||
|
-> Seq Scan on bar1 bar_1 |
||||||
|
Filter: (a = one()) |
||||||
|
|
||||||
|
Update on foo |
||||||
|
Update on foo1_1 foo_1 |
||||||
|
Update on foo1_2 foo_2 |
||||||
|
-> Append |
||||||
|
Subplans Removed: 1 |
||||||
|
-> Index Scan using foo1_1_a on foo1_1 foo_1 |
||||||
|
Index Cond: (a = ANY (ARRAY[one(), two()])) |
||||||
|
-> Seq Scan on foo1_2 foo_2 |
||||||
|
Filter: ((a = one()) OR (a = two())) |
||||||
|
(37 rows) |
||||||
|
|
||||||
|
step s2lock: SELECT pg_advisory_lock(12345); |
||||||
|
pg_advisory_lock |
||||||
|
---------------- |
||||||
|
|
||||||
|
(1 row) |
||||||
|
|
||||||
|
step s1exec3: LOAD 'delay_execution'; |
||||||
|
SET delay_execution.executor_start_lock_id = 12345; |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q3; <waiting ...> |
||||||
|
step s2dropi: DROP INDEX foo1_1_a; |
||||||
|
step s2unlock: SELECT pg_advisory_unlock(12345); |
||||||
|
pg_advisory_unlock |
||||||
|
------------------ |
||||||
|
t |
||||||
|
(1 row) |
||||||
|
|
||||||
|
step s1exec3: <... completed> |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is not valid |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is valid |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is valid |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is valid |
||||||
|
QUERY PLAN |
||||||
|
-------------------------------------------------------- |
||||||
|
Nested Loop |
||||||
|
-> Append |
||||||
|
Subplans Removed: 1 |
||||||
|
-> Seq Scan on foo1_1 foo_1 |
||||||
|
Filter: ((a = one()) OR (a = two())) |
||||||
|
-> Seq Scan on foo1_2 foo_2 |
||||||
|
Filter: ((a = one()) OR (a = two())) |
||||||
|
-> Materialize |
||||||
|
-> Append |
||||||
|
Subplans Removed: 1 |
||||||
|
-> Seq Scan on bar1 bar_1 |
||||||
|
Filter: (a = one()) |
||||||
|
|
||||||
|
Update on bar |
||||||
|
Update on bar1 bar_1 |
||||||
|
-> Nested Loop |
||||||
|
-> Append |
||||||
|
Subplans Removed: 1 |
||||||
|
-> Seq Scan on foo1_1 foo_1 |
||||||
|
Filter: ((a = one()) OR (a = two())) |
||||||
|
-> Seq Scan on foo1_2 foo_2 |
||||||
|
Filter: ((a = one()) OR (a = two())) |
||||||
|
-> Materialize |
||||||
|
-> Append |
||||||
|
Subplans Removed: 1 |
||||||
|
-> Seq Scan on bar1 bar_1 |
||||||
|
Filter: (a = one()) |
||||||
|
|
||||||
|
Update on foo |
||||||
|
Update on foo1_1 foo_1 |
||||||
|
Update on foo1_2 foo_2 |
||||||
|
-> Append |
||||||
|
Subplans Removed: 1 |
||||||
|
-> Seq Scan on foo1_1 foo_1 |
||||||
|
Filter: ((a = one()) OR (a = two())) |
||||||
|
-> Seq Scan on foo1_2 foo_2 |
||||||
|
Filter: ((a = one()) OR (a = two())) |
||||||
|
(37 rows) |
||||||
|
|
||||||
|
|
||||||
|
starting permutation: s1prep4 s2lock s1exec4 s2dropi s2unlock |
||||||
|
step s1prep4: SET plan_cache_mode = force_generic_plan; |
||||||
|
PREPARE q4 AS SELECT * FROM generate_series(1, 1) WHERE EXISTS (SELECT * FROM foov WHERE a = $1 FOR UPDATE); |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q4 (1); |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is valid |
||||||
|
QUERY PLAN |
||||||
|
------------------------------------------------------------- |
||||||
|
Result |
||||||
|
One-Time Filter: (InitPlan 1).col1 |
||||||
|
InitPlan 1 |
||||||
|
-> LockRows |
||||||
|
-> Append |
||||||
|
Subplans Removed: 2 |
||||||
|
-> Index Scan using foo1_1_a on foo1_1 foo_1 |
||||||
|
Index Cond: (a = $1) |
||||||
|
-> Function Scan on generate_series |
||||||
|
(9 rows) |
||||||
|
|
||||||
|
step s2lock: SELECT pg_advisory_lock(12345); |
||||||
|
pg_advisory_lock |
||||||
|
---------------- |
||||||
|
|
||||||
|
(1 row) |
||||||
|
|
||||||
|
step s1exec4: LOAD 'delay_execution'; |
||||||
|
SET delay_execution.executor_start_lock_id = 12345; |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q4 (1); <waiting ...> |
||||||
|
step s2dropi: DROP INDEX foo1_1_a; |
||||||
|
step s2unlock: SELECT pg_advisory_unlock(12345); |
||||||
|
pg_advisory_unlock |
||||||
|
------------------ |
||||||
|
t |
||||||
|
(1 row) |
||||||
|
|
||||||
|
step s1exec4: <... completed> |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is not valid |
||||||
|
s1: NOTICE: Finished ExecutorStart(): CachedPlan is valid |
||||||
|
QUERY PLAN |
||||||
|
-------------------------------------------- |
||||||
|
Result |
||||||
|
One-Time Filter: (InitPlan 1).col1 |
||||||
|
InitPlan 1 |
||||||
|
-> LockRows |
||||||
|
-> Append |
||||||
|
Subplans Removed: 2 |
||||||
|
-> Seq Scan on foo1_1 foo_1 |
||||||
|
Filter: (a = $1) |
||||||
|
-> Function Scan on generate_series |
||||||
|
(9 rows) |
||||||
|
|
||||||
@ -0,0 +1,86 @@ |
|||||||
|
# Test to check that invalidation of cached generic plans during ExecutorStart |
||||||
|
# is correctly detected causing an updated plan to be re-executed. |
||||||
|
|
||||||
|
setup |
||||||
|
{ |
||||||
|
CREATE TABLE foo (a int, b text) PARTITION BY RANGE (a); |
||||||
|
CREATE TABLE foo1 PARTITION OF foo FOR VALUES FROM (MINVALUE) TO (3) PARTITION BY RANGE (a); |
||||||
|
CREATE TABLE foo1_1 PARTITION OF foo1 FOR VALUES FROM (MINVALUE) TO (2); |
||||||
|
CREATE TABLE foo1_2 PARTITION OF foo1 FOR VALUES FROM (2) TO (3); |
||||||
|
CREATE INDEX foo1_1_a ON foo1_1 (a); |
||||||
|
CREATE TABLE foo2 PARTITION OF foo FOR VALUES FROM (3) TO (MAXVALUE); |
||||||
|
INSERT INTO foo SELECT generate_series(-1000, 1000); |
||||||
|
CREATE VIEW foov AS SELECT * FROM foo; |
||||||
|
CREATE FUNCTION one () RETURNS int AS $$ BEGIN RETURN 1; END; $$ LANGUAGE PLPGSQL STABLE; |
||||||
|
CREATE FUNCTION two () RETURNS int AS $$ BEGIN RETURN 2; END; $$ LANGUAGE PLPGSQL STABLE; |
||||||
|
CREATE TABLE bar (a int, b text) PARTITION BY LIST(a); |
||||||
|
CREATE TABLE bar1 PARTITION OF bar FOR VALUES IN (1); |
||||||
|
CREATE INDEX ON bar1(a); |
||||||
|
CREATE TABLE bar2 PARTITION OF bar FOR VALUES IN (2); |
||||||
|
CREATE RULE update_foo AS ON UPDATE TO foo DO ALSO UPDATE bar SET a = a WHERE a = one(); |
||||||
|
CREATE RULE update_bar AS ON UPDATE TO bar DO ALSO SELECT 1; |
||||||
|
ANALYZE; |
||||||
|
} |
||||||
|
|
||||||
|
teardown |
||||||
|
{ |
||||||
|
DROP VIEW foov; |
||||||
|
DROP RULE update_foo ON foo; |
||||||
|
DROP TABLE foo, bar; |
||||||
|
DROP FUNCTION one(), two(); |
||||||
|
} |
||||||
|
|
||||||
|
session "s1" |
||||||
|
step "s1prep" { SET plan_cache_mode = force_generic_plan; |
||||||
|
PREPARE q AS SELECT * FROM foov WHERE a = $1 FOR UPDATE; |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q (1); } |
||||||
|
|
||||||
|
step "s1prep2" { SET plan_cache_mode = force_generic_plan; |
||||||
|
PREPARE q2 AS SELECT * FROM foov WHERE a = one() or a = two(); |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q2; } |
||||||
|
|
||||||
|
step "s1prep3" { SET plan_cache_mode = force_generic_plan; |
||||||
|
PREPARE q3 AS UPDATE foov SET a = a WHERE a = one() or a = two(); |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q3; } |
||||||
|
|
||||||
|
step "s1prep4" { SET plan_cache_mode = force_generic_plan; |
||||||
|
PREPARE q4 AS SELECT * FROM generate_series(1, 1) WHERE EXISTS (SELECT * FROM foov WHERE a = $1 FOR UPDATE); |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q4 (1); } |
||||||
|
|
||||||
|
step "s1exec" { LOAD 'delay_execution'; |
||||||
|
SET delay_execution.executor_start_lock_id = 12345; |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q (1); } |
||||||
|
step "s1exec2" { LOAD 'delay_execution'; |
||||||
|
SET delay_execution.executor_start_lock_id = 12345; |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q2; } |
||||||
|
step "s1exec3" { LOAD 'delay_execution'; |
||||||
|
SET delay_execution.executor_start_lock_id = 12345; |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q3; } |
||||||
|
step "s1exec4" { LOAD 'delay_execution'; |
||||||
|
SET delay_execution.executor_start_lock_id = 12345; |
||||||
|
EXPLAIN (COSTS OFF) EXECUTE q4 (1); } |
||||||
|
|
||||||
|
session "s2" |
||||||
|
step "s2lock" { SELECT pg_advisory_lock(12345); } |
||||||
|
step "s2unlock" { SELECT pg_advisory_unlock(12345); } |
||||||
|
step "s2dropi" { DROP INDEX foo1_1_a; } |
||||||
|
|
||||||
|
# In all permutations below, while "s1exec", "s1exec2", etc. wait to |
||||||
|
# acquire the advisory lock, "s2drop" drops the index being used in the |
||||||
|
# cached plan. When "s1exec" and others are unblocked and begin initializing |
||||||
|
# the plan, including acquiring necessary locks on partitions, the concurrent |
||||||
|
# index drop is detected. This causes plan initialization to be aborted, |
||||||
|
# prompting the caller to retry with a new plan. |
||||||
|
|
||||||
|
# Case with runtime pruning using EXTERN parameter |
||||||
|
permutation "s1prep" "s2lock" "s1exec" "s2dropi" "s2unlock" |
||||||
|
|
||||||
|
# Case with runtime pruning using stable function |
||||||
|
permutation "s1prep2" "s2lock" "s1exec2" "s2dropi" "s2unlock" |
||||||
|
|
||||||
|
# Case with a rule adding another query causing the CachedPlan to contain |
||||||
|
# multiple PlannedStmts |
||||||
|
permutation "s1prep3" "s2lock" "s1exec3" "s2dropi" "s2unlock" |
||||||
|
|
||||||
|
# Case with run-time pruning inside a subquery |
||||||
|
permutation "s1prep4" "s2lock" "s1exec4" "s2dropi" "s2unlock" |
||||||
Loading…
Reference in new issue