open-dsi
/
postgres
mirror of https://github.com/postgres/postgres
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Report progress of REINDEX operations

Browse Source 
This uses the same infrastructure that the CREATE INDEX progress
reporting uses.  Add a column to pg_stat_progress_create_index to
report the OID of the index being worked on.  This was not necessary
for CREATE INDEX, but it's useful for REINDEX.

Also edit the phase descriptions a bit to be more consistent with the
source code comments.

Discussion: https://www.postgresql.org/message-id/ef6a6757-c36a-9e81-123f-13b19e36b7d7%402ndquadrant.com
pull/41/head
Peter Eisentraut 6 years ago
parent 106f2eb664
commit 03f9e5cba0
7 changed files with 101 additions and 23 deletions
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Show Stats Download Patch File Download Diff File
  1. 50
      doc/src/sgml/monitoring.sgml
  2. 10
      src/backend/catalog/index.c
  3. 9
      src/backend/catalog/system_views.sql
  4. 41
      src/backend/commands/indexcmds.c
  5. 2
      src/include/catalog/catversion.h
  6. 3
      src/include/commands/progress.h
  7. 9
      src/test/regress/expected/rules.out

50
doc/src/sgml/monitoring.sgml
Unescape View File

@ -346,7 +346,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<row> <row>
<entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry> <entry><structname>pg_stat_progress_create_index</structname><indexterm><primary>pg_stat_progress_create_index</primary></indexterm></entry>
<entry>One row for each backend running <command>CREATE INDEX</command>, showing <entry>One row for each backend running <command>CREATE INDEX</command> or <command>REINDEX</command>, showing
current progress. current progress.
See <xref linkend='create-index-progress-reporting'/>. See <xref linkend='create-index-progress-reporting'/>.
</entry> </entry>
@ -3477,7 +3477,7 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
<title>CREATE INDEX Progress Reporting</title> <title>CREATE INDEX Progress Reporting</title>
<para> <para>
Whenever <command>CREATE INDEX</command> is running, the Whenever <command>CREATE INDEX</command> or <command>REINDEX</command> is running, the
<structname>pg_stat_progress_create_index</structname> view will contain <structname>pg_stat_progress_create_index</structname> view will contain
one row for each backend that is currently creating indexes. The tables one row for each backend that is currently creating indexes. The tables
below describe the information that will be reported and provide information below describe the information that will be reported and provide information
@ -3516,6 +3516,12 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
<entry><type>oid</type></entry> <entry><type>oid</type></entry>
<entry>OID of the table on which the index is being created.</entry> <entry>OID of the table on which the index is being created.</entry>
</row> </row>
<row>
<entry><structfield>index_relid</structfield></entry>
<entry><type>oid</type></entry>
<entry>OID of the index being created or reindexed. During a
non-concurrent <command>CREATE INDEX</command>, this is 0.</entry>
</row>
<row> <row>
<entry><structfield>phase</structfield></entry> <entry><structfield>phase</structfield></entry>
<entry><type>text</type></entry> <entry><type>text</type></entry>
@ -3605,15 +3611,15 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
<row> <row>
<entry><literal>initializing</literal></entry> <entry><literal>initializing</literal></entry>
<entry> <entry>
<command>CREATE INDEX</command> is preparing to create the index. This <command>CREATE INDEX</command> or <command>REINDEX</command> is preparing to create the index. This
phase is expected to be very brief. phase is expected to be very brief.
</entry> </entry>
</row> </row>
<row> <row>
<entry><literal>waiting for old snapshots</literal></entry> <entry><literal>waiting for writers before build</literal></entry>
<entry> <entry>
<command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions <command>CREATE INDEX CONCURRENTLY</command> or <command>REINDEX CONCURRENTLY</command> is waiting for transactions
that can potentially see the table to release their snapshots. with write locks that can potentially see the table to finish.
This phase is skipped when not in concurrent mode. This phase is skipped when not in concurrent mode.
Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
and <structname>current_locker_pid</structname> contain the progress and <structname>current_locker_pid</structname> contain the progress
@ -3632,10 +3638,10 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
</entry> </entry>
</row> </row>
<row> <row>
<entry><literal>waiting for writer snapshots</literal></entry> <entry><literal>waiting for writers before validation</literal></entry>
<entry> <entry>
<command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions <command>CREATE INDEX CONCURRENTLY</command> or <command>REINDEX CONCURRENTLY</command> is waiting for transactions
that can potentially write into the table to release their snapshots. with write locks that can potentially write into the table to finish.
This phase is skipped when not in concurrent mode. This phase is skipped when not in concurrent mode.
Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
and <structname>current_locker_pid</structname> contain the progress and <structname>current_locker_pid</structname> contain the progress
@ -3670,9 +3676,9 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
</entry> </entry>
</row> </row>
<row> <row>
<entry><literal>waiting for reader snapshots</literal></entry> <entry><literal>waiting for old snapshots</literal></entry>
<entry> <entry>
<command>CREATE INDEX CONCURRENTLY</command> is waiting for transactions <command>CREATE INDEX CONCURRENTLY</command> or <command>REINDEX CONCURRENTLY</command> is waiting for transactions
that can potentially see the table to release their snapshots. This that can potentially see the table to release their snapshots. This
phase is skipped when not in concurrent mode. phase is skipped when not in concurrent mode.
Columns <structname>lockers_total</structname>, <structname>lockers_done</structname> Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
@ -3680,6 +3686,28 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
information for this phase. information for this phase.
</entry> </entry>
</row> </row>
<row>
<entry><literal>waiting for readers before marking dead</literal></entry>
<entry>
<command>REINDEX CONCURRENTLY</command> is waiting for transactions
with read locks on the table to finish, before marking the old index dead.
This phase is skipped when not in concurrent mode.
Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
and <structname>current_locker_pid</structname> contain the progress
information for this phase.
</entry>
</row>
<row>
<entry><literal>waiting for readers before dropping</literal></entry>
<entry>
<command>REINDEX CONCURRENTLY</command> is waiting for transactions
with read locks on the table to finish, before dropping the old index.
This phase is skipped when not in concurrent mode.
Columns <structname>lockers_total</structname>, <structname>lockers_done</structname>
and <structname>current_locker_pid</structname> contain the progress
information for this phase.
</entry>
</row>
</tbody> </tbody>
</tgroup> </tgroup>
</table> </table>

10
src/backend/catalog/index.c
Unescape View File

@ -3286,12 +3286,20 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence,
heapId = IndexGetRelation(indexId, false); heapId = IndexGetRelation(indexId, false);
heapRelation = table_open(heapId, ShareLock); heapRelation = table_open(heapId, ShareLock);
pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
heapId);
pgstat_progress_update_param(PROGRESS_CREATEIDX_INDEX_OID,
indexId);
/* /*
* Open the target index relation and get an exclusive lock on it, to * Open the target index relation and get an exclusive lock on it, to
* ensure that no one else is touching this particular index. * ensure that no one else is touching this particular index.
*/ */
iRel = index_open(indexId, AccessExclusiveLock); iRel = index_open(indexId, AccessExclusiveLock);
pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID,
iRel->rd_rel->relam);
/* /*
* The case of reindexing partitioned tables and indexes is handled * The case of reindexing partitioned tables and indexes is handled
* differently by upper layers, so this case shouldn't arise. * differently by upper layers, so this case shouldn't arise.
@ -3442,6 +3450,8 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence,
errdetail_internal("%s", errdetail_internal("%s",
pg_rusage_show(&ru0)))); pg_rusage_show(&ru0))));
pgstat_progress_end_command();
/* Close rels, but keep locks */ /* Close rels, but keep locks */
index_close(iRel, NoLock); index_close(iRel, NoLock);
table_close(heapRelation, NoLock); table_close(heapRelation, NoLock);

9
src/backend/catalog/system_views.sql
Unescape View File

@ -946,16 +946,19 @@ CREATE VIEW pg_stat_progress_create_index AS
SELECT SELECT
S.pid AS pid, S.datid AS datid, D.datname AS datname, S.pid AS pid, S.datid AS datid, D.datname AS datname,
S.relid AS relid, S.relid AS relid,
CAST(S.param7 AS oid) AS index_relid,
CASE S.param10 WHEN 0 THEN 'initializing' CASE S.param10 WHEN 0 THEN 'initializing'
WHEN 1 THEN 'waiting for old snapshots' WHEN 1 THEN 'waiting for writers before build'
WHEN 2 THEN 'building index' || WHEN 2 THEN 'building index' ||
COALESCE((': ' || pg_indexam_progress_phasename(S.param9::oid, S.param11)), COALESCE((': ' || pg_indexam_progress_phasename(S.param9::oid, S.param11)),
'') '')
WHEN 3 THEN 'waiting for writer snapshots' WHEN 3 THEN 'waiting for writers before validation'
WHEN 4 THEN 'index validation: scanning index' WHEN 4 THEN 'index validation: scanning index'
WHEN 5 THEN 'index validation: sorting tuples' WHEN 5 THEN 'index validation: sorting tuples'
WHEN 6 THEN 'index validation: scanning table' WHEN 6 THEN 'index validation: scanning table'
WHEN 7 THEN 'waiting for reader snapshots' WHEN 7 THEN 'waiting for old snapshots'
WHEN 8 THEN 'waiting for readers before marking dead'
WHEN 9 THEN 'waiting for readers before dropping'
END as phase, END as phase,
S.param4 AS lockers_total, S.param4 AS lockers_total,
S.param5 AS lockers_done, S.param5 AS lockers_done,

41
src/backend/commands/indexcmds.c
Unescape View File

@ -478,6 +478,12 @@ DefineIndex(Oid relationId,
pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX, pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
relationId); relationId);
/*
* No index OID to report yet
*/
pgstat_progress_update_param(PROGRESS_CREATEIDX_INDEX_OID,
InvalidOid);
/* /*
* count key attributes in index * count key attributes in index
*/ */
@ -1244,6 +1250,12 @@ DefineIndex(Oid relationId,
CommitTransactionCommand(); CommitTransactionCommand();
StartTransactionCommand(); StartTransactionCommand();
/*
* The index is now visible, so we can report the OID.
*/
pgstat_progress_update_param(PROGRESS_CREATEIDX_INDEX_OID,
indexRelationId);
/* /*
* Phase 2 of concurrent index build (see comments for validate_index() * Phase 2 of concurrent index build (see comments for validate_index()
* for an overview of how this works) * for an overview of how this works)
@ -2873,6 +2885,13 @@ ReindexRelationConcurrently(Oid relationOid, int options)
heapRel = table_open(indexRel->rd_index->indrelid, heapRel = table_open(indexRel->rd_index->indrelid,
ShareUpdateExclusiveLock); ShareUpdateExclusiveLock);
pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX,
RelationGetRelid(heapRel));
pgstat_progress_update_param(PROGRESS_CREATEIDX_INDEX_OID,
indexId);
pgstat_progress_update_param(PROGRESS_CREATEIDX_ACCESS_METHOD_OID,
indexRel->rd_rel->relam);
/* Choose a temporary relation name for the new index */ /* Choose a temporary relation name for the new index */
concurrentName = ChooseRelationName(get_rel_name(indexId), concurrentName = ChooseRelationName(get_rel_name(indexId),
NULL, NULL,
@ -2967,7 +2986,9 @@ ReindexRelationConcurrently(Oid relationOid, int options)
* DefineIndex() for more details. * DefineIndex() for more details.
*/ */
WaitForLockersMultiple(lockTags, ShareLock, false); pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
PROGRESS_CREATEIDX_PHASE_WAIT_1);
WaitForLockersMultiple(lockTags, ShareLock, true);
CommitTransactionCommand(); CommitTransactionCommand();
forboth(lc, indexIds, lc2, newIndexIds) forboth(lc, indexIds, lc2, newIndexIds)
@ -3009,7 +3030,9 @@ ReindexRelationConcurrently(Oid relationOid, int options)
* for more details. * for more details.
*/ */
WaitForLockersMultiple(lockTags, ShareLock, false); pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
PROGRESS_CREATEIDX_PHASE_WAIT_2);
WaitForLockersMultiple(lockTags, ShareLock, true);
CommitTransactionCommand(); CommitTransactionCommand();
foreach(lc, newIndexIds) foreach(lc, newIndexIds)
@ -3057,7 +3080,9 @@ ReindexRelationConcurrently(Oid relationOid, int options)
* before the reference snap was taken, we have to wait out any * before the reference snap was taken, we have to wait out any
* transactions that might have older snapshots. * transactions that might have older snapshots.
*/ */
WaitForOlderSnapshots(limitXmin, false); pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
PROGRESS_CREATEIDX_PHASE_WAIT_3);
WaitForOlderSnapshots(limitXmin, true);
CommitTransactionCommand(); CommitTransactionCommand();
} }
@ -3128,7 +3153,9 @@ ReindexRelationConcurrently(Oid relationOid, int options)
* index_drop() for more details. * index_drop() for more details.
*/ */
WaitForLockersMultiple(lockTags, AccessExclusiveLock, false); pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
PROGRESS_CREATEIDX_PHASE_WAIT_4);
WaitForLockersMultiple(lockTags, AccessExclusiveLock, true);
foreach(lc, indexIds) foreach(lc, indexIds)
{ {
@ -3150,7 +3177,9 @@ ReindexRelationConcurrently(Oid relationOid, int options)
* Drop the old indexes. * Drop the old indexes.
*/ */
WaitForLockersMultiple(lockTags, AccessExclusiveLock, false); pgstat_progress_update_param(PROGRESS_CREATEIDX_PHASE,
PROGRESS_CREATEIDX_PHASE_WAIT_4);
WaitForLockersMultiple(lockTags, AccessExclusiveLock, true);
PushActiveSnapshot(GetTransactionSnapshot()); PushActiveSnapshot(GetTransactionSnapshot());
@ -3225,6 +3254,8 @@ ReindexRelationConcurrently(Oid relationOid, int options)
MemoryContextDelete(private_context); MemoryContextDelete(private_context);
pgstat_progress_end_command();
return true; return true;
} }

2
src/include/catalog/catversion.h
Unescape View File

@ -53,6 +53,6 @@
*/ */
/* yyyymmddN */ /* yyyymmddN */
#define CATALOG_VERSION_NO 201904071 #define CATALOG_VERSION_NO 201904072
#endif #endif

3
src/include/commands/progress.h
Unescape View File

@ -59,6 +59,7 @@
/* Progress parameters for CREATE INDEX */ /* Progress parameters for CREATE INDEX */
/* 3, 4 and 5 reserved for "waitfor" metrics */ /* 3, 4 and 5 reserved for "waitfor" metrics */
#define PROGRESS_CREATEIDX_INDEX_OID 6
#define PROGRESS_CREATEIDX_ACCESS_METHOD_OID 8 #define PROGRESS_CREATEIDX_ACCESS_METHOD_OID 8
#define PROGRESS_CREATEIDX_PHASE 9 /* AM-agnostic phase # */ #define PROGRESS_CREATEIDX_PHASE 9 /* AM-agnostic phase # */
#define PROGRESS_CREATEIDX_SUBPHASE 10 /* phase # filled by AM */ #define PROGRESS_CREATEIDX_SUBPHASE 10 /* phase # filled by AM */
@ -76,6 +77,8 @@
#define PROGRESS_CREATEIDX_PHASE_VALIDATE_SORT 5 #define PROGRESS_CREATEIDX_PHASE_VALIDATE_SORT 5
#define PROGRESS_CREATEIDX_PHASE_VALIDATE_TABLESCAN 6 #define PROGRESS_CREATEIDX_PHASE_VALIDATE_TABLESCAN 6
#define PROGRESS_CREATEIDX_PHASE_WAIT_3 7 #define PROGRESS_CREATEIDX_PHASE_WAIT_3 7
#define PROGRESS_CREATEIDX_PHASE_WAIT_4 8
#define PROGRESS_CREATEIDX_PHASE_WAIT_5 9
/* /*
* Subphases of CREATE INDEX, for index_build. * Subphases of CREATE INDEX, for index_build.

9
src/test/regress/expected/rules.out
Unescape View File

@ -1867,15 +1867,18 @@ pg_stat_progress_create_index| SELECT s.pid,
s.datid, s.datid,
d.datname, d.datname,
s.relid, s.relid,
(s.param7)::oid AS index_relid,
CASE s.param10 CASE s.param10
WHEN 0 THEN 'initializing'::text WHEN 0 THEN 'initializing'::text
WHEN 1 THEN 'waiting for old snapshots'::text WHEN 1 THEN 'waiting for writers before build'::text
WHEN 2 THEN ('building index'::text || COALESCE((': '::text || pg_indexam_progress_phasename((s.param9)::oid, s.param11)), ''::text)) WHEN 2 THEN ('building index'::text || COALESCE((': '::text || pg_indexam_progress_phasename((s.param9)::oid, s.param11)), ''::text))
WHEN 3 THEN 'waiting for writer snapshots'::text WHEN 3 THEN 'waiting for writers before validation'::text
WHEN 4 THEN 'index validation: scanning index'::text WHEN 4 THEN 'index validation: scanning index'::text
WHEN 5 THEN 'index validation: sorting tuples'::text WHEN 5 THEN 'index validation: sorting tuples'::text
WHEN 6 THEN 'index validation: scanning table'::text WHEN 6 THEN 'index validation: scanning table'::text
WHEN 7 THEN 'waiting for reader snapshots'::text WHEN 7 THEN 'waiting for old snapshots'::text
WHEN 8 THEN 'waiting for readers before marking dead'::text
WHEN 9 THEN 'waiting for readers before dropping'::text
ELSE NULL::text ELSE NULL::text
END AS phase, END AS phase,
s.param4 AS lockers_total, s.param4 AS lockers_total,

Write Preview
Loading…
Cancel
Save