63221 Commits (d46aa32ea5ce0c61a464cdc2c74fa9a428df8bc1)

 

All Branches   

Search

Author	SHA1	Message	Date
Michael Paquier	d46aa32ea5	Fix build inconsistency due to the generation of wait-event code ... The build generates four files based on the wait event contents stored in wait_event_names.txt: - wait_event_types.h - pgstat_wait_event.c - wait_event_funcs_data.c - wait_event_types.sgml The SGML file is generated as part of a documentation build, with its data stored in doc/src/sgml/ for meson and configure. The three others are handled differently for meson and configure: - In configure, all the files are created in src/backend/utils/activity/. A link to wait_event_types.h is created in src/include/utils/. - In meson, all the files are created in src/include/utils/. The two C files, pgstat_wait_event.c and wait_event_funcs_data.c, are then included in respectively wait_event.c and wait_event_funcs.c, without the "utils/" path. For configure, this does not present a problem. For meson, this has to be combined with a trick in src/backend/utils/activity/meson.build, where include_directories needs to point to include/utils/ to make the inclusion of the C files work properly, causing builds to pull in PostgreSQL headers rather than system headers in some build paths, as src/include/utils/ would take priority. In order to fix this issue, this commit reworks the way the C/H files are generated, becoming consistent with guc_tables.inc.c: - For meson, basically nothing changes. The files are still generated in src/include/utils/. The trick with include_directories is removed. - For configure, the files are now generated in src/backend/utils/, with links in src/include/utils/ pointing to the ones in src/backend/. This requires extra rules in src/backend/utils/activity/Makefile so as a make command in this sub-directory is able to work. - The three files now fall under header-stamp, which is actually simpler as guc_tables.inc.c does the same. - wait_event_funcs_data.c and pgstat_wait_event.c are now included with "utils/" in their path. This problem has not been an issue in the buildfarm; it has been noted with AIX and a conflict with float.h. This issue could, however, create conflicts in the buildfarm depending on the environment with unexpected headers pulled in, so this fix is backpatched down to where the generation of the wait-event files has been introduced. While on it, this commit simplifies wait_event_names.txt regarding the paths of the files generated, to mention just the names of the files generated. The paths where the files are generated became incorrect. The path of the SGML path was wrong. This change has been tested in the CI, down to v17. Locally, I have run tests with configure (with and without VPATH), as well as meson, on the three branches. Combo oversight in fa88928470 and 1e68e43d3f. Reported-by: Aditya Kamath <aditya.kamath1@ibm.com> Discussion: https://postgr.es/m/LV8PR15MB64888765A43D229EA5D1CFE6D691A@LV8PR15MB6488.namprd15.prod.outlook.com Backpatch-through: 17	4 months ago
Tom Lane	6918434a4a	Make psql/t/030_pager.pl more robust. ... Similarly to the preceding commit, 030_pager.pl was assuming that patterns it looks for in interactive psql output would appear by themselves on a line, but that assumption tends to fall over in builds made --without-readline: the output we get might have a psql prompt immediately followed by the expected line of output. For several of these tests, just checking for the pattern followed by newline seems sufficient, because we could not get a false match against the command echo, nor against the unreplaced command output if the pager fails to be invoked when expected. However, that's fairly scary for the test that was relying on information_schema.referential_constraints: "\d+" could easily appear at the end of a line in that view. Let's get rid of that hazard by making a custom test view instead of using information_schema.referential_constraints. This test script is new in v19, so no need for back-patch. Reported-by: Oleg Tselebrovskiy <o.tselebrovskiy@postgrespro.ru> Author: Oleg Tselebrovskiy <o.tselebrovskiy@postgrespro.ru> Co-authored-by: Tom Lane <tgl@sss.pgh.pa.us> Reviewed-by: Soumya S Murali <soumyamurali.work@gmail.com> Discussion: https://postgr.es/m/db6fdb35a8665ad3c18be01181d44b31@postgrespro.ru	4 months ago
Tom Lane	a1d7ae2b2e	Improve guards against false regex matches in BackgroundPsql.pm. ... BackgroundPsql needs to wait for all the output from an interactive psql command to come back. To make sure that's happened, it issues the command, then issues \echo and \warn psql commands that echo a "banner" string (which we assume won't appear in the command's output), then waits for the banner strings to appear. The hazard in this approach is that the banner will also appear in the echoed psql commands themselves, so we need to distinguish those echoes from the desired output. Commit 8b886a4e3 tried to do that by positing that the desired output would be directly preceded and followed by newlines, but it turns out that that assumption is timing-sensitive. In particular, it tends to fail in builds made --without-readline, wherein the command echoes will be made by the pty driver and may be interspersed with prompts issued by psql proper. It does seem safe to assume that the banner output we want will be followed by a newline, since that should be the last output before things quiesce. Therefore, we can improve matters by putting quotes around the banner strings in the \echo and \warn psql commands, so that their echoes cannot include banner directly followed by newline, and then checking for just banner-and-newline in the match pattern. While at it, spruce up the pump() call in sub query() to look like the neater version in wait_connect(), and don't die on timeout until after printing whatever we got. Reported-by: Oleg Tselebrovskiy <o.tselebrovskiy@postgrespro.ru> Diagnosed-by: Oleg Tselebrovskiy <o.tselebrovskiy@postgrespro.ru> Author: Tom Lane <tgl@sss.pgh.pa.us> Reviewed-by: Soumya S Murali <soumyamurali.work@gmail.com> Discussion: https://postgr.es/m/db6fdb35a8665ad3c18be01181d44b31@postgrespro.ru Backpatch-through: 14	4 months ago
Heikki Linnakangas	e2362eb2bd	Move shmem allocator's fields from PGShmemHeader to its own struct ... For readability. It was a slight modularity violation to have fields in PGShmemHeader that were only used by the allocator code in shmem.c. And it was inconsistent that ShmemLock was nevertheless not stored there. Moving all the allocator-related fields to a separate struct makes it more consistent and modular, and removes the need to allocate and pass ShmemLock separately via BackendParameters. Merge InitShmemAccess() and InitShmemAllocation() into a single function that initializes the struct when called from postmaster, and when called from backends in EXEC_BACKEND mode, re-establishes the global variables. That's similar to all the *ShmemInit() functions that we have. Co-authored-by: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Discussion: https://www.postgresql.org/message-id/CAExHW5uNRB9oT4pdo54qAo025MXFX4MfYrD9K15OCqe-ExnNvg@mail.gmail.com	4 months ago
Álvaro Herrera	e76221bd95	Minor cosmetic tweaks ... These changes should have been done by 2f9661311b, but were overlooked. I noticed while reviewing the code for commit b8926a5b4b. Author: Álvaro Herrera <alvherre@kurilemu.de> Discussion: https://postgr.es/m/18984-0f4778a6599ac3ae@postgresql.org	4 months ago
Álvaro Herrera	1eb09ed63a	Use C99 designated designators in a couple of places ... This makes the arrays somewhat easier to read. Author: Álvaro Herrera <alvherre@kurilemu.de> Reviewed-by: Peter Eisentraut <peter@eisentraut.org> Reviewed-by: Melanie Plageman <melanieplageman@gmail.com> Reviewed-by: Jelte Fennema-Nio <postgres@jeltef.nl> Discussion: https://postgr.es/m/202601281204.sdxbr5qvpunk@alvherre.pgsql	4 months ago
Fujii Masao	bb26a81ee2	Remove unused argument from ApplyLogicalMappingFile(). ... Author: Yugo Nagata <nagata@sraoss.co.jp> Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com> Discussion: https://postgr.es/m/20260128120056.b2a3e8184712ab5a537879eb@sraoss.co.jp	4 months ago
Andres Freund	87f7b824f2	tableam: Perform CheckXidAlive check once per scan ... Previously, the CheckXidAlive check was performed within the table_scan*next* functions. This caused the check to be executed for every fetched tuple, an unnecessary overhead. To fix, move the check to table_beginscan* so it is performed once per scan rather than once per row. Note: table_tuple_fetch_row_version() does not use a scan descriptor; therefore, the CheckXidAlive check is retained in that function. The overhead is unlikely to be relevant for the existing callers. Reported-by: Andres Freund <andres@anarazel.de> Author: Dilip Kumar <dilipbalaut@gmail.com> Suggested-by: Andres Freund <andres@anarazel.de> Suggested-by: Amit Kapila <akapila@postgresql.org> Reviewed-by: Andres Freund <andres@anarazel.de> Discussion: https://www.postgresql.org/message-id/tlpltqm5jjwj7mp66dtebwwhppe4ri36vdypux2zoczrc2i3mp%40dhv4v4nikyfg	4 months ago
Andres Freund	333f586372	bufmgr: Allow conditionally locking of already locked buffer ... In fcb9c977aa I included an assertion in BufferLockConditional() to detect if a conditional lock acquisition is done on a buffer that we already have locked. The assertion was added in the course of adding other assertions. Unfortunately I failed to realize that some of our code relies on such lock acquisitions to silently fail. E.g. spgist and nbtree may try to conditionally lock an already locked buffer when acquiring a empty buffer. LWLockAcquireConditional(), which was previously used to implement ConditionalLockBuffer(), does not have such an assert. Instead of just removing the assert, and relying on the lock acquisition to fail due to the buffer already locked, this commit changes the behaviour of conditional content lock acquisition to fail if the current backend has any pre-existing lock on the buffer, even if the lock modes would not conflict. The reason for that is that we currently do not have space to track multiple lock acquisitions on a single buffer. Allowing multiple locks on the same buffer by a backend also seems likely to lead to bugs. There is only one non-self-exclusive conditional content lock acquisition, in GetVictimBuffer(), but it only is used if the target buffer is not pinned and thus can't already be locked by the current backend. Reported-by: Alexander Lakhin <exclusion@gmail.com> Discussion: https://postgr.es/m/90bd2cbb-49ce-4092-9f61-5ac2ab782c94@gmail.com	4 months ago
Tom Lane	bd9dfac8b1	Further fix extended alignment for older g++. ... Commit 6ceef9408 was still one brick shy of a load, because it caused any usage at all of PGIOAlignedBlock or PGAlignedXLogBlock to fail under older g++. Notably, this broke "headerscheck --cplusplus". We can permit references to these structs as abstract structs though; only actual declaration of such a variable needs to be forbidden. Discussion: https://www.postgresql.org/message-id/3119480.1769189606@sss.pgh.pa.us	4 months ago
Jeff Davis	de90bb7db1	Fix theoretical memory leaks in pg_locale_libc.c. ... The leaks were hard to reach in practice and the impact was low. The callers provide a buffer the same number of bytes as the source string (plus one for NUL terminator) as a starting size, and libc never increases the number of characters. But, if the byte length of one of the converted characters is larger, then it might need a larger destination buffer. Previously, in that case, the working buffers would be leaked. Even in that case, the call typically happens within a context that will soon be reset. Regardless, it's worth fixing to avoid such assumptions, and the fix is simple so it's worth backporting. Discussion: https://postgr.es/m/e2b7a0a88aaadded7e2d19f42d5ab03c9e182ad8.camel@j-davis.com Backpatch-through: 18	4 months ago
Álvaro Herrera	ec31744071	Replace literal 0 with InvalidXLogRecPtr for XLogRecPtr assignments ... Use the proper constant InvalidXLogRecPtr instead of literal 0 when assigning XLogRecPtr variables and struct fields. This improves code clarity by making it explicit that these are invalid LSN values rather than ambiguous zero literals. Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com> Discussion: https://postgr.es/m/aRtd2dw8FO1nNX7k@ip-10-97-1-34.eu-west-3.compute.internal	4 months ago
Robert Haas	71c1136989	Fix mistakes in commit 4020b370f2 ... cost_tidrangescan() was setting the disabled_nodes value correctly, and then immediately resetting it to zero, due to poor code editing on my part. materialized_finished_plan correctly set matpath.parent to zero, but forgot to also set matpath.parallel_workers = 0, causing an access to uninitialized memory in cost_material. (This shouldn't result in any real problem, but it makes valgrind unhappy.) reparameterize_path was dereferencing a variable before verifying that it was not NULL. Reported-by: Tom Lane <tgl@sss.pgh.pa.us> (issue #1) Reported-by: Michael Paquier <michael@paquier.xyz> (issue #1) Diagnosed-by: Lukas Fittl <lukas@fittl.com> (issue #1) Reported-by: Zsolt Parragi <zsolt.parragi@percona.com> (issue #2) Reported-by: Richard Guo <guofenglinux@gmail.com> (issue #3) Discussion: http://postgr.es/m/CAN4CZFPvwjNJEZ_JT9Y67yR7C=KMNa=LNefOB8ZY7TKDcmAXOA@mail.gmail.com Discussion: http://postgr.es/m/aXrnPgrq6Gggb5TG@paquier.xyz	4 months ago
Alexander Korotkov	20a8f783e1	Wake LSN waiters before recovery target stop ... Move WaitLSNWakeup() immediately after ApplyWalRecord() so waiters are signaled even when recoveryStopsAfter() breaks out for pause/promotion targets. Discussion: https://postgr.es/m/9533608f-f289-44bd-b881-9e5a73203c5b%40iki.fi Discussion: https://postgr.es/m/CABPTF7Wdq6KbvC3EhLX3Pz%3DODCCPEX7qVQ%2BE%3DcokkB91an2E-A%40mail.gmail.com Reported-by: Heikki Linnakangas <hlinnaka@iki.fi> Author: Xuneng Zhou <xunengzhou@gmail.com>	4 months ago
Michael Paquier	4b77282f25	psql: Disable %P (pipeline status) for non-active connection ... In the psql prompt, %P prompt shows the current pipeline status. Unlike most of the other options, its status was showing up in the output generated even if psql was not connected to a database. This was confusing, because without a connection a pipeline status makes no sense. Like the other options, %P is updated so as its data is now hidden without an active connection. Author: Chao Li <li.evan.chao@gmail.com> Discussion: https://postgr.es/m/86EF76B5-6E62-404D-B9EC-66F4714D7D5F@gmail.com Backpatch-through: 18	4 months ago
Michael Paquier	740a1494f4	Fix two error messages in extended_stats_funcs.c ... These have been fat-fingered in 0e80f3f88d and 302879bd68. The error message for ndistinct had an incorrect grammar, while the one for dependencies had finished with a period (incorrect based on the project guidelines). Discussion: https://postgr.es/m/aXrsjZQbVuB6236u@paquier.xyz	4 months ago
Michael Paquier	fc365e4fcc	Add test doing some cloning of extended statistics data ... The test added in this commit copies the data of an ANALYZE run on one relation to a secondary relation with the same attribute definitions and extended statistics objects. Once the clone is done, the target and origin should have the same extended statistics information, with no differences. This test would have been able to catch e3094679b9, for example, as we expect the full range of statistics to be copied over, with no differences generated between the results of an ANALYZE and the data copied to the cloned relation. Note that this new test should remain at the bottom of stats_import.sql, so as any additions in the main relation and its clone are automatically covered when copying their statistics, so as it would work as a sanity check in the future. Author: Corey Huinker <corey.huinker@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/CADkLM=dpz3KFnqP-dgJ-zvRvtjsa8UZv8wDAQdqho=qN3kX0Zg@mail.gmail.com	4 months ago
Michael Paquier	0b7beec42a	Add test for pg_restore_extended_stats() with multiranges ... The restore of extended statistics has some paths dedicated to multirange types and expressions for all the stats kinds supported, and we did not have coverage for the case where an extended stats object uses a multirange attribute with or without an expression. Extracted from a larger patch by the same author, with a couple of tweaks from me regarding the format of the output generated, to make it more readable to the eye. Author: Corey Huinker <corey.huinker@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/CADkLM=dpz3KFnqP-dgJ-zvRvtjsa8UZv8wDAQdqho=qN3kX0Zg@mail.gmail.com	4 months ago
Amit Kapila	3a98f989e8	Fix CI failure introduced in commit 851f6649cc. ... The test added in commit 851f6649cc uses a backup taken from a node created by the previous test to perform standby related checks. On Windows, however, the standby failed to start with the following error: FATAL: could not rename file "backup_label" to "backup_label.old": Permission denied This occurred because some background sessions from the earlier test were still active. These leftover processes continued accessing the parent directory of the backup_label file, likely preventing the rename and causing the failure. Ensuring that these sessions are cleanly terminated resolves the issue in local testing. Additionally, the has_restoring => 1 option has been removed, as it was not required by the new test. Reported-by: Robert Haas <robertmhaas@gmail.com> Backpatch-through: 17 Discussion: https://postgr.es/m/CA+TgmobdVhO0ckZfsBZ0wqDO4qHVCwZZx8sf=EinafvUam-dsQ@mail.gmail.com	4 months ago
Michael Paquier	efbebb4e85	Add support for "mcv" in pg_restore_extended_stats() ... This commit adds support for the restore of extended statistics of the kind "mcv", aka most-common values. This format is different from n_distinct and dependencies stat types in that it is the combination of three of the four different arrays from the pg_stats_ext view which in turn require three different input parameters on pg_restore_extended_statistics(). These are translated into three input arguments for the function: - "most_common_vals", acting as a leader of the others. It is a 2-dimension array, that includes the common values. - "most_common_freqs", 1-dimension array of float8[], with a number of elements that has to match with "most_common_vals". - "most_common_base_freqs", 1-dimension array of float8[], with a number of elements that has to match with "most_common_vals". All three arrays are required to achieve the restore of this type of extended statistics (if "most_common_vals" happens to be NULL in the catalogs, the rest is NULL by design). Note that "most_common_val_nulls" is not required in input, its data is rebuilt from the decomposition of the "most_common_vals" array based on its text[] representation. The initial versions of the patch provided this option in input, but we do not require it and it simplifies a lot the result. Support in pg_dump is added down to v13 which is where the support for this type of extended statistics has been added, when --statistics is used. This means that upgrade and dumps can restore extended statistics data transparently, like "dependencies", "ndistinct", attribute and relation statistics. For MCV, the values are directly queried from the relevant catalogs. Author: Corey Huinker <corey.huinker@gmail.com> Co-authored-by: Chao Li <li.evan.chao@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/CADkLM=dpz3KFnqP-dgJ-zvRvtjsa8UZv8wDAQdqho=qN3kX0Zg@mail.gmail.com	4 months ago
Michael Paquier	e09e5ad69a	Fix whitespace issue in regression test stats_import ... Issue noticed while playing this area of the tests for a different patch.	4 months ago
Nathan Bossart	ef1c865206	Add a couple of recent commits to .git-blame-ignore-revs.	4 months ago
Masahiko Sawada	8f1e2dfe03	Consolidate replication origin session globals into a single struct. ... This commit moves the separate global variables for replication origin state into a single ReplOriginXactState struct. This groups logically related variables, which improves code readability and simplifies state management (e.g., resetting the state) by handling them as a unit. Author: Chao Li <lic@highgo.com> Suggested-by: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com> Reviewed-by: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Discussion: https://postgr.es/m/CAEoWx2=pYvfRthXHTzSrOsf5_FfyY4zJyK4zV2v4W=yjUij1cA@mail.gmail.com	4 months ago
Masahiko Sawada	227eb4eea2	Refactor replication origin state reset helpers. ... Factor out common logic for clearing replorigin_session_* variables into a dedicated helper function, replorigin_xact_clear(). This removes duplicated assignments of these variables across multiple call sites, and makes the intended scope of each reset explicit. Author: Chao Li <lic@highgo.com> Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com> Reviewed-by: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Álvaro Herrera <alvherre@kurilemu.de> Discussion: https://postgr.es/m/CAEoWx2=pYvfRthXHTzSrOsf5_FfyY4zJyK4zV2v4W=yjUij1cA@mail.gmail.com	4 months ago
Masahiko Sawada	1fdbca159e	Standardize replication origin naming to use "ReplOrigin". ... The replication origin code was using inconsistent naming conventions. Functions were typically prefixed with 'replorigin', while typedefs and constants used "RepOrigin". This commit unifies the naming convention by renaming RepOriginId to ReplOriginId. Reviewed-by: Chao Li <li.evan.chao@gmail.com> Discussion: https://postgr.es/m/CAD21AoBDgm3hDqUZ+nqu=ViHmkCnJBuJyaxG_yvv27BAi2zBmQ@mail.gmail.com	4 months ago
Robert Haas	4020b370f2	Allow for plugin control over path generation strategies. ... Each RelOptInfo now has a pgs_mask member which is a mask of acceptable strategies. For most rels, this is populated from PlannerGlobal's default_pgs_mask, which is computed from the values of the enable_* GUCs at the start of planning. For baserels, get_relation_info_hook can be used to adjust pgs_mask for each new RelOptInfo, at least for rels of type RTE_RELATION. Adjusting pgs_mask is less useful for other types of rels, but if it proves to be necessary, we can revisit the way this hook works or add a new one. For joinrels, two new hooks are added. joinrel_setup_hook is called each time a joinrel is created, and one thing that can be done from that hook is to manipulate pgs_mask for the new joinrel. join_path_setup_hook is called each time we're about to add paths to a joinrel by considering some particular combination of an outer rel, an inner rel, and a join type. It can modify the pgs_mask propagated into JoinPathExtraData to restrict strategy choice for that particular combination of rels. To make joinrel_setup_hook work as intended, the existing calls to build_joinrel_partition_info are moved later in the calling functions; this is because that function checks whether the rel's pgs_mask includes PGS_CONSIDER_PARTITIONWISE, so we want it to only be called after plugins have had a chance to alter pgs_mask. Upper rels currently inherit pgs_mask from the input relation. It's unclear that this is the most useful behavior, but at the moment there are no hooks to allow the mask to be set in any other way. Reviewed-by: Lukas Fittl <lukas@fittl.com> Reviewed-by: Jakub Wartak <jakub.wartak@enterprisedb.com> Reviewed-by: Greg Burd <greg@burd.me> Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com> Reviewed-by: Amit Langote <amitlangote09@gmail.com> Reviewed-by: Haibo Yan <tristan.yim@gmail.com> Discussion: http://postgr.es/m/CA+TgmoZ-Jh1T6QyWoCODMVQdhTUPYkaZjWztzP1En4=ZHoKPzw@mail.gmail.com	4 months ago
Álvaro Herrera	e6d6e32f42	Fix duplicate arbiter detection during REINDEX CONCURRENTLY on partitions ... Commit 90eae926a fixed ON CONFLICT handling during REINDEX CONCURRENTLY on partitioned tables by treating unparented indexes as potential arbiters. However, there's a remaining race condition: when pg_inherits records are swapped between consecutive calls to get_partition_ancestors(), two different child indexes can appear to have the same parent, causing duplicate entries in the arbiter list and triggering "invalid arbiter index list" errors. Note that this is not a new problem introduced by 90eae926a. The same error could occur before that commit in a slightly different scenario: an index is selected during planning, then index_concurrently_swap() commits, and a subsequent call to get_partition_ancestors() uses a new catalog snapshot that sees zero ancestors for that index. Fix by tracking which parent indexes have already been processed. If a subsequent call to get_partition_ancestors() returns a parent we've already seen, treat that index as unparented instead, allowing it to be matched via IsIndexCompatibleAsArbiter() like other concurrent reindex scenarios. Author: Mihail Nikalayeu <mihailnikalayeu@gmail.com> Reported-by: Alexander Lakhin <exclusion@gmail.com> Reviewed-by: Álvaro Herrera <alvherre@kurilemu.de> Discussion: https://postgr.es/m/e5a8c1df-04e5-4343-85ef-5df2a7e3d90c@gmail.com	4 months ago
Michael Paquier	e3094679b9	Fix pg_restore_extended_stats() with expressions ... This commit fixes an issue with the restore of ndistinct and dependencies statistics, causing the operation to fail when any of these kinds included expressions. In extended statistics, expressions use strictly negative attribute numbers, decremented from -1. For example, let's imagine an object defined as follows: CREATE STATISTICS stats_obj (dependencies) ON lower(name), upper(name) FROM tab_obj; This object would generate dependencies stats using -1 and -2 as attribute numbers, like that: [{"attributes": [-1], "dependency": -2, "degree": 1.000000}, {"attributes": [-2], "dependency": -1, "degree": 1.000000}] However, pg_restore_extended_stats() forgot to account for the number of expressions defined in an extended statistics object. This would cause the validation step of ndistinct and dependencies data to fail, preventing a restore of their stats even if the input is valid. This issue has come up due to an incorrect split of the patch set. Some tests are included to cover this behavior. Author: Corey Huinker <corey.huinker@gmail.com> Co-authored-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/aXl4bMfSTQUxM_yy@paquier.xyz	4 months ago
Michael Paquier	f9562b95c6	Add output test for pg_dependencies statistics import ... Commit 302879bd68 has added the ability to restore extended stats of the type "dependencies", but it has forgotten the addition of a test to verify that the value restored was actually set. This test is the pg_dependencies equivalent of the test added for pg_ndistinct in 0e80f3f88d. Author: Corey Huinker <corey.huinker@gmail.com> Discussion: https://postgr.es/m/CADkLM=dZr_Ut3jKw94_BisyyDtNZPRJWeOALXVzcJz=ZFTAhvQ@mail.gmail.com	4 months ago
Jacob Champion	c6a10a89fe	oauth: Correct test dependency on oauth_hook_client ... The oauth_validator tests missed the lessons of c89525d57 et al, so certain combinations of command-line build order and `meson test` options can result in Command 'oauth_hook_client' not found in [...] at src/test/perl/PostgreSQL/Test/Utils.pm line 427. Add the missing dependency on the test executable. This fixes, for example, $ ninja clean && ninja meson-test-prereq && PG_TEST_EXTRA=oauth meson test --no-rebuild Reported-by: Jonathan Gonzalez V. <jonathan.abdiel@gmail.com> Author: Jonathan Gonzalez V. <jonathan.abdiel@gmail.com> Discussion: https://postgr.es/m/6e8f4f7c23faf77c4b6564c4b7dc5d3de64aa491.camel@gmail.com Discussion: https://postgr.es/m/qh4c5tvkgjef7jikjig56rclbcdrrotngnwpycukd2n3k25zi2%4044hxxvtwmgum Backpatch-through: 18	4 months ago
Robert Haas	9a446d0256	pg_waldump: Remove file-level global WalSegSz. ... It's better style to pass the value around to just the places that need it. This makes it easier to determine whether the value is always properly initialized before use. Reviewed-by: Amul Sul <sulamul@gmail.com> Discussion: http://postgr.es/m/CAAJ_b94+wObPn-z1VECipnSFhjMJ+R2cpTmKVYLjyQuVn+B5QA@mail.gmail.com	4 months ago
Amit Kapila	851f6649cc	Prevent invalidation of newly synced replication slots. ... A race condition could cause a newly synced replication slot to become invalidated between its initial sync and the checkpoint. When syncing a replication slot to a standby, the slot's initial restart_lsn is taken from the publisher's remote_restart_lsn. Because slot sync happens asynchronously, this value can lag behind the standby's current redo pointer. Without any interlocking between WAL reservation and checkpoints, a checkpoint may remove WAL required by the newly synced slot, causing the slot to be invalidated. To fix this, we acquire ReplicationSlotAllocationLock before reserving WAL for a newly synced slot, similar to commit 006dd4b2e5. This ensures that if WAL reservation happens first, the checkpoint process must wait for slotsync to update the slot's restart_lsn before it computes the minimum required LSN. However, unlike in ReplicationSlotReserveWal(), this lock alone cannot protect a newly synced slot if a checkpoint has already run CheckPointReplicationSlots() before slotsync updates the slot. In such cases, the remote restart_lsn may be stale and earlier than the current redo pointer. To prevent relying on an outdated LSN, we use the oldest WAL location available if it is greater than the remote restart_lsn. This ensures that newly synced slots always start with a safe, non-stale restart_lsn and are not invalidated by concurrent checkpoints. Author: Zhijie Hou <houzj.fnst@fujitsu.com> Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com> Reviewed-by: Amit Kapila <amit.kapila16@gmail.com> Reviewed-by: Vitaly Davydov <v.davydov@postgrespro.ru> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Backpatch-through: 17 Discussion: https://postgr.es/m/TY4PR01MB16907E744589B1AB2EE89A31F94D7A%40TY4PR01MB16907.jpnprd01.prod.outlook.com	4 months ago
Michael Paquier	c32fb29e97	Include extended statistics data in pg_dump ... This commit integrates the new pg_restore_extended_stats() function into pg_dump, so as the data of extended statistics is detected and included in dumps when the --statistics switch is specified. Currently, the same extended stats kinds as the ones supported by the SQL function can be dumped: "n_distinct" and "dependencies". The extended statistics data can be dumped down to PostgreSQL 10, with the following changes depending on the backend version dealt with: - In v19 and newer versions, the format of pg_ndistinct and pg_dependencies has changed, catalogs can be directly queried. - In v18 and older versions, the format is translated to the new format supported by the backend. - In v14 and older versions, inherited extended statistics are not supported. - In v11 and older versions, the data for ndistinct and dependencies was stored in pg_statistic_ext. These have been moved to pg_stats_ext in v12. - Extended Statistics have been introduced in v10, no support is needed for versions older than that. The extended statistics data is dumped if it can be found in the catalogs. If the catalogs are empty, then no restore of the stats data is attempted. Author: Corey Huinker <corey.huinker@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/CADkLM=dpz3KFnqP-dgJ-zvRvtjsa8UZv8wDAQdqho=qN3kX0Zg@mail.gmail.com	4 months ago
Fujii Masao	1ea44d7ddf	Remove unnecessary abort() from WalSndShutdown(). ... WalSndShutdown() previously called abort() after proc_exit(0) to silence compiler warnings. This is no longer needed, because both WalSndShutdown() and proc_exit() are declared pg_noreturn, allowing the compiler to recognize that the function does not return. Also there are already other functions, such as CheckpointerMain(), that call proc_exit() without an abort(), and they do not produce warnings. Therefore this abort() call in WalSndShutdown() is useless and this commit removes it. Author: Fujii Masao <masao.fujii@gmail.com> Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi> Discussion: https://postgr.es/m/CAHGQGwHPX1yoixq+YB5rF4zL90TMmSEa3FpHURtqW3Jc5+=oSA@mail.gmail.com	4 months ago
Tomas Vondra	09c37015d4	pgindent fix for 3fccbd94cb ... Backpatch-through: 18	4 months ago
Tomas Vondra	3fccbd94cb	Handle ENOENT status when querying NUMA node ... We've assumed that touching the memory is sufficient for a page to be located on one of the NUMA nodes. But a page may be moved to a swap after we touch it, due to memory pressure. We touch the memory before querying the status, but there is no guarantee it won't be moved to the swap in the meantime. The touching happens only on the first call, so later calls are more likely to be affected. And the batching increases the window too. It's up to the kernel if/when pages get moved to swap. We have to accept ENOENT (-2) as a valid result, and handle it without failing. This patch simply treats it as an unknown node, and returns NULL in the two affected views (pg_shmem_allocations_numa and pg_buffercache_numa). Hugepages cannot be swapped out, so this affects only regular pages. Reported by Christoph Berg, investigation and fix by me. Backpatch to 18, where the two views were introduced. Reported-by: Christoph Berg <myon@debian.org> Discussion: 18 Backpatch-through: https://postgr.es/m/aTq5Gt_n-oS_QSpL@msg.df7cb.de	4 months ago
Michael Paquier	302879bd68	Add support for "dependencies" in pg_restore_extended_stats() ... This commit adds support for the restore of extended statistics of the kind "dependencies", for the following input data: [{"attributes": [2], "dependency": 3, "degree": 1.000000}, {"attributes": [3], "dependency": 2, "degree": 1.000000}] This relies on the existing routines of "dependencies" to cross-check the input data with the definition of the extended statistics objects for the attribute numbers. An input argument of type "pg_dependencies" is required for this new option. Thanks to the work done in 0e80f3f88d for the restore function and e1405aa5e3 for the input handling of data type pg_dependencies, this addition is straight-forward. This will be used so as it is possible to transfer these statistics across dumps and upgrades, removing the need for a post-operation ANALYZE for these kinds of statistics. Author: Corey Huinker <corey.huinker@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/CADkLM=dpz3KFnqP-dgJ-zvRvtjsa8UZv8wDAQdqho=qN3kX0Zg@mail.gmail.com	4 months ago
Melanie Plageman	19af794b66	Refactor lazy_scan_prune() VM clear logic into helper ... Encapsulating the cases that clear the visibility map after vacuum phase I, when corruption is detected, into in a helper makes the code cleaner and enables further refactoring in future commits. Author: Melanie Plageman <melanieplageman@gmail.com> Reviewed-by: Andres Freund <andres@anarazel.de> Reviewed-by: Kirill Reshke <reshkekirill@gmail.com> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru> Discussion: https://postgr.es/m/7ib3sa55sapwjlaz4sijbiq7iezna27kjvvvar4dpgkmadml6t%40gfpkkwmdnepx	4 months ago
Melanie Plageman	648a7e28d7	Eliminate use of cached VM value in lazy_scan_prune() ... lazy_scan_prune() takes a parameter from lazy_scan_heap() indicating whether the page was marked all-visible in the VM at the time it was last checked in find_next_unskippable_block(). This behavior is historical, dating back to commit 608195a3a3, when we did not pin the VM page until deciding we must read it. Now that the VM page is already pinned, there is no meaningful benefit to relying on a cached VM status. Removing this cached value simplifies the logic in both lazy_scan_heap() and lazy_scan_prune(). It also clarifies future work that will set the visibility map on-access: such paths will not have a cached value available, which would make the logic harder to reason about. And eliminating it enables us to detect and repair VM corruption on-access. Along with removing the cached value and unconditionally checking the visibility status of the heap page, this commit also moves the VM corruption handling to occur first. This reordering should have no performance impact, since the checks are inexpensive and performed only once per page. It does, however, make the control flow easier to understand. The new restructuring also makes it possible to set the VM after fixing corruption (if pruning found the page all-visible). Now that no callers of visibilitymap_set() use its return value, change its (and visibilitymap_set_vmbits()) return type to void. Author: Melanie Plageman <melanieplageman@gmail.com> Reviewed-by: Andres Freund <andres@anarazel.de> Reviewed-by: Kirill Reshke <reshkekirill@gmail.com> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru> Reviewed-by: Xuneng Zhou <xunengzhou@gmail.com> Discussion: https://postgr.es/m/5CEAA162-67B1-44DA-B60D-8B65717E8B05%40gmail.com	4 months ago
Melanie Plageman	21796c267d	Combine visibilitymap_set() cases in lazy_scan_prune() ... lazy_scan_prune() previously had two separate cases that called visibilitymap_set() after pruning and freezing. These branches were nearly identical except that one attempted to avoid dirtying the heap buffer. However, that situation can never occur — the heap buffer cannot be clean at that point (and we would hit an assertion if it were). In lazy_scan_prune(), when we change a previously all-visible page to all-frozen and the page was recorded as all-visible in the visibility map by find_next_unskippable_block(), the heap buffer will always be dirty. Either we have just frozen a tuple and already dirtied the buffer, or the buffer was modified between find_next_unskippable_block() and heap_page_prune_and_freeze() and then pruned in heap_page_prune_and_freeze(). Additionally, XLogRegisterBuffer() asserts that the buffer is dirty, so attempting to add a clean heap buffer to the WAL chain would assert out anyway. Since the “clean heap buffer with already set VM” case is impossible, the two visibilitymap_set() branches in lazy_scan_prune() can be merged. Doing so makes the intent clearer and emphasizes that the heap buffer must always be marked dirty before being added to the WAL chain. This commit also adds a test case for vacuuming when no heap modifications are required. Currently this ensures that the heap buffer is marked dirty before it is added to the WAL chain, but if we later remove the heap buffer from the VM-set WAL chain or pass it with the REGBUF_NO_CHANGES flag, this test would guard that behavior. Author: Melanie Plageman <melanieplageman@gmail.com> Reviewed-by: Andres Freund <andres@anarazel.de> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Reviewed-by: Srinath Reddy Sadipiralla <srinath2133@gmail.com> Reviewed-by: Kirill Reshke <reshkekirill@gmail.com> Reviewed-by: Xuneng Zhou <xunengzhou@gmail.com> Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru> Discussion: https://postgr.es/m/5CEAA162-67B1-44DA-B60D-8B65717E8B05%40gmail.com Discussion: https://postgr.es/m/flat/CAAKRu_ZWx5gCbeCf7PWCv8p5%3D%3Db7EEws0VD2wksDxpXCvCyHvQ%40mail.gmail.com	4 months ago
Tomas Vondra	f6e5d21bf7	Exercise parallel GIN builds in regression tests ... Modify two places creating GIN indexes in regression tests, so that the build is parallel. This provides a basic test coverage, even if the amounts of data are fairly small. Reported-by: Kirill Reshke <reshkekirill@gmail.com> Backpatch-through: 18 Discussion: https://postgr.es/m/CALdSSPjUprTj+vYp1tRKWkcLYzdy=N=O4Cn4y_HoxNSqQwBttg@mail.gmail.com	4 months ago
Tomas Vondra	db14dcdec6	Lookup the correct ordering for parallel GIN builds ... When building a tuplesort during parallel GIN builds, the function incorrectly looked up the default B-Tree operator, not the function associated with the GIN opclass (through GIN_COMPARE_PROC). Fixed by using the same logic as initGinState(), and the other place in parallel GIN builds. This could cause two types of issues. First, a data type might not have a B-Tree opclass, in which case the PrepareSortSupportFromOrderingOp() fails with an ERROR. Second, a data type might have both B-Tree and GIN opclasses, defining order/equality in different ways. This could lead to logical corruption in the index. Backpatch to 18, where parallel GIN builds were introduced. Discussion: https://postgr.es/m/73a28b94-43d5-4f77-b26e-0d642f6de777@iki.fi Reported-by: Heikki Linnakangas <hlinnaka@iki.fi> Backpatch-through: 18	4 months ago
Robert Haas	4cbaf4dcd2	Reduce length of TAP test file name. ... Buildfarm member fairywren hit the Windows limitation on the length of a file path. While there may be other things we should also do to prevent this from happening, it's certainly the case that the length of this test file name is much longer than others in the same directory, so make it shorter. Reported-by: Alexander Lakhin <exclusion@gmail.com> Discussion: http://postgr.es/m/274e0a1a-d7d2-4bc8-8b56-dd09f285715e@gmail.com Backpatch-through: 17	4 months ago
Peter Eisentraut	5ca5f12c2c	Fix accidentally cast away qualifiers ... This fixes cases where a qualifier (const, in all cases here) was dropped by a cast, but the cast was otherwise necessary or desirable, so the straightforward fix is to add the qualifier into the cast. Co-authored-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/b04f4d3a-5e70-4e73-9ef2-87f777ca4aac%40eisentraut.org	4 months ago
Fujii Masao	33a92632b7	doc: Clarify that \d and \d+ output lists are illustrative, not exhaustive. ... The psql documentation for the \d and \d+ meta-commands lists objects that may be shown, but previously the wording could be read as exhaustive even though additional objects can also appear in the output. This commit clarifies the description by adding phrasing such as "for example" or "such as", making it clear that the listed objects are illustrative rather than a complete list. While the change is small, it helps avoid potential user confusion. As this is a documentation clarification rather than a bug fix, it is not backpatched. Author: Peter Smith <smithpb2250@gmail.com> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Reviewed-by: Andreas Karlsson <andreas@proxel.se> Reviewed-by: Fujii Masao <masao.fujii@gmail.com> Discussion: https://postgr.es/m/CAHut+Pt1DBtaUqfJftkkaQLJJJenYJBtb6Ec6s6vu82KEMh46A@mail.gmail.com	4 months ago
David Rowley	7027dd499d	Remove deduplication logic from find_window_functions ... This code thought it was optimizing WindowAgg evaluation by getting rid of duplicate WindowFuncs, but it turns out all it does today is lead to cost-underestimations and makes it possible that optimize_window_clauses could miss some of the WindowFuncs that must receive an updated winref. The deduplication likely was useful when it was first added, but since the projection code was changed in b8d7f053c, the list of WindowFuncs gathered by find_window_functions isn't used during execution. Instead, the expression evaluation code will process the node's targetlist to find the WindowFuncs. The reason the deduplication could cause issues for optimize_window_clauses() is because if a WindowFunc is moved to another WindowClause, the winref is adjusted to reference the new WindowClause. If any duplicate WindowFuncs were discarded in find_window_functions() then the WindowFuncLists may not include all the WindowFuncs that need their winref adjusted. This could lead to an error message such as: ERROR: WindowFunc with winref 2 assigned to WindowAgg with winref 1 The back-branches will receive a different fix so that the WindowAgg costs are not affected. Author: Meng Zhang <mza117jc@gmail.com> Reviewed-by: David Rowley <dgrowleyml@gmail.com> Discussion: https://postgr.es/m/CAErYLFAuxmW0UVdgrz7iiuNrxGQnFK_OP9hBD5CUzRgjrVrz=Q@mail.gmail.com	4 months ago
Peter Eisentraut	6ceef9408c	Disable extended alignment uses on older g++ ... Fix for commit a9bdb63bba. The previous plan of redefining alignas didn't work, because it interfered with other C++ header files (e.g., LLVM). So now the new workaround is to just disable the affected typedefs under the affected compilers. These are not typically used in extensions anyway. Discussion: https://www.postgresql.org/message-id/3119480.1769189606%40sss.pgh.pa.us	4 months ago
Michael Paquier	d9abd9e105	Add test for MAINTAIN permission with pg_restore_extended_stats() ... Like its cousin functions for the restore of relation and attribute stats, pg_restore_extended_stats() needs to be run by a user that is the database owner or has MAINTAIN privileges on the table whose stats are restored. This commit adds a regression test ensuring that MAINTAIN is required when calling the function. This test also checks that a ShareUpdateExclusive lock is taken on the table whose stats are restored. This has been split from the commit that has introduced pg_restore_extended_stats(), for clarity. Author: Corey Huinker <corey.huinker@gmail.com> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Discussion: https://postgr.es/m/CADkLM=dpz3KFnqP-dgJ-zvRvtjsa8UZv8wDAQdqho=qN3kX0Zg@mail.gmail.com	4 months ago
Michael Paquier	114e84c532	Fix missing initialization in pg_restore_extended_stats() ... The tuple data upserted into pg_statistic_ext_data was missing an initialization for the nulls flag of stxoid and stxdinherit. This would cause an incorrect handling of the stats data restored. This issue has been spotted by CatalogTupleCheckConstraints(), translating to a NOT NULL constraint inconsistency, while playing more with the follow-up portions of the patch set. Oversight in 0e80f3f88d (mea culpa). Surprisingly, the buildfarm did not complain yet. Discussion: https://postgr.es/m/CADkLM=c7DY3Jv6ef0n_MGUJ1FyTMUoT697LbkST05nraVGNHYg@mail.gmail.com	4 months ago
Michael Paquier	0e80f3f88d	Add pg_restore_extended_stats() ... This function closely mirror its relation and attribute counterparts, but for extended statistics (i.e. CREATE STATISTICS) objects, being able to restore extended statistics for an extended stats object. Like the other functions, the goal of this feature is to ease the dump or upgrade of clusters so as ANALYZE would not be required anymore after these operations, stats being directly loaded into the target cluster without any post-dump/upgrade computation. The caller of this function needs the following arguments for the extended stats to restore: - The name of the relation. - The schema name of the relation. - The name of the extended stats object. - The schema name of the extended stats object. - If the stats are inherited or not. - One or more extended stats kind with its data. This commit adds only support for the restore of the extended statistics kind "n_distinct", building the basic infrastructure for the restore of more extended statistics kinds in follow-up commits, including MVC and dependencies. The support for "n_distinct" is eased in this commit thanks to the previous work done particularly in commits 1f927cce44 and 44eba8f06e, that have added the input function for the type pg_ndistinct, used as data type in input of this new restore function. Bump catalog version. Author: Corey Huinker <corey.huinker@gmail.com> Co-authored-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Discussion: https://postgr.es/m/CADkLM=dpz3KFnqP-dgJ-zvRvtjsa8UZv8wDAQdqho=qN3kX0Zg@mail.gmail.com	4 months ago