After disabling the subscription, the failed test was changing the
two_phase option for the subscription. We can't change the two_phase
option for a subscription till the corresponding apply worker is active.
The check to ensure that the replication apply worker has exited was
incorrect.
Author: Vignesh C
Discussion: https://postgr.es/m/CALDaNm3YY+bzj+JWJbY+DsUgJ2mPk8OR1ttjVX2cywKr4BUgxw@mail.gmail.com
The buffer is used only locally within the function. Also, the
initialization to '0' characters was unnecessary, the initial content
were always overwritten with sprintf(). I don't understand why it was
done that way, but it's been like that since forever.
In the passing, change from sprintf() to snprintf(). The buffer was
long enough so sprintf() was fine, but this makes it more obvious that
there's no risk of a buffer overflow.
Reviewed-by: Robert Haas
Discussion: https://www.postgresql.org/message-id/7f86e06a-98c5-4ce3-8ec9-3885c8de0358@iki.fi
For clarity. The code was correct, and the buffer was large enough,
but string manipulation with no bounds checking is scary.
This incurs an extra palloc+pfree to every call, but in quick
performance testing, it doesn't seem to be significant.
Reviewed-by: Robert Haas
Discussion: https://www.postgresql.org/message-id/7f86e06a-98c5-4ce3-8ec9-3885c8de0358@iki.fi
Commit 9d9b9d46f3 removed the function (or rather, moved it to a
different source file and renamed it to SendCancelRequest), but forgot
the declaration in the header file.
This removes an inconsistency in the treatment of different datatypes by
the jsonpath timestamp_tz() function. Conversions from data types that
are not timestamp-aware, such as date and timestamp, are now treated
consistently with conversion from those that are such as timestamptz.
Author: David Wheeler
Reviewed-by: Junwang Zhao and Jeevan Chalke
Discussion: https://postgr.es/m/7DE080CE-6D8C-4794-9BD1-7D9699172FAB%40justatheory.com
Backpatch to release 17.
Commits e2562667 and 81385261 removed the configure equivalents, but
forgot to remove these options from meson_options.txt.
Revealed by the fact that build farm animals rorqual and francolin
didn't fail, despite being configured to set those options to off. They
should now fail with unknown option, until they are adjusted.
Previously we had a fallback implementation that made a harmless system
call, based on the assumption that system calls must contain a memory
barrier. That shouldn't be reached on any current system, and it seems
highly likely that we can easily find out how to request explicit memory
barriers, if we've already had to find out how to do atomics on a
hypothetical new system.
Removed comments and a function name referred to a spinlock used for
fallback memory barriers, but that changed in 1b468a13, which left some
misleading words behind in a few places.
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Suggested-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/721bf39a-ed8a-44b0-8b8e-be3bd81db748%40technowledgy.de
Discussion: https://postgr.es/m/3351991.1697728588%40sss.pgh.pa.us
Previously we had a fallback implementation of pg_compiler_barrier()
that called an empty function across a translation unit boundary so the
compiler couldn't see what it did. That shouldn't be needed on any
current systems, and might not even work with a link time optimizer.
Since we now require compiler-specific knowledge of how to implement
atomics, we should also know how to implement compiler barriers on a
hypothetical new system.
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Suggested-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/721bf39a-ed8a-44b0-8b8e-be3bd81db748%40technowledgy.de
Discussion: https://postgr.es/m/3351991.1697728588%40sss.pgh.pa.us
Modern versions of all relevant architectures and tool chains have
atomics support. Since edadeb07, there is no remaining reason to carry
code that simulates atomic flags and uint32 imperfectly with spinlocks.
64 bit atomics are still emulated with spinlocks, if needed, for now.
Any modern compiler capable of implementing C11 <stdatomic.h> must have
the underlying operations we need, though we don't require C11 yet. We
detect certain compilers and architectures, so hypothetical new systems
might need adjustments here.
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> (concept, not the patch)
Reviewed-by: Andres Freund <andres@anarazel.de> (concept, not the patch)
Discussion: https://postgr.es/m/3351991.1697728588%40sss.pgh.pa.us
A later change will require atomic support, so it wouldn't make sense
for a hypothetical new system not to be able to implement spinlocks.
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> (concept, not the patch)
Reviewed-by: Andres Freund <andres@anarazel.de> (concept, not the patch)
Discussion: https://postgr.es/m/3351991.1697728588%40sss.pgh.pa.us
The tests had a race condition if autovacuum was set to off. Instead we
create all the tables we are interested in with autovacuum disabled, so
they are only ever touched when in danger of wraparound.
Discussion: https://postgr.es/m/3e2cbd24-f45e-4b2b-ba83-8149214f0a4d@dunslane.net
Masahiko Sawada (slightly tweaked by me)
Backpatch to release 17 where these tests were introduced.
The problem is that the tool is using the LSN returned by
pg_create_logical_replication_slot() as recovery_target_lsn. This LSN is
ahead of the current WAL position and the recovery waits until the
publisher writes a WAL record to reach the target and ends the recovery.
On idle systems, this wait time is unpredictable and could lead to failure
in promoting the subscriber. To avoid that, insert a harmless WAL record.
Reported-by: Alexander Lakhin and Tom Lane
Diagnosed-by: Hayato Kuroda
Author: Euler Taveira
Reviewed-by: Hayato Kuroda, Amit Kapila
Backpatch-through: 17
Discussion: https://postgr.es/m/2377319.1719766794%40sss.pgh.pa.us
Discussion: https://postgr.es/m/CA+TgmoYcY+Wb67NAwaHT7MvxCSeV86oSc+va9hHKaasE42ukyw@mail.gmail.com
Add some comments to mention that this value must be at least 2 and also
add a StaticAssertDecl to cause compilation failure if anyone tries to
build with an invalid value.
The multiInsertBuffers list must have at least two elements due to how the
code in CopyMultiInsertInfoFlush() pushes the current ResultRelInfo's
CopyMultiInsertBuffer to the end of the list. If the first element is
also the last element, bad things will happen.
Author: Zhang Mingli <avamingli@gmail.com>
Discussion: https://postgr.es/m/CAApHDvpQ6t9ROcqbD-OgqR04Kfq4vQKw79Vo6r5j%2BciHwsSfkA%40mail.gmail.com
Now that the result of pg_newlocale_from_collation() is always
non-NULL, then we can move the collate_is_c and ctype_is_c flags into
pg_locale_t. That simplifies the logic in lc_collate_is_c() and
lc_ctype_is_c(), removing the dependence on setlocale().
This commit also eliminates the multi-stage initialization of the
collation cache.
As long as we have catalog access, then it's now safe to call
pg_newlocale_from_collation() without checking lc_collate_is_c()
first.
Discussion: https://postgr.es/m/cfd9eb85-c52a-4ec9-a90e-a5e4de56e57d@eisentraut.org
Reviewed-by: Peter Eisentraut, Andreas Karlsson
To determine if the two relations being joined can use partitionwise
join, we need to verify the existence of equi-join conditions
involving pairs of matching partition keys for all partition keys.
Currently we do that by looking through the join's restriction
clauses. However, it has been discovered that this approach is
insufficient, because there might be partition keys known equal by a
specific EC, but they do not form a join clause because it happens
that other members of the EC than the partition keys are constrained
to become a join clause.
To address this issue, in addition to examining the join's restriction
clauses, we also check if any partition keys are known equal by ECs,
by leveraging function exprs_known_equal(). To accomplish this, we
enhance exprs_known_equal() to check equality per the semantics of the
opfamily, if provided.
It could be argued that exprs_known_equal() could be called O(N^2)
times, where N is the number of partition key expressions, resulting
in noticeable performance costs if there are a lot of partition key
expressions. But I think this is not a problem. The number of a
joinrel's partition key expressions would only be equal to the join
degree, since each base relation within the join contributes only one
partition key expression. That is to say, it does not scale with the
number of partitions. A benchmark with a query involving 5-way joins
of partitioned tables, each with 3 partition keys and 1000 partitions,
shows that the planning time is not significantly affected by this
patch (within the margin of error), particularly when compared to the
impact caused by partitionwise join.
Thanks to Tom Lane for the idea of leveraging exprs_known_equal() to
check if partition keys are known equal by ECs.
Author: Richard Guo, Tom Lane
Reviewed-by: Tom Lane, Ashutosh Bapat, Robert Haas
Discussion: https://postgr.es/m/CAN_9JTzo_2F5dKLqXVtDX5V6dwqB0Xk+ihstpKEt3a1LT6X78A@mail.gmail.com
Parameterized partial paths are not supported, and we have several
checks in try_partial_xxx_path functions to enforce this. For a
partial nestloop join path, we need to ensure that if the inner path
is parameterized, the parameterization is fully satisfied by the
proposed outer path. For a partial merge/hashjoin join path, we need
to ensure that the inner path is not parameterized. In all cases, we
need to ensure that the outer path is not parameterized.
However, the comment in try_partial_hashjoin_path does not describe
this correctly. This patch fixes that.
In addtion, this patch simplifies the checks peformed in
try_partial_hashjoin_path and try_partial_mergejoin_path with the help
of macro PATH_REQ_OUTER, and also adds asserts that the outer path is
not parameterized in try_partial_xxx_path functions.
Author: Richard Guo
Discussion: https://postgr.es/m/CAMbWs48mKJ6g_GnYNa7dnw04MHaMK-jnAEBrMVhTp2uUg3Ut4A@mail.gmail.com
In sort_inner_and_outer, we create mergejoin join paths by explicitly
sorting both relations on each possible ordering of the available
mergejoin clauses. However, if there are no available mergejoin
clauses, we can skip this process entirely.
This patch introduces a check for mergeclause_list at the beginning of
sort_inner_and_outer and exits the function if it is found to be
empty. This might help skip all the statements that come before the
call to select_outer_pathkeys_for_merge, including the build of
UniquePaths in the case of JOIN_UNIQUE_OUTER or JOIN_UNIQUE_INNER.
I doubt there's any measurable performance improvement, but throughout
the run of the regression tests, sort_inner_and_outer is called a
total of 44,424 times. Among these calls, there are 11,064 instances
where mergeclause_list is found to be empty, which accounts for
approximately one-fourth. I think this suggests that implementing
this shortcut is worthwhile.
Author: Richard Guo
Reviewed-by: Ashutosh Bapat
Discussion: https://postgr.es/m/CAMbWs48RKiZGFEd5A0JtztRY5ZdvVvNiHh0AKeuoz21F+0dVjQ@mail.gmail.com
This is useful to know which part of a stats file is corrupted when
reading it, adding to the server logs a WARNING with details about what
could not be read before giving up with the remaining data in the file.
Author: Michael Paquier
Reviewed-by: Bertrand Drouvot
Discussion: https://postgr.es/m/Zp8o6_cl0KSgsnvS@paquier.xyz
The current method of coercing the boolean result value of
JsonPathExists() to the target type specified for an EXISTS column,
which is to call the type's input function via json_populate_type(),
leads to an error when the target type is integer, because the
integer input function doesn't recognize boolean literal values as
valid.
Instead use the boolean-to-integer cast function for coercion in that
case so that using integer or domains thereof as type for EXISTS
columns works. Note that coercion for ON ERROR values TRUE and FALSE
already works like that because the parser creates a cast expression
including the cast function, but the coercion of the actual result
value is not handled by the parser.
Tests by Jian He.
Reported-by: Jian He <jian.universality@gmail.com>
Author: Jian He <jian.universality@gmail.com>
Author: Amit Langote <amitlangote09@gmail.com>
Discussion: https://postgr.es/m/CACJufxEo4sUjKCYtda0_qt9tazqqKPmF1cqhW9KBOUeJFqQd2g@mail.gmail.com
Backpatch-through: 17
1. Remove the special case handling when casting the JsonBehavior
expressions to types with typmod, like 86d33987 did for the casting
of SQL/JSON constructor functions.
2. Fix casting for fixed-length character and bit string types by
using assignment-level casts. This is again similar to what
86d33987 did, but for ON ERROR / EMPTY expressions.
3. Use runtime coercion for the boolean ON ERROR constants so that
using fixed-length character string types, for example, for an
EXISTS column doesn't cause a "value too long for type
character(n)" when the parser tries to coerce the default ON ERROR
value "false" to that type, that is, even when clause is not
specified.
4. Simplify the conditions of when to use runtime coercion vs
creating the cast expression in the parser itself. jsonb-valued
expressions are now always coerced at runtime and boolean
expressions too if the target type is a string type for the
reasons mentioned above.
Tests are taken from a patch that Jian He posted.
Reported-by: Jian He <jian.universality@gmail.com>
Author: Jian He <jian.universality@gmail.com>
Author: Amit Langote <amitlangote09@gmail.com>
Discussion: https://postgr.es/m/CACJufxEo4sUjKCYtda0_qt9tazqqKPmF1cqhW9KBOUeJFqQd2g@mail.gmail.com
Backpatch-through: 17
Previously, pg_newlocale_from_collation() returned NULL as a special
case for the DEFAULT_COLLATION_OID if the provider was libc. In that
case the behavior would depend on the last call to setlocale().
Now, consistent with the other providers, it will return a pointer to
default_locale, which is not dependent on setlocale().
Note: for the C and POSIX locales, the locale_t structure within the
pg_locale_t will still be zero, because those locales are implemented
with internal logic and do not use libc at all.
lc_collate_is_c() and lc_ctype_is_c() still depend on setlocale() to
determine the current locale, which will be removed in a subsequent
commit.
Discussion: https://postgr.es/m/cfd9eb85-c52a-4ec9-a90e-a5e4de56e57d@eisentraut.org
Reviewed-by: Peter Eisentraut, Andreas Karlsson
These tools were used to read the koi-iso.tab, koi-win.tab, and
koi-alt.tab files, which contained the mappings between the
single-byte cyrillic encodings. However, those data files were removed
in commit 4c3c8c048d, back in 2003. These code generators have been
unused and unusable ever since.
The generated tables live in cyrillic_and_mic.c. There has been one
change to the tables since they were generated in 1999, in commit
f4b7624eb0. So if we resurrected the original data tables, that
change would need to be taken into account.
So this code is very dead. The tables in cyrillic_and_mic.c, which
were originally generated by these tools, are now the authoritative
source for these mappings.
Reviewed-by: Tom Lane, Aleksander Alekseev
Discussion: https://www.postgresql.org/message-id/flat/a821c3dc-36ec-4cee-8b41-7ccaa17adb18@iki.fi
The initial implementation in commit 959b38d77 counted one action
per TOC entry (except for some special cases for multi-blob BLOBS
entries). This assumes that TOC entries are all about equally
complex, but it turns out that that assumption doesn't hold up very
well in binary-upgrade mode. For example, even after the previous
commit I was able to cause backend bloat with tables having many
inherited constraints. There may be other cases too. (Since no
serious problems have been reported with --single-transaction mode,
we can conclude that the backend copes well with psql's regular
restore scripts; but before 959b38d77 we never ran binary-upgrade
restores with multi-command transactions.)
To fix, count multi-command TOC entries as N actions, allowing the
transaction size to be scaled down when we hit a complex TOC entry.
Rather than add a SQL parser to pg_restore, approximate "multi
command" by counting semicolons in the TOC entry's defn string.
This will be fooled by semicolons appearing in string literals ---
but the error is in the conservative direction, so it doesn't seem
worth working harder. The biggest risk is with function/procedure
TOC entries, but we can just explicitly skip those.
(This is undoubtedly a hack, and maybe someday we'll be able to
revert it after fixing the backend's bloat issues or rethinking
what pg_dump emits in binary upgrade mode. But that surely isn't
a project for v17.)
Thanks to Alexander Korotkov for the let's-count-semicolons idea.
Per report from Justin Pryzby. Back-patch to v17 where txn_size mode
was introduced.
Discussion: https://postgr.es/m/ZqEND4ZcTDBmcv31@pryzbyj2023
Avoid issuing a separate SQL UPDATE command for each column when
directly manipulating pg_attribute contents in binary upgrade mode.
With the separate updates, we triggered a relcache invalidation with
each update. For a table with N columns, that causes O(N^2) relcache
bloat in txn_size mode because the table's newly-created relcache
entry can't be flushed till end of transaction. Reducing the number
of commands should make it marginally faster as well as avoiding that
problem.
While at it, likewise avoid issuing a separate UPDATE on pg_constraint
for each inherited constraint. This is less exciting, first because
inherited (non-partitioned) constraints are relatively rare, and
second because the backend has a good deal of trouble anyway with
restoring tables containing many such constraints, due to
MergeConstraintsIntoExisting being horribly inefficient. But it seems
more consistent to do it this way here too, and it surely can't hurt.
In passing, fix one place in dumpTableSchema that failed to use ONLY
in ALTER TABLE. That's not a live bug, but it's inconsistent.
Also avoid silently casting away const from string literals.
Per report from Justin Pryzby. Back-patch to v17 where txn_size mode
was introduced.
Discussion: https://postgr.es/m/ZqEND4ZcTDBmcv31@pryzbyj2023
Commit 9d9b9d46f3 added spinlocks to protect the fields in ProcSignal
flags, but in EmitProcSignalBarrier(), the spinlock was released
twice. With most spinlock implementations, releasing a lock that's not
held is not easy to notice, because most of the time it does nothing,
but if the spinlock was concurrently acquired by another process, it
could lead to more serious issues. Fortunately, with the
--disable-spinlocks emulation implementation, it caused more visible
failures.
In the passing, fix a type in comment and add an assertion that the
procNumber passed to SendProcSignal looks valid.
Discussion: https://www.postgresql.org/message-id/b8ce284c-18a2-4a79-afd3-1991a2e7d246@iki.fi
Move responsibility of generating the cancel key to the backend
process. The cancel key is now generated after forking, and the
backend advertises it in the ProcSignal array. When a cancel request
arrives, the backend handling it scans the ProcSignal array to find
the target pid and cancel key. This is similar to how this previously
worked in the EXEC_BACKEND case with the ShmemBackendArray, just
reusing the ProcSignal array.
One notable change is that we no longer generate cancellation keys for
non-backend processes. We generated them before just to prevent a
malicious user from canceling them; the keys for non-backend processes
were never actually given to anyone. There is now an explicit flag
indicating whether a process has a valid key or not.
I wrote this originally in preparation for supporting longer cancel
keys, but it's a nice cleanup on its own.
Reviewed-by: Jelte Fennema-Nio
Discussion: https://www.postgresql.org/message-id/508d0505-8b7a-4864-a681-e7e5edfe32aa@iki.fi
In try_partitionwise_join, we aim to break down the join between two
partitioned relations into joins between matching partitions. To
achieve this, we iterate through each pair of partitions from the two
joining relations and create child-join relations for them. With
potentially thousands of partitions, the local objects allocated in
each iteration can accumulate significant memory usage. Therefore, we
opt to eagerly free these local objects at the end of each iteration.
In line with this approach, this patch frees the bitmap set that
represents the relids of child-join relations at the end of each
iteration. Additionally, it modifies build_child_join_rel() to reuse
the AppendRelInfo structures generated within each iteration.
Author: Ashutosh Bapat
Reviewed-by: David Christensen, Richard Guo
Discussion: https://postgr.es/m/CAExHW5s4EqY43oB=ne6B2=-xLgrs9ZGeTr1NXwkGFt2j-OmaQQ@mail.gmail.com
We don't currently support parameterized MergeAppend paths: there's
little use for an ordered path on the inside of a nestloop. Given
this, we can simplify create_merge_append_path by directly setting
param_info to NULL instead of calling get_appendrel_parampathinfo. We
can also simplify the Assert for child paths a little bit.
This change won't make any measurable difference in performance; it's
just for clarity's sake.
Author: Richard Guo
Reviewed-by: Alena Rybakina, Paul A Jungwirth
Discussion: https://postgr.es/m/CAMbWs4_n1bgH2nACMuGsXZct3KH6PBFS0tPdQsXdstRfyxTunQ@mail.gmail.com
32d3ed816 moved the logic for setting the context's name and ident into
a reusable function. I missed adding a pointer dereference after
copying and pasting the code into that function. The ident parameter is
a pointer to the ident variable in the calling function, so the
dereference is required to correctly determine if the contents of that
variable is NULL or not.
In passing, adjust the if condition to include an == NULL to make it
more clear that it's not checking for == '\0'.
Reported-by: Tom Lane, Coverity
Discussion: https://postgr.es/m/2256588.1722184287@sss.pgh.pa.us
Speeds up text comparison expressions when using a collation other
than the database default collation. Does not affect larger operations
such as ORDER BY, because the lookup is only done once.
Discussion: https://postgr.es/m/7bb9f018d20a7b30b9a7f6231efab1b5e50c7720.camel@j-davis.com
Reviewed-by: John Naylor, Andreas Karlsson
Currently, when amcheck validates a unique constraint, it visits the heap for
each index tuple. This commit implements skipping keys, which have only one
non-dedeuplicated index tuple (quite common case for unique indexes). That
gives substantial economy on index checking time.
Reported-by: Noah Misch
Discussion: https://postgr.es/m/20240325020323.fd.nmisch%40google.com
Author: Alexander Korotkov, Pavel Borisov
pg_size_pretty(bigint) would return the value in bytes rather than PB
for the smallest-most bigint value. This happened due to an incorrect
assumption that the absolute value of -9223372036854775808 could be
stored inside a signed 64-bit type.
Here we fix that by instead storing that value in an unsigned 64-bit type.
This bug does exist in versions prior to 15 but the code there is
sufficiently different and the bug seems sufficiently non-critical that
it does not seem worth risking backpatching further.
Author: Joseph Koshakow <koshy44@gmail.com>
Discussion: https://postgr.es/m/CAAvxfHdTsMZPWEHUrZ=h3cky9Ccc3Mtx2whUHygY+ABP-mCmUw@mail.gmail.com
Backpatch-through: 15
Commit 453c468737 introduced a use of strerror() into libpq, but that
is not thread-safe. Fix by using strerror_r() instead.
In passing, update some of the code comments added by 453c468737, as
we have learned more about the reason for the change in OpenSSL that
started this.
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: Discussion: https://postgr.es/m/b6fb018b-f05c-4afd-abd3-318c649faf18@highgo.ca
There were quite a few places where we either had a non-NUL-terminated
string or a text Datum which we needed to call escape_json() on. Many of
these places required that a temporary string was created due to the fact
that escape_json() needs a NUL-terminated cstring. For text types, those
first had to be converted to cstring before calling escape_json() on them.
Here we introduce two new functions to make escaping JSON more optimal:
escape_json_text() can be given a text Datum to append onto the given
buffer. This is more optimal as it foregoes the need to convert the text
Datum into a cstring. A temporary allocation is only required if the text
Datum needs to be detoasted.
escape_json_with_len() can be used when the length of the cstring is
already known or the given string isn't NUL-terminated. Having this
allows various places which were creating a temporary NUL-terminated
string to just call escape_json_with_len() without any temporary memory
allocations.
Discussion: https://postgr.es/m/CAApHDvpLXwMZvbCKcdGfU9XQjGCDm7tFpRdTXuB9PVgpNUYfEQ@mail.gmail.com
Reviewed-by: Melih Mutlu, Heikki Linnakangas
To build with -Dreadline=enabled one can use either readline or
libedit. The -Dlibedit_preferred flag is supposed to control the order
of names to lookup. This works fine when either both libraries are
present or -Dreadline is set to auto. However, explicitly enabling
readline with only libedit present, but not setting libedit_preferred,
or alternatively enabling readline with only readline present, but
setting libedit_preferred, too, are both broken. This is because
cc.find_library will throw an error for a not found dependency as soon
as the first required dependency is checked, thus it's impossible to
fallback to the alternative.
Here we only check the second of the two dependencies for
requiredness, thus we only fail when none of the two can be found.
Author: Wolfgang Walther
Reviewed-by: Nazir Bilal Yavuz, Alvaro Herrera, Peter Eisentraut
Reviewed-by: Tristan Partin
Discussion: https://www.postgresql.org/message-id/ca8f37e1-a2c3-40e2-91f6-59c3d3652ad4@technowledgy.de
Backpatch: 16-, where meson support was added
Peter Eisentraut