This is currently only added for meson, but could also be added for
make.
This has to be setup before the actual TAP tests are run as they are run
in parallel and as such would all try to setup the template at the same
time if we let them use the same folder for it without it being
pre-generated.
This seems to shorten the test suite run-time by ~25% on my laptop, so
it seems worth doing.
As of 15.4, macOS has strchrnul(), but access to it is blocked behind
a check for MACOSX_DEPLOYMENT_TARGET >= 15.4. But our does-it-link
configure check finds it, so we try to use it, and fail with the
present default deployment target (namely 15.0). This accounts for
today's buildfarm failures on indri and sifaka.
This is the identical problem that we faced some years ago when Apple
introduced preadv and pwritev in the same way. We solved that in
commit f014b1b9b by using AC_CHECK_DECLS instead of AC_CHECK_FUNCS
to check the functions' availability. So do the same now for
strchrnul(). Interestingly, we already had a workaround for
"the link check doesn't agree with <string.h>" cases with glibc,
which we no longer need since only the header declaration is being
checked.
Testing this revealed that the meson version of this check has never
worked, because it failed to use "-Werror=unguarded-availability-new".
(Apparently nobody's tried to build with meson on macOS versions that
lack preadv/pwritev as standard.) Adjust that while at it. Also,
we had never put support for "-Werror=unguarded-availability-new"
into v13, but we need that now.
Co-authored-by: Tom Lane <tgl@sss.pgh.pa.us>
Co-authored-by: Peter Eisentraut <peter@eisentraut.org>
Discussion: https://postgr.es/m/385134.1743523038@sss.pgh.pa.us
Backpatch-through: 13
Previously the portlock logic, added in 9b4eafcaf4, didn't actually work
properly when the tests were run via meson. 9b4eafcaf4 used the
MESON_BUILD_ROOT environment variable to determine the directory for the port
lock directory, but that's never set for running the tests. That meant that
each test used its own portlock dir, unless the PG_TEST_PORT_DIR environment
variable was set.
Fix the problem by setting top_builddir for the environment. That's also used
for the autoconf/make build.
Backpatch back to 16, where meson support was added.
Reported-by: Zharkov Roman <r.zharkov@postgrespro.ru>
Reviewed-by: Andrew Dunstan <andrew@dunslane.net>
Backpatch-through: 16
Before OpenSSL 1.1.0 the legacy names ssleay32 and libeay32 were
still used on Windows, and while we have support for this auto-
conf the meson buildsystem only used the new names on all plat-
forms. This adds support for the old name scheme when building
on Windows.
This patch only applies to 17 and 16 as master no longer support
OpenSSL 1.0.2.
Author: Darek Ślusarczyk <dslusarczyk@splunk.com>
Reviewed-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/CAN55FZ1Nk8wqY=mTrN78H026TuGV50h2H6uq1PwxhTauPYi3ug@mail.gmail.com
Backpatch-through: 16
The missing dependency was, e.g., visible when doing
ninja clean && ninja meson-test-prereq && meson test --no-rebuild --suite setup --suite libpq
This is a bit more complicated than other related fixes, because until now
libpq's tests depended on 'frontend_code', which includes a dependency on
fe_utils, which in turns on libpq. That in turn required
src/interfaces/libpq/test to be entered from the top-level, not from
libpq/meson.build. Because of that the test definitions in libpq/meson.build
could not declare a dependency on the binaries defined in
libpq/test/meson.build.
To fix this, this commit creates frontend_no_fe_utils_code, which allows us to
recurse into libpq/test from withing libpq/meson.build.
Apply this to all branches with meson support, as part of an effort to fix
incorrect test dependencies that can lead to test failures.
Discussion: https://postgr.es/m/CAGECzQSvM3iSDmjF+=Kof5an6jN8UbkP_4cKKT9w6GZavmb5yQ@mail.gmail.com
Discussion: https://postgr.es/m/bdba588f-69a9-4f3e-9b95-62d07210a32e@eisentraut.org
Backpatch: 16-, where meson support was added
The missing dependency was, e.g., visible when doing
ninja clean && ninja meson-test-prereq && meson test --no-rebuild --suite setup --suite cube
because meson (and thus its internal meson-test-prereq target) did not know
about a lot of the required targets.
Previously tmp_install did not actually depend on the relevant files being
built. That was mostly not visible, because "meson test" currently uses the
'default' targets as a test's dependency if no dependency is specified.
However, there are plans to narrow that on the meson side, to make it quicker
to run tests.
Apply this to all branches with meson support, as part of an effort to fix
incorrect test dependencies that can lead to test failures.
Discussion: https://postgr.es/m/CAGECzQSvM3iSDmjF+=Kof5an6jN8UbkP_4cKKT9w6GZavmb5yQ@mail.gmail.com
Discussion: https://postgr.es/m/bdba588f-69a9-4f3e-9b95-62d07210a32e@eisentraut.org
Backpatch: 16-, where meson support was added
Commit bc5a4dfc accidentally left a check for <stdbool.h> in
meson.build's header_checks. Synchronize with configure, which no
longer defines HAVE_STDBOOL_H.
There is still a reference to <stdbool.h> in an earlier test to see if
we need -std=c99 to get C99 features, like autoconf 2.69's
AC_PROG_CC_C99. (Therefore the test remove by this commit was
tautological since day one: you'd have copped "C compiler does not
support C99" before making it this far.)
Back-patch to 16, where meson begins.
On ARM platforms where the baseline CPU target lacks CRC instructions,
we need to supply a -march flag to persuade the compiler to compile
such instructions. It turns out that our existing choice of
"-march=armv8-a+crc" has not worked for some time, because recent gcc
will interpret that as selecting software floating point, and then
will spit up if the platform requires hard-float ABI, as most do
nowadays. The end result was to silently fall back to software CRC,
which isn't very desirable since in practice almost all currently
produced ARM chips do have hardware CRC.
We can fix this by using "-march=armv8-a+crc+simd" to enable the
correct ABI choice. (This has no impact on the code actually
generated, since neither of the files we compile with this flag
does any floating-point stuff, let alone SIMD.) Keep the test for
"-march=armv8-a+crc" since that's required for soft-float ABI,
but try that second since most platforms we're likely to build on
use hard-float.
Since this isn't working as-intended on the last several years'
worth of gcc releases, back-patch to all supported branches.
Discussion: https://postgr.es/m/4496616.iHFcN1HehY@portable-bastien
Previously we checked "for <stdbool.h> that conforms to C99" using
autoconf's AC_HEADER_STDBOOL macro. We've required C99 since PostgreSQL
12, so the test was redundant, and under C23 it was broken: autoconf
2.69's implementation doesn't understand C23's new empty header (the
macros it's looking for went away, replaced by language keywords).
Later autoconf versions fixed that, but let's just remove the
anachronistic test.
HAVE_STDBOOL_H and HAVE__BOOL will no longer be defined, but they
weren't directly tested in core or likely extensions (except in 11, see
below). PG_USE_STDBOOL (or USE_STDBOOL in 11 and 12) is still defined
when sizeof(bool) is 1, which should be true on all modern systems.
Otherwise we define our own bool type and values of size 1, which would
fail to compile under C23 as revealed by the broken test. (We'll
probably clean that dead code up in master, but here we want a minimal
back-patchable change.)
This came to our attention when GCC 15 recently started using using C23
by default and failed to compile the replacement code, as reported by
Sam James and build farm animal alligator.
Back-patch to all supported releases, and then two older versions that
also know about <stdbool.h>, per the recently-out-of-support policy[1].
12 requires C99 so it's much like the supported releases, but 11 only
assumes C89 so it now uses AC_CHECK_HEADERS instead of the overly picky
AC_HEADER_STDBOOL. (I could find no discussion of which historical
systems had <stdbool.h> but failed the conformance test; if they ever
existed, they surely aren't relevant to that policy's goals.)
[1] https://wiki.postgresql.org/wiki/Committing_checklist#Policies
Reported-by: Sam James <sam@gentoo.org>
Reviewed-by: Peter Eisentraut <peter@eisentraut.org> (master version)
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> (approach)
Discussion: https://www.postgresql.org/message-id/flat/87o72eo9iu.fsf%40gentoo.org
This commit modifies the build scripts so that `llvmjit.so` links
LLVM statically instead of dynamically. No other binary affected,
the change shouldn't be user visible at all.
The memory footprint of postgres is also unaffected: only the
`llvmjit.so` dynamic library is affected. If postgres doesn't use
JIT, the LLVM library isn't loaded, as before. If it uses JIT, it
is loaded as part of `llvmjit.so`, sa before.
The only change is the binary size of `llvmjit.so`.
There was a mistake in the new cap calculations during the cache extension. It popped up only when a new cache size was multiple of a cache record (every 256 records). Which lead to the usage of the memory beyond an allocated size. This commit fixes it along with mlock size for reallocated pages.
Also fixed a typo in a variable name.
Fixes PG-1248
Issue: pg_upgrade and some other tests are failing
This is caused by some mistake during the 17.2 rebase, where the commits
that fixed this issue previously, by changing the version and product name,
simply disappeared.
Fix: reintroduce the same changes, with this, the format of the name / version
is the same as for 17.0
This also modifies the pg_waldump build so it is now correctly uses
pg_tde as a static library, to handle warnings and custom flags for KMIP correctly.
This commit also enables generic -Werror for meson builds, so that we are forced
to fix warnings before merging in the future.
This commit:
- Adds build infrastructure for the frontend tools to compile with pg_tde. pg_tde dependencies will be built only if build flag `precone_ext` is on. It also makes `openssl` and `curl` required if `precone_ext` is on.
- Makes pg_waldump work with the TDE encrypted WAL. If user set `-k` flag it will try to init tde keys etc and decode encrypted pages. If no `-k` options set, it behaves like community version - won't try to ini tde and won't be able to decrypt WAL.
- Adds tap tests for encrypted WAL to pg_waldump.
- Fixes Percona versioning in ./configure
For PG-1003, PG-1005
Depends https://github.com/percona/pg_tde/pull/362
This commit:
1. Removes autoconf builds for pg_tde so it can be together built with Postgres (now, used would have to go to contrib/pg_tde and build it explicitly after building Postgres) with make. There are still left pg_tde builds in CI tests since this PR depends on https://github.com/percona/postgres/pull/20. So those leftover will be removed after merging PG PR
2. Adds necessary changes regarding new code (like kmip) so frontend tools (pg_waldump et al) can be compiled with pg_tde
2. Get rid of realpath as it has issues with optimised builds
For: PG-1003, PG-1005
Also includes some refactoring because libkmip and postgres headers
are not compatible. To avoid compilation errors, keyring_kmip.c does
not include postgres headers, and keyring_kmip_ereport.c does not
include libkmip headers.
We use tablespaceId as a part of IV for the internal keys encryption
which doesn't add any security because dbId (used as well) is unique
anyway.
But having tablespaceId really complicates things as a principal
created for the entire database but then different relations in this db
can be located in different tablespaces...
So it is better not to use tablespace with the principal key (database
level) as it belongs to the relation level.
When a relation moved to a new location it causes the change of relfilenode id
for it. Hence we must re-encrypt and store its internal key with the new id.
Also, we have to store the changed internal key in the new physical location,
and copy there principal key info and keyring data.
Fixes https://perconadev.atlassian.net/browse/PG-1038
* PG-1058 Fix MergeJoin issue
Resolved an issue in MergeJoin by ensuring the decrypted buffer contents are
also copied from the source to the destination tuple slot during
slot copy operations.
Co Authored by:
Andrew Pogrebnoy <absourd.noise@gmail.com>
Artem Gavrilov <artem.gavrilov@percona.com>
* Create table always checked the principal key and tried to create
it, event when we didn't try to create a tde_heap table
* Alter table wasn't handled, and because of this changing a table
to tde_heap access method didn't result in an encrypted table.
* defaut_table_access_method wasn't handled, and because of this,
creating a table using that also resulted in a non encrypted
table.
* PG-1056 Add failing test
* PG-1056 Use proper AM in test
* Fix UPDATE SET ... RETURNING processing for encrypted tuples
If `get_heap_tuple` is NULL, the core uses `copy_heap_tuple` instead. The former returns a pointer to a tuple in the slot and the latter makes a copy of such a tuple. For UPDATE SET, the core uses the slot for INSERT and later for RETURNING processing. If we copy the tuple the next happens:
1. The core creates a slot with the generic tuple.
2. It passed to `pg_tdeam_tuple_update()` and it gets a copy of the tuple here [6d4f7e5b7b/src17/access/pg_tdeam_handler.c (L336)].
3. This generic tuple is filled with the proper data and used for the update here [6d4f7e5b7b/src17/access/pg_tdeam_handler.c (L343)].
4. Later on, RETURNING processing uses the slot's tuple but is still a generic unmodified one because of the copy.
5. That results in wrong RETURNING data.
To avoid this, we should return a pointer to the slot's tuple instead of copying it.
Fixes PG-1056
* PG-1056 Split 'update' testcase for tde_heap and tde_heap_basic
---------
Co-authored-by: Andrew Pogrebnoy <absourd.noise@gmail.com>
The original part of the version is kept as is, and Percona specific
information is added after.
For example, psql displays the following after this commit:
SELECT version();
PostgreSQL 17.0.1 (Percona Server for PostgreSQL) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 6
4-bit
Where the addition is ".1 (Percona Server for PostgreSQL)", with the
".1" being the Percona version number.
Similarly, commands show the same information:
bin/pg_ctl --version
pg_ctl (PostgreSQL) 17.0.1 (Percona Server for PostgreSQL)
* Make related code compilable with frontend
This commit makes the code around keyring, principal keys and WAL
encryption compilable with frontend tools. Namely:
- Hide everything that isn't compatible and of no use behind
'#ifndef FRONTEND'
- Redefine code that is needed in both versions but should have
different code. E.g. error handling, file descriptors and locks
- Make use of frontend lists instead of backend ones where needed.
For https://perconadev.atlassian.net/browse/PG-857
The current method of detecting perl build flags breaks if the path to
perl contains a space. This change makes two improvements. First,
instead of getting a list of ldflags and ccdlflags and then trying to
filter those out of the reported ldopts, we tell perl to suppress
reporting those in the first instance. Second, it tells perl to parse
those and output them, one per line. Thus any space on the option in a
file name, for example, is preserved.
Issue reported off-list by Muralikrishna Bandaru
Discussion: https://postgr.es/01117f88-f465-bf6c-9362-083bd72ca305@dunslane.net
Backpatch to release 16.
Latest versions of Strawberry Perl define USE_THREAD_SAFE_LOCALE, and we
therefore get a handshake error when building against such instances.
The solution is to perform a test to see if USE_THREAD_SAFE_LOCALE is
defined and only define NO_THREAD_SAFE_LOCALE if it isn't.
Backpatch the meson.build fix back to release 16 and apply the same
logic to Mkvcbuild.pm in releases 12 through 16.
Original report of the issue from Muralikrishna Bandaru.
Anders Åstrand