Commit 619bc23a1 changed "make dist" to invoke "git archive",
but hard-wired the call to specify that the HEAD revision should
be packaged. Our tarball building process needs to be able to
specify which git commit to package (notably, for packaging
back branches). While we could make that work with some hackery
to operate in detached-HEAD state, it's a lot nicer just to expose
git archive's ability to specify what to package. Hence, invent
a new make variable PG_GIT_REVISION. This is undocumented, but
so is "make dist".
Also make corresponding changes in the meson scripts. We have no
near-term intention of using that for package building, but it
will likely happen eventually, so stay prepared.
Discussion: https://postgr.es/m/3552543.1713909947@sss.pgh.pa.us
Presently, pg_popcount() processes data in 32-bit or 64-bit chunks
when possible. Newer hardware that supports AVX-512 instructions
can use 512-bit chunks, which provides a nice speedup, especially
for larger buffers. This commit introduces the infrastructure
required to detect compiler and CPU support for the required
AVX-512 intrinsic functions, and it adds a new pg_popcount()
implementation that uses these functions. If CPU support for this
optimized implementation is detected at runtime, a function pointer
is updated so that it is used by subsequent calls to pg_popcount().
Most of the existing in-tree calls to pg_popcount() should benefit
from these instructions, and calls with smaller buffers should at
least not regress compared to v16. The new infrastructure
introduced by this commit can also be used to optimize
visibilitymap_count(), but that is left for a follow-up commit.
Co-authored-by: Paul Amonson, Ants Aasma
Reviewed-by: Matthias van de Meent, Tom Lane, Noah Misch, Akash Shankaran, Alvaro Herrera, Andres Freund, David Rowley
Discussion: https://postgr.es/m/BL1PR11MB5304097DF7EA81D04C33F3D1DCA6A%40BL1PR11MB5304.namprd11.prod.outlook.com
Commit 619bc23a1a introduced
WARNING: Project targets '>=0.54' but uses feature introduced in '0.55.0': Passing executable/found program object to script parameter of add_dist_script
Work around that by wrapping the offending line in a meson version check.
Author: Tristan Partin <tristan@neon.tech>
Discussion: https://www.postgresql.org/message-id/flat/D096Q3NFFVH1.1T5RE4MOO9ZFH%40neon.tech
This changes "make dist" to directly use "git archive", rather than
the custom shell script it currently runs.
This is to make the creation of the distribution tarball more directly
traceable to the git repository. That is why we removed the "make
distprep" step.
"make dist" continues to produce a .gz and a .bz2 tarball as before.
The archives produced this way are deterministic and reproducible,
meaning for a given commit the result file should always be
bit-for-bit identical. The exception is that if you use a git version
older than 2.38.0, gzip records the platform in the archive, so you'd
get a different output on Windows vs. macOS vs. "UNIX" (everything
else). In git 2.38.0, this was changed so that everything is recorded
as "UNIX" now. This is just something to keep in mind. This issue is
specific to the gzip format, it does not affect other compression
formats.
Meson has its own distribution building command (meson dist), but we
are not using that at this point. The main problem is that, the way
they have implemented it, it is not deterministic in the above sense.
Also, we want a "make" version for the time being. But the target
name "dist" in meson is reserved for that reason, so we call the
custom target "pgdist" (so call something like "meson compile -C build
pgdist").
Reviewed-by: Tristan Partin <tristan@neon.tech>
Discussion: https://www.postgresql.org/message-id/flat/40e80f77-a294-4f29-a16f-e21bc7bc75fc%40eisentraut.org
Starting with the Sonoma toolchain macos' linker emits warnings when the same
library is linked to twice. That's ill considered, as the same library can be
used by multiple subsidiary libraries. Luckily there's a flag to suppress that
warning.
On Ventura meson's default of -Wl,-undefined,dynamic_lookup caused warnings,
which we suppressed with -Wl,-undefined,error. Unfortunately that causes a
warning on Sonoma, which is absurd, as it's documented linker default. To
avoid that warning, only add -Wl,-undefined,error if it does not trigger
warnings. Luckily dynamic_lookup doesn't trigger a warning on Sonoma anymore.
Discussion: https://postgr.es/m/20231201040515.p5bshhhtfru7d3da@awork3.anarazel.de
Backpatch: 16-, where the meson build was added
The copy_file_range() system call is available on at least Linux and
FreeBSD, and asks the kernel to use efficient ways to copy ranges of a
file. Options available to the kernel include sharing block ranges
(similar to --clone mode), and pushing down block copies to the storage
layer.
For automated testing, see PG_TEST_PG_UPGRADE_MODE. (Perhaps in a later
commit we could consider setting this mode for one of the CI targets.)
Reviewed-by: Peter Eisentraut <peter@eisentraut.org>
Discussion: https://postgr.es/m/CA%2BhUKGKe7Hb0-UNih8VD5UNZy5-ojxFb3Pr3xSBBL8qj2M2%3DdQ%40mail.gmail.com
There isn't a lot of user demand for AIX support, we have a bunch of
hacks to work around AIX-specific compiler bugs and idiosyncrasies,
and no one has stepped up to the plate to properly maintain it.
Remove support for AIX to get rid of that maintenance overhead. It's
still supported for stable versions.
The acute issue that triggered this decision was that after commit
8af2565248, the AIX buildfarm members have been hitting this
assertion:
TRAP: failed Assert("(uintptr_t) buffer == TYPEALIGN(PG_IO_ALIGN_SIZE, buffer)"), File: "md.c", Line: 472, PID: 2949728
Apperently the "pg_attribute_aligned(a)" attribute doesn't work on AIX
for values larger than PG_IO_ALIGN_SIZE, for a static const variable.
That could be worked around, but we decided to just drop the AIX support
instead.
Discussion: https://www.postgresql.org/message-id/20240224172345.32@rfd.leadboat.com
Reviewed-by: Andres Freund, Noah Misch, Thomas Munro
portname is set to host_system, but host_system might still be changed
later in the file. As a consequence, platforms where host_system is
changed later wouldn't find some of their platform-specific files
(driven by portname), and the build would fail. Move the assignment
of portname further down after the last change of host_system (but
before further overriding assignments to portname).
Discussion: https://www.postgresql.org/message-id/flat/CAC7zN94TdsHhY88XkroJzSMx7E%3DBQpV9LKKjNSEnTM04ihoWCA%40mail.gmail.com
Injection points are a new facility that makes possible for developers
to run custom code in pre-defined code paths. Its goal is to provide
ways to design and run advanced tests, for cases like:
- Race conditions, where processes need to do actions in a controlled
ordered manner.
- Forcing a state, like an ERROR, FATAL or even PANIC for OOM, to force
recovery, etc.
- Arbitrary sleeps.
This implements some basics, and there are plans to extend it more in
the future depending on what's required. Hence, this commit adds a set
of routines in the backend that allows developers to attach, detach and
run injection points:
- A code path calling an injection point can be declared with the macro
INJECTION_POINT(name).
- InjectionPointAttach() and InjectionPointDetach() to respectively
attach and detach a callback to/from an injection point. An injection
point name is registered in a shmem hash table with a library name and a
function name, which will be used to load the callback attached to an
injection point when its code path is run.
Injection point names are just strings, so as an injection point can be
declared and run by out-of-core extensions and modules, with callbacks
defined in external libraries.
This facility is hidden behind a dedicated switch for ./configure and
meson, disabled by default.
Note that backends use a local cache to store callbacks already loaded,
cleaning up their cache if a callback has found to be removed on a
best-effort basis. This could be refined further but any tests but what
we have here was fine with the tests I've written while implementing
these backend APIs.
Author: Michael Paquier, with doc suggestions from Ashutosh Bapat.
Reviewed-by: Ashutosh Bapat, Nathan Bossart, Álvaro Herrera, Dilip
Kumar, Amul Sul, Nazir Bilal Yavuz
Discussion: https://postgr.es/m/ZTiV8tn_MIb_H2rE@paquier.xyz
sed is used only if dtrace or selinux are enabled. Those options are
only used on Unix platforms, which should have sed. But we don't want
to make sed a hard requirement on Windows, which was the case in meson
until now.
This just changes sed to be not-required by meson. If you happen to
use a system with, say, dtrace but without sed, you might get a
slightly complicated error from meson during the build, but that seems
better than making the requiredness a complicated conditional that
will need to be maintained.
Discussion: https://www.postgresql.org/message-id/flat/ZQzp_VMJcerM1Cs_%40paquier.xyz
This commit removes all the scripts located in src/tools/msvc/ to build
PostgreSQL with Visual Studio on Windows, meson becoming the recommended
way to achieve that. The scripts held some information that is still
relevant with meson, information kept and moved to better locations.
Comments that referred directly to the scripts are removed.
All the documentation still relevant that was in install-windows.sgml
has been moved to installation.sgml under a new subsection for Visual.
All the content specific to the scripts is removed. Some adjustments
for the documentation are planned in a follow-up set of changes.
Author: Michael Paquier
Reviewed-by: Peter Eisentraut, Andres Freund
Discussion: https://postgr.es/m/ZQzp_VMJcerM1Cs_@paquier.xyz
Commit 252dcb3239 introduced initdb template caching to speed up
tests by re-using initdb output. The initdb command didn't however
use the --no-clean option to preserve generated data in case initdb
crashes unlike pg_regress which does do this. This adds the option
to initdb to aid debugging.
While changing the commandline, switch to using long options for
initdb to make the code more self-documenting.
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/CAEze2WhSTjfK_M+Ea4GSQp8odrEOaQS8HyORd1TJUEiyXaB+rw@mail.gmail.com
The just released meson 1.3 strongly deprecated a hack we were using, emitting
a noisy warning (the hack basically depended on an implementation detail to
work). Turns out there has been a better way available for a while, I just
hadn't found it. 1.4 added a more convenient approach, but we can't rely on
that.
Reviewed-by: Tristan Partin <tristan@neon.tech>
Discussion: https://postgr.es/m/20231129185053.s6c7f73eg7b4ztfi@awork3.anarazel.de
Backpatch: 16-, where the meson build was added.
When we look for the Python installation using the meson python
module, we should make it use the python program previously determined
by the 'PYTHON' option. Otherwise, it will just use its own search
and the 'PYTHON' option won't affect it. We need this to be able to
select the Python installation to build PL/Python against.
Reviewed-by: Andres Freund <andres@anarazel.de>
Discussion: https://www.postgresql.org/message-id/flat/3f5427e8-cef6-474d-ae88-13db1bfdc1a7%40eisentraut.org
We should have done it this way all along, but we accidentally got
away with using the wrong BIO field up until OpenSSL 3.2. There,
the library's BIO routines that we rely on use the "data" field
for their own purposes, and our conflicting use causes assorted
weird behaviors up to and including core dumps when SSL connections
are attempted. Switch to using the approved field for the purpose,
i.e. app_data.
While at it, remove our configure probes for BIO_get_data as well
as the fallback implementation. BIO_{get,set}_app_data have been
there since long before any OpenSSL version that we still support,
even in the back branches.
Also, update src/test/ssl/t/001_ssltests.pl to allow for a minor
change in an error message spelling that evidently came in with 3.2.
Tristan Partin and Bo Andreson. Back-patch to all supported branches.
Discussion: https://postgr.es/m/CAN55FZ1eDDYsYaL7mv+oSLUij2h_u6hvD4Qmv-7PK7jkji0uyQ@mail.gmail.com
Currently important build targets are somewhat hard to discover. This commit
documents important meson build targets in the sgml documentation. But it's
awkward to have to lookup build targets in the docs when hacking, so this also
adds a 'help' target, printing out the same information. To avoid having to
duplicate information in two places, generate both docbook and interactive
docs from a single source.
Reviewed-by: Peter Eisentraut <peter@eisentraut.org>
Discussion: https://postgr.es/m/20231108232121.ww542mt6lfo6f26f@awork3.anarazel.de
If the tap_tests option is disabled under Meson, the TAP tests are
currently not registered at all. But this makes it harder to see what
is going on, why suddently there are fewer tests than before.
Instead, run testwrap with an option that marks the test as skipped.
That way, the total list and count of tests is constant whether the
option is enabled or not.
Reviewed-by: Andres Freund <andres@anarazel.de>
Discussion: https://www.postgresql.org/message-id/ad5ec96d-69ec-317b-a137-367ea5019b61@eisentraut.org
A PostgreSQL release tarball contains a number of prebuilt files, in
particular files produced by bison, flex, perl, and well as html and
man documentation. We have done this consistent with established
practice at the time to not require these tools for building from a
tarball. Some of these tools were hard to get, or get the right
version of, from time to time, and shipping the prebuilt output was a
convenience to users.
Now this has at least two problems:
One, we have to make the build system(s) work in two modes: Building
from a git checkout and building from a tarball. This is pretty
complicated, but it works so far for autoconf/make. It does not
currently work for meson; you can currently only build with meson from
a git checkout. Making meson builds work from a tarball seems very
difficult or impossible. One particular problem is that since meson
requires a separate build directory, we cannot make the build update
files like gram.h in the source tree. So if you were to build from a
tarball and update gram.y, you will have a gram.h in the source tree
and one in the build tree, but the way things work is that the
compiler will always use the one in the source tree. So you cannot,
for example, make any gram.y changes when building from a tarball.
This seems impossible to fix in a non-horrible way.
Second, there is increased interest nowadays in precisely tracking the
origin of software. We can reasonably track contributions into the
git tree, and users can reasonably track the path from a tarball to
packages and downloads and installs. But what happens between the git
tree and the tarball is obscure and in some cases non-reproducible.
The solution for both of these issues is to get rid of the step that
adds prebuilt files to the tarball. The tarball now only contains
what is in the git tree (*). Getting the additional build
dependencies is no longer a problem nowadays, and the complications to
keep these dual build modes working are significant. And of course we
want to get the meson build system working universally.
This commit removes the make distprep target altogether. The make
dist target continues to do its job, it just doesn't call distprep
anymore.
(*) - The tarball also contains the INSTALL file that is built at make
dist time, but not by distprep. This is unchanged for now.
The make maintainer-clean target, whose job it is to remove the
prebuilt files in addition to what make distclean does, is now just an
alias to make distprep. (In practice, it is probably obsolete given
that git clean is available.)
The following programs are now hard build requirements in configure
(they were already required by meson.build):
- bison
- flex
- perl
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Andres Freund <andres@anarazel.de>
Discussion: https://www.postgresql.org/message-id/flat/e07408d9-e5f2-d9fd-5672-f53354e9305e@eisentraut.org
Previously we errored out if no python installation could be found (but we did
handle not having enough of python installed to build plpython
against). Presumably nobody hit this so far, as python is likely installed due
to meson requiring python.
Author: Tristan Partin <tristan@neon.tech>
Discussion: https://postgr.es/m/CSPIJVUDZFKX.3KHMOAVGF94RV@c3po
Backpatch: 16-, where meson support was added
-exported_symbols_list=... works on Ventura and earlier, but not on
Sonoma. The easiest way to fix it is to -Wl,-exported_symbols_list,@0@ which
actually seems more appropriate anyway, it's obviously a linker argument. It
is easier to use the -Wl,, syntax than passing multiple arguments, due to the
way the export_fmt is used (a single string that's formatted), but if it turns
out to be necessary, we can go for multiple arguments as well.
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/20230928222248.jw6s7yktpfsfczha@alap3.anarazel.de
Backpatch: 16-, where the meson based buildsystem was added
When running all (or just many) of our tests, a significant portion of both
CPU time and IO is spent running initdb. Most of those initdb runs don't
specify any options influencing properties of the created data directory.
Avoid most of that overhead by creating a "template" data directory, alongside
the temporary installation. Instead of running initdb, pg_regress and tap
tests can copy that data directory. When a tap test specifies options to
initdb, the template data directory is not used. That could be relaxed for
some options, but it's not clear it's worth the effort.
There unfortunately is some duplication between pg_regress.c and Cluster.pm,
but there are no easy ways of sharing that code without introducing additional
complexity.
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/20220120021859.3zpsfqn4z7ob7afz@alap3.anarazel.de
Commit d2e15083 got rid of the main configure probe and HAVE_FDATASYNC
macro, but we still searched -lrt and -lposix4 for old Solaris systems.
It's in the C library on modern Solaris, as on other supported systems.
Reviewed-by: Peter Eisentraut <peter@eisentraut.org>
Reviewed-by: Tristan Partin <tristan@neon.tech>
Discussion: https://postgr.es/m/CA%2BhUKGLLVx4drdvXats9PxH3eeB%2BE2NkJReJ%2BRGmaOpU%3D9rgEA%40mail.gmail.com
As with the Intel and Arm CRC instructions, compiler intrinsics for
them must be supported by the compiler. In contrast, no runtime check
is needed. Aligned memory access is faster, so use the Arm coding as
a model.
YANG Xudong
Discussion: https://postgr.es/m/b522a0c5-e3b2-99cc-6387-58134fb88cbe%40ymatrix.cn
In the event the llvm option is defined to be 'auto', it is possible
that the host machine might not have a C++ compiler. If that is the
case, then we shouldn't continue reaching for the llvm dependency.
To make it easier to understand the case where LLVM support is disabled due to
lacking a C++ compiler, add a message noting that fact.
Author: Tristan Partin <tristan@neon.tech>
Reviewed-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/CSPIJVUDZFKX.3KHMOAVGF94RV@c3po
Backpatch: 16-, where meson support was added
That was already done in a lot of places, but not all.
Backpatch this to keep the the meson files aligned as long as reasonably
possible.
Author: Tristan Partin <tristan@neon.tech>
Reviewed-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/CSPIJVUDZFKX.3KHMOAVGF94RV@c3po
Backpatch: 16-, where meson support was added
All supported computers have either POSIX or Windows threads, and we no
longer have any automated testing of --disable-thread-safety. We define
a vestigial ENABLE_THREAD_SAFETY macro to 1 in ecpg_config.h in case it
is useful, but we no longer test it anywhere in PostgreSQL code, and
associated dead code paths are removed.
The Meson and perl-based Windows build scripts never had an equivalent
build option.
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Peter Eisentraut <peter@eisentraut.org>
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Discussion: https://postgr.es/m/CA%2BhUKGLtmexrpMtxBRLCVePqV_dtWG-ZsEbyPrYc%2BNBB2TkNsw%40mail.gmail.com
locale_t is defined by POSIX.1-2008 and SUSv4, and available on all
targeted systems. For Windows, win32_port.h redirects to a partial
implementation called _locale_t. We can now remove a lot of
compile-time tests for HAVE_LOCALE_T, and associated comments and dead
code branches that were needed for older computers.
Since configure + MinGW builds didn't detect locale_t but now we assume
that all systems have it, further inconsistencies among the 3 Windows build
systems were revealed. With this commit, we no longer define
HAVE_WCSTOMBS_L and HAVE_MBSTOWCS_L on any Windows build system, but
we have logic to deal with that so that replacements are available where
appropriate.
Reviewed-by: Noah Misch <noah@leadboat.com>
Reviewed-by: Tristan Partin <tristan@neon.tech>
Reviewed-by: Peter Eisentraut <peter@eisentraut.org>
Discussion: https://postgr.es/m/CA%2BhUKGLg7_T2GKwZFAkEf0V7vbnur-NfCjZPKZb%3DZfAXSV1ORw%40mail.gmail.com
The comments referring to SSL_CTX_set_cert_cb() in ./configure and
./configure.ac were inconsistent, as `autoreconf -i` would show. While
on it, fix a typo on a comment related to the same check in
meson.build.
Issue introduced in 8e278b6, that removed support for OpenSSL 1.0.1.
Per offlist report from Thomas Munro.
Like bison, this is useful to know which version and command of flex are
detected by meson. For example, this was useful for me when using
win_flex.exe on Windows.
Reviewed-by: Peter Eisentraut
Discussion: https://postgr.es/m/ZDT3FbmrjBuBthmR@paquier.xyz
Here are some notes about this change:
- As X509_get_signature_nid() should always exist (OpenSSL and
LibreSSL), hence HAVE_X509_GET_SIGNATURE_NID is now gone.
- OPENSSL_API_COMPAT is bumped to 0x10002000L.
- One comment related to 1.0.1e introduced by 74242c2 is removed.
Upstream OpenSSL still provides long-term support for 1.0.2 in a closed
fashion, so removing it is out of scope for a few years, at least.
Reviewed-by: Jacob Champion, Daniel Gustafsson
Discussion: https://postgr.es/m/ZG3JNursG69dz1lr@paquier.xyz
run_command(check: true) already would have errorred out before the check is
reached.
Author: Tristan Partin <tristan@neon.tech>
Discussion: CSPIJVUDZFKX.3KHMOAVGF94RV@c3po
Meson validates 'choice' options for us, so technically this case is
impossible. A better error message helps people reading the code
understand what is going on in that branch.
Author: Tristan Partin <tristan@neon.tech>
Discussion: https://www.postgresql.org/message-id/flat/CSPIJVUDZFKX.3KHMOAVGF94RV%40c3po
Tom Lane