There was a mistake in the new cap calculations during the cache extension. It popped up only when a new cache size was multiple of a cache record (every 256 records). Which lead to the usage of the memory beyond an allocated size. This commit fixes it along with mlock size for reallocated pages.
Also fixed a typo in a variable name.
Fixes PG-1248
Issue: pg_upgrade and some other tests are failing
This is caused by some mistake during the 17.2 rebase, where the commits
that fixed this issue previously, by changing the version and product name,
simply disappeared.
Fix: reintroduce the same changes, with this, the format of the name / version
is the same as for 17.0
This also modifies the pg_waldump build so it is now correctly uses
pg_tde as a static library, to handle warnings and custom flags for KMIP correctly.
This commit also enables generic -Werror for meson builds, so that we are forced
to fix warnings before merging in the future.
This commit:
- Adds build infrastructure for the frontend tools to compile with pg_tde. pg_tde dependencies will be built only if build flag `precone_ext` is on. It also makes `openssl` and `curl` required if `precone_ext` is on.
- Makes pg_waldump work with the TDE encrypted WAL. If user set `-k` flag it will try to init tde keys etc and decode encrypted pages. If no `-k` options set, it behaves like community version - won't try to ini tde and won't be able to decrypt WAL.
- Adds tap tests for encrypted WAL to pg_waldump.
- Fixes Percona versioning in ./configure
For PG-1003, PG-1005
Depends https://github.com/percona/pg_tde/pull/362
This commit:
1. Removes autoconf builds for pg_tde so it can be together built with Postgres (now, used would have to go to contrib/pg_tde and build it explicitly after building Postgres) with make. There are still left pg_tde builds in CI tests since this PR depends on https://github.com/percona/postgres/pull/20. So those leftover will be removed after merging PG PR
2. Adds necessary changes regarding new code (like kmip) so frontend tools (pg_waldump et al) can be compiled with pg_tde
2. Get rid of realpath as it has issues with optimised builds
For: PG-1003, PG-1005
Also includes some refactoring because libkmip and postgres headers
are not compatible. To avoid compilation errors, keyring_kmip.c does
not include postgres headers, and keyring_kmip_ereport.c does not
include libkmip headers.
We use tablespaceId as a part of IV for the internal keys encryption
which doesn't add any security because dbId (used as well) is unique
anyway.
But having tablespaceId really complicates things as a principal
created for the entire database but then different relations in this db
can be located in different tablespaces...
So it is better not to use tablespace with the principal key (database
level) as it belongs to the relation level.
When a relation moved to a new location it causes the change of relfilenode id
for it. Hence we must re-encrypt and store its internal key with the new id.
Also, we have to store the changed internal key in the new physical location,
and copy there principal key info and keyring data.
Fixes https://perconadev.atlassian.net/browse/PG-1038
* PG-1058 Fix MergeJoin issue
Resolved an issue in MergeJoin by ensuring the decrypted buffer contents are
also copied from the source to the destination tuple slot during
slot copy operations.
Co Authored by:
Andrew Pogrebnoy <absourd.noise@gmail.com>
Artem Gavrilov <artem.gavrilov@percona.com>
* Create table always checked the principal key and tried to create
it, event when we didn't try to create a tde_heap table
* Alter table wasn't handled, and because of this changing a table
to tde_heap access method didn't result in an encrypted table.
* defaut_table_access_method wasn't handled, and because of this,
creating a table using that also resulted in a non encrypted
table.
* PG-1056 Add failing test
* PG-1056 Use proper AM in test
* Fix UPDATE SET ... RETURNING processing for encrypted tuples
If `get_heap_tuple` is NULL, the core uses `copy_heap_tuple` instead. The former returns a pointer to a tuple in the slot and the latter makes a copy of such a tuple. For UPDATE SET, the core uses the slot for INSERT and later for RETURNING processing. If we copy the tuple the next happens:
1. The core creates a slot with the generic tuple.
2. It passed to `pg_tdeam_tuple_update()` and it gets a copy of the tuple here [6d4f7e5b7b/src17/access/pg_tdeam_handler.c (L336)].
3. This generic tuple is filled with the proper data and used for the update here [6d4f7e5b7b/src17/access/pg_tdeam_handler.c (L343)].
4. Later on, RETURNING processing uses the slot's tuple but is still a generic unmodified one because of the copy.
5. That results in wrong RETURNING data.
To avoid this, we should return a pointer to the slot's tuple instead of copying it.
Fixes PG-1056
* PG-1056 Split 'update' testcase for tde_heap and tde_heap_basic
---------
Co-authored-by: Andrew Pogrebnoy <absourd.noise@gmail.com>
The original part of the version is kept as is, and Percona specific
information is added after.
For example, psql displays the following after this commit:
SELECT version();
PostgreSQL 17.0.1 (Percona Server for PostgreSQL) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 6
4-bit
Where the addition is ".1 (Percona Server for PostgreSQL)", with the
".1" being the Percona version number.
Similarly, commands show the same information:
bin/pg_ctl --version
pg_ctl (PostgreSQL) 17.0.1 (Percona Server for PostgreSQL)
* Make related code compilable with frontend
This commit makes the code around keyring, principal keys and WAL
encryption compilable with frontend tools. Namely:
- Hide everything that isn't compatible and of no use behind
'#ifndef FRONTEND'
- Redefine code that is needed in both versions but should have
different code. E.g. error handling, file descriptors and locks
- Make use of frontend lists instead of backend ones where needed.
For https://perconadev.atlassian.net/browse/PG-857
The current method of detecting perl build flags breaks if the path to
perl contains a space. This change makes two improvements. First,
instead of getting a list of ldflags and ccdlflags and then trying to
filter those out of the reported ldopts, we tell perl to suppress
reporting those in the first instance. Second, it tells perl to parse
those and output them, one per line. Thus any space on the option in a
file name, for example, is preserved.
Issue reported off-list by Muralikrishna Bandaru
Discussion: https://postgr.es/01117f88-f465-bf6c-9362-083bd72ca305@dunslane.net
Backpatch to release 16.
Latest versions of Strawberry Perl define USE_THREAD_SAFE_LOCALE, and we
therefore get a handshake error when building against such instances.
The solution is to perform a test to see if USE_THREAD_SAFE_LOCALE is
defined and only define NO_THREAD_SAFE_LOCALE if it isn't.
Backpatch the meson.build fix back to release 16 and apply the same
logic to Mkvcbuild.pm in releases 12 through 16.
Original report of the issue from Muralikrishna Bandaru.
* PG-853: Access control of pg_tde SQL functions
Add SQL interfaces for granting and revoking access to key management and viewer
functions. This commit introduces four new SQL functions to manage access to
key-related functionalities in the `pg_tde` extension:
- `pg_tde_grant_key_management_to_role`: Grants execute permissions on key
management functions to the specified user or role.
- `pg_tde_revoke_key_management_from_role`: Revokes execute permissions on
key management functions from the specified user or role.
- `pg_tde_grant_key_viewer_to_role`: Grants execute permissions on key
viewer functions to the specified user or role.
- `pg_tde_revoke_key_viewer_from_role`: Revokes execute permissions on
key viewer functions from the specified user or role.
Additionally, upon creating the extension, all execute permissions are revoked
from the `PUBLIC` role. Therefore, a superuser must explicitly grant the
necessary permissions to non-superusers to access these functions after the
extension is created.
These additions provide a more controlled and secure way to manage permissions
for key management and viewer functionalities within the extension.
This commit replaces dependency of the keyring code on JSON backend
functions with common JSON API.
Usage of the backend JSON funcs prevents the code to be used by
frontend tools like pg_waldump. Besides it requiers extra conversions
to Datums and DirectFunctionCall-s.
For: https://perconadev.atlassian.net/browse/PG-857
Recent commits in the PG17 code added additional API changes,
making the "single src directory with ifdefs" approach inpractical.
This commit adds a new python based script (documented with comments
in the file) to help with version specific merges, where the copied
heap files reside in srcXX directories, where XX is the version.
- Rename database key rotation functions to make room for the global space ones.
- Now, during the first start, we would create a default temporary key provider for the global space. A user can (and should) create their own key provider afterwards. This allows use the same codepath and internal interfaces for the keyring management across databases and the global space.
- Now need to cache the principal key for the global space as we use it only at the server start to decrypt internal key. Then internal key persists in the memory cache.
Fixes https://perconadev.atlassian.net/browse/PG-835, https://perconadev.atlassian.net/browse/PG-833
To build with -Dreadline=enabled one can use either readline or
libedit. The -Dlibedit_preferred flag is supposed to control the order
of names to lookup. This works fine when either both libraries are
present or -Dreadline is set to auto. However, explicitly enabling
readline with only libedit present, but not setting libedit_preferred,
or alternatively enabling readline with only readline present, but
setting libedit_preferred, too, are both broken. This is because
cc.find_library will throw an error for a not found dependency as soon
as the first required dependency is checked, thus it's impossible to
fallback to the alternative.
Here we only check the second of the two dependencies for
requiredness, thus we only fail when none of the two can be found.
Author: Wolfgang Walther
Reviewed-by: Nazir Bilal Yavuz, Alvaro Herrera, Peter Eisentraut
Reviewed-by: Tristan Partin
Discussion: https://www.postgresql.org/message-id/ca8f37e1-a2c3-40e2-91f6-59c3d3652ad4@technowledgy.de
Backpatch: 16-, where meson support was added
Passing an absolute bindir/libdir will install the binaries and
libraries to <build>/tmp_install/<bindir> and
<build>/tmp_install/<libdir> respectively.
This path is correctly passed to the regression test suite via
configure/make, but not via meson, yet. This is because the "/"
operator in the following expression throws away the whole left side
when the right side is an absolute path:
test_install_location / get_option('libdir')
This was already correctly handled for dir_prefix, which is likely
absolute as well. This patch handles both bindir and libdir in the
same way - prefixing absolute paths with the tmp_install path
correctly.
Author: Wolfgang Walther
Reviewed-by: Nazir Bilal Yavuz, Alvaro Herrera, Peter Eisentraut
Reviewed-by: Tristan Partin
Discussion: https://www.postgresql.org/message-id/ca8f37e1-a2c3-40e2-91f6-59c3d3652ad4@technowledgy.de
Backpatch: 16-, where meson support was added
Some distributions put clang into a different path than the llvm
binary path.
For example, this is the case on NixOS / nixpkgs, which failed to find
clang with meson before this patch.
Author: Wolfgang Walther
Reviewed-by: Nazir Bilal Yavuz, Alvaro Herrera, Peter Eisentraut
Reviewed-by: Tristan Partin
Discussion: https://www.postgresql.org/message-id/ca8f37e1-a2c3-40e2-91f6-59c3d3652ad4@technowledgy.de
Backpatch: 16-, where meson support was added
The upstream name for the ossp-uuid package / pkg-config file is
"uuid". Many distributions change this to be "ossp-uuid" to not
conflict with e2fsprogs.
This lookup fails on distributions which don't change this name, for
example NixOS / nixpkgs. Both "ossp-uuid" and "uuid" are also checked
in configure.ac.
Author: Wolfgang Walther
Reviewed-by: Nazir Bilal Yavuz, Alvaro Herrera, Peter Eisentraut
Reviewed-by: Tristan Partin
Discussion: https://www.postgresql.org/message-id/ca8f37e1-a2c3-40e2-91f6-59c3d3652ad4@technowledgy.de
Backpatch: 16-, where meson support was added
OpenSSL supports two types of session tickets for TLSv1.3, stateless
and stateful. The option we've used only turns off stateless tickets
leaving stateful tickets active. Use the new API introduced in 1.1.1
to disable all types of tickets.
Backpatch to all supported versions.
Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi>
Reported-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20240617173803.6alnafnxpiqvlh3g@awork3.anarazel.de
Backpatch-through: v12
Particularly on windows it's useful to look up dependencies via cmake, instead
of pkg-config. Meson supports doing so. Unfortunately the dependency names
used by various projects often differs between their pkg-config and cmake
files.
This would look a lot neater if we could rely on meson >= 0.60.0...
Reviewed-by: Tristan Partin <tristan@partin.io>
Discussion: https://postgr.es/m/20240709065101.xhc74r3mdg2lmn4w@awork3.anarazel.de
Backpatch: 16-, where meson support was added
This is necessary as ossp-uuid on windows installs neither a pkg-config nor a
cmake dependency information. Nor is there another supported uuid
implementation available on windows.
Reported-by: Dave Page <dpage@pgadmin.org>
Reviewed-by: Tristan Partin <tristan@partin.io>
Discussion: https://postgr.es/m/20240709065101.xhc74r3mdg2lmn4w@awork3.anarazel.de
Backpatch: 16-, where meson support was added
This is required as MIT Kerberos does provide neither pkg-config nor cmake
dependency information on windows.
Reported-by: Dave Page <dpage@pgadmin.org>
Reviewed-by: Tristan Partin <tristan@partin.io>
Discussion: https://postgr.es/m/20240709065101.xhc74r3mdg2lmn4w@awork3.anarazel.de
Backpatch: 16-, where meson support was added
These were missing since the initial introduction of the meson based build, in
e6927270cd. As-is this is unlikely to cause an issue, but a future commit
will add support for detecting gssapi without use of dependency(), which could
fail due to this.
Discussion: https://postgr.es/m/20240708225659.gmyqoosi7km6ysgn@awork3.anarazel.de
Backpatch: 16-, where the meson based build was added
Zsolt Parragi