We want to add timeline information to the wal keys and cannot easily do
so without affecting existing clusters' relation key files.
This commit does the bare minimum to separate the two completely and as
such contains a fair bit of duplicated code.
The file format for the WAL key file is exactly the same before and
after this commit.
There is _a lot_ of cleanup that will have to be done on both sides of
this separation, but this is a bit of "it gets worse before it gets
better".
This function probably belongs elsewhere than in the key file code, but
that's where it currently resides so expose it so it can also be used
elsewhere.
The code we ran when redoing a smgrcreate() was overly complex and not
necessary to run. If the file descriptor was open we skipped it but if
not we ran a bit of pointless code since the key creation is handled by
it is own WAL record which is always before the SMGR creation.
Also improve some outdated comments.
These tests are copy of original pg_resetwal tests with enalbed WAL
encryption and removed flags validation as we interested here only in
proper enrypted WAL handling.
While the restore_command will never be called for partial WAL segments
we should sitll make sure to decrypt them when archiving so a sysadmin
manually could use them. As the comemnt explains we also add the same
logic to the restore command for symmetry and if someone ever would call
it manually or PostgreSQL would add support for restoring partial WAL
segments.
If you want to learn more about partial WAL segments read the comment
for CleanupAfterArchiveRecovery().
For some to me unknown reason pgbackrest archive-push did not like
reading from a pipe so we create a temporary file instead. The location
for the temporary file is for now hardcoded to be /dev/shm since that is
guaranteed to be on a tmpfs mount on all common Linux distributions. We
might in the future want to make this configurable since it is a Linux
specific thing.
I tested this with PgBackRest 2.55.0.
To support some common WAL archiving tools, e.g. PgBackRest, we
implement an archive_command and a restore_command which can wrap any
command and use pipe() to create fake file to either read from or write
to. The restore command makes sure to write encrypted files if WAL
encryption is enabled. It uses the fresh WAL key generated by the server
on the current start which works fine because we then just let the first
invocation of the restore command set the start LSN of the key.
For e.g. PgBackRest you would have the following commands:
archive_command = 'pg_tde_archive_decrypt %p pgbackrest --stanza=demo archive-push %p'
restore_command = 'pg_tde_restore_encrypt %f %p pgbackrest --stanza=demo archive-get %f "%p"'
Reorganized how the ToC website interaction is done by renaming index.md
from CLI, architecture, advanced topics, KMS and Overview (index
folder). This meant updating links in multiple other files as index
files needed to be renamed for the structure to work.
Reorganized how Overview is displayed, removed RC2 mention in
Limitations, removed important note as it is no longer needed since it
was an RC2 mention.
Reworded the button texts for better interaction and user expectation in
the Overview chapter topics.
Added a short intro for Benefits of pg_tde topic, rewrote admonition.
Updated KMS titles to reflect Percona Style Guide.
Error messages are not supposed to be proper sentences, and according to
the error style guide "cannot" is preferable over "can't".
Also add an appropriate errcode.
With ifdefs all over the place it was hard to expose the write functions
to frontend tools so we reduce the number of ifdefs by having one clear
set of data structures fror backend and one for frontend.
Additionally we give access to WAL key generation and setting the start_lsn
of a key to the frontend code.
Since some frontend tools will need to write WAL while others will not
it makes sense to split the initalization so only some frontend tools
and the backend needs to initialize the WAL write related stuff.
It is a bit unclear what this optimization was supposed to do,
presumably it was to break from the loop once the whole buffer was
decrypted, but the logic was very confused since if we want to do a
similar optimization it should be something like
if (dec_off + read_sz == offset + readsz).
This function counts the number of encryption keys in the key file
associated with the given OID. Name it accordingly.
Also remove comment about only user which is no longer true.
These tests are a carbon copy of upstream's pg_rewind tests. Changes
for WAL encryption will be added in the following commit.
`standby_source` test is not copied as it fails because of changes
needed in pg_basebackup.
The pg_tde_change_key_provider tool should act more like PostgreSQL's
own CLI tools, which includes changing the usage slightly (but not
entirely) to match, making the error messages more similar to
PostgreSQL's and making the code a bit more PG-like.
This commit makes a suppression list as specific as possible, so it won't
cover up new issues. And adds comments to existing suppressions. Also,
fixes memory leaks in bin/pgctl code (all related to frontend usage) and
low-hanging fruits in pgctl.
Added initial files and modifications to include 1.0 release notes to
the TOC and variables. Updates:
* updated the ToC names to make them in line with style guide
* updated variable with new release branch and fixed small release note
name
* updated ## Release Highlights with topics:
* Added tickets
* Updated Upgrade considerations
Updated the Architecture topic with the following:
- New intro detailing the long term tde goals in a paragraph
- Updated the ## Typical setup scenarios topic with better writing and
improved flow
- Added note to WAL Encryption that it is not to be used in prod env
- General small fixes to paragraphs, wrongly written words and such
Updated the introduction with the proper extension name, updated the
intro to reflect this. Removed important note about not meant for
production and added the No upgrade warning from previous versions (like
RC2) to GA. Updates:
* removed the block announcement for RC2 at the top of the HTML page in
the intro
* Added the warning note before installation begins too.
* Updated site name to full name.
In set-principal-key.md:
* updated with correct code example using set_server_key_using_global
parameter
* updated note to reflect correct config
In features.md:
* Removed temporary tables feature to clear confusion, removed logical
replication mention, removed WAL encryption as a feature.
In functions.md:
* Added ON FUNCTION for grant/revoke execution
* Modified sensitive info bolded paragraph to important note
* Small modifications to notes display, title cases and text fixes
* added note to Add or modify Vault providers for keeping the same
principal key.
* Added warning for WAL in pg_tde_create_key_using_global_key_provider
In general:
* Removed all logical replication mentions except the FAQ and in RC2
release note.
Anders Åstrand