Just logging that the function was called at DEBUG2 is not very helpful
to anyone and is presumably jsut a leftover from someone's attempt at
debugging a particular issue they had at some point.
Breaking these particular snippets out as separate functions did not
improve readability and was only done because they use to be called from
multiple locations.
This change has already been done in the WAL key code.
When WAL is streamed during the backup (default mode), it comes in
unencrypted. But we need keys to encrypt it. For now, we expect that
the user would put `pg_tde` dir containing the `1664_key` and
`1664_providers` into the destination directory before starting the
backup. We encrypt the streamed WAL according to internal keys. No
`pg_tde` dir means no streamed WAL encryption.
Also rename enum variants for consistency plus renumber the types for
the WAL keys which is fine since this file is newly introduced which
makes breaking backwards compatibility not an issue.
Before this commit, WAL keys didn't mind TLI at all. But after
pg_rewind, for example, pg_wal/ may contain segments from two
timelines. And the wal reader choosing the key may pick the wrong one
because LSNs of different TLIs may overlap. There was also another bug:
There is a key with the start LSN 0/30000 in TLI 1. And after the start
in TLI 2, the wal writer creates a new key with the SN 0/30000, but in
TLI 2. But the reader wouldn't fetch the latest key because w/o TLI,
these are the same.
This commit adds TLI to the Internal keys and makes use of it along
with LSN for key compares.
The other keys are stored in <oid>_keys so wal_keys fits better into
that pattern than the more redundant wal_encryption_keys where
"encryption" does not add any information but just makes the path
longer.
Instead of 1664_keys it's now called wal_encryption_keys.
This lets us use a constant name for it instead of generating it from an
Oid pretending it's a relation key file.
Also remove some now unused Oid parameters to functions.
We want to add timeline information to the wal keys and cannot easily do
so without affecting existing clusters' relation key files.
This commit does the bare minimum to separate the two completely and as
such contains a fair bit of duplicated code.
The file format for the WAL key file is exactly the same before and
after this commit.
There is _a lot_ of cleanup that will have to be done on both sides of
this separation, but this is a bit of "it gets worse before it gets
better".
This function probably belongs elsewhere than in the key file code, but
that's where it currently resides so expose it so it can also be used
elsewhere.
The code we ran when redoing a smgrcreate() was overly complex and not
necessary to run. If the file descriptor was open we skipped it but if
not we ran a bit of pointless code since the key creation is handled by
it is own WAL record which is always before the SMGR creation.
Also improve some outdated comments.
These tests are copy of original pg_resetwal tests with enalbed WAL
encryption and removed flags validation as we interested here only in
proper enrypted WAL handling.
As pg_resetwal removes old WAL segments and creates new one with empty
record we can do that write in unencrypted mode. However that requires
new WAL key creation in case if encryption was enabled before.
While the restore_command will never be called for partial WAL segments
we should sitll make sure to decrypt them when archiving so a sysadmin
manually could use them. As the comemnt explains we also add the same
logic to the restore command for symmetry and if someone ever would call
it manually or PostgreSQL would add support for restoring partial WAL
segments.
If you want to learn more about partial WAL segments read the comment
for CleanupAfterArchiveRecovery().
For some to me unknown reason pgbackrest archive-push did not like
reading from a pipe so we create a temporary file instead. The location
for the temporary file is for now hardcoded to be /dev/shm since that is
guaranteed to be on a tmpfs mount on all common Linux distributions. We
might in the future want to make this configurable since it is a Linux
specific thing.
I tested this with PgBackRest 2.55.0.
To support some common WAL archiving tools, e.g. PgBackRest, we
implement an archive_command and a restore_command which can wrap any
command and use pipe() to create fake file to either read from or write
to. The restore command makes sure to write encrypted files if WAL
encryption is enabled. It uses the fresh WAL key generated by the server
on the current start which works fine because we then just let the first
invocation of the restore command set the start LSN of the key.
For e.g. PgBackRest you would have the following commands:
archive_command = 'pg_tde_archive_decrypt %p pgbackrest --stanza=demo archive-push %p'
restore_command = 'pg_tde_restore_encrypt %f %p pgbackrest --stanza=demo archive-get %f "%p"'
Reorganized how the ToC website interaction is done by renaming index.md
from CLI, architecture, advanced topics, KMS and Overview (index
folder). This meant updating links in multiple other files as index
files needed to be renamed for the structure to work.
Reorganized how Overview is displayed, removed RC2 mention in
Limitations, removed important note as it is no longer needed since it
was an RC2 mention.
Reworded the button texts for better interaction and user expectation in
the Overview chapter topics.
Added a short intro for Benefits of pg_tde topic, rewrote admonition.
Updated KMS titles to reflect Percona Style Guide.
Error messages are not supposed to be proper sentences, and according to
the error style guide "cannot" is preferable over "can't".
Also add an appropriate errcode.
With ifdefs all over the place it was hard to expose the write functions
to frontend tools so we reduce the number of ifdefs by having one clear
set of data structures fror backend and one for frontend.
Additionally we give access to WAL key generation and setting the start_lsn
of a key to the frontend code.
Since some frontend tools will need to write WAL while others will not
it makes sense to split the initalization so only some frontend tools
and the backend needs to initialize the WAL write related stuff.
It is a bit unclear what this optimization was supposed to do,
presumably it was to break from the loop once the whole buffer was
decrypted, but the logic was very confused since if we want to do a
similar optimization it should be something like
if (dec_off + read_sz == offset + readsz).
Andreas Karlsson