Since you can take a copy of a PostgreSQL data directory and start both
the old and the new version you could get two versions where the same
encrypted counter is used for CTR which would mean we could comapre
them and potentially decrypt the data. For this reason we need to
generate a new WAL key every time we start the server.
According to to the manual we should return a null pointer, but it is
contradicted by an example on the same page but let's follow what
plpgsql does.
An event trigger function must return a NULL pointer
(not an SQL null value, that is, do not
set isNull true).
The security of the encryption is reduced if we reuse the same
initiation vector more than necessary so we make sure to use a unique IV
per relation fork, with the exception of the initialization fork which
is used by unlogged indexes when restarting the server after a crash. It
is copied with low-level file system functions to the main fork on crash
recovery so it needs to use the same IV as the main fork.
The init fork issue is in no way more a security issue than to the
extent that ideally we should pick a new IV when truncating unlogged
tables on crash recovery but to fix this we would need to change the
SMGR API and moving the copying of the intialization fork into that. And
in the long term this might be what we want to do.
While the frontend sources right now are a strict subset of the backend
sources that might not always be the case so keep them as separate lists
but clean them up. In the long term we want to refactor this entirely
anyway so let's not overcomplicate this.
Sinc we want to add validation for that the principal keys still all
are there some simple test would not suffice so let's instead wait
until we do that to write propoer tests for this.
We had a bug in the default key roitation which was fixed in commit
cb06bea253 but that commit did not add any
regression test so let's add one where we clear the buffer cache to make
sure we can read the internal keys and use them to decrypt the table
data.
Commit e3a87b4991 added a way to call
ereport() without any extra parentheses which was backported all the way
back to PostgreSQL 12. The new modern way improves code readability
slightly.
When we did a recovery, even if tde_heap was not used, 98% of the time
was spent in pg_tde_get_key_from_file() due to our SMGR missing the
shortcicruit from mdcreate() which skips running if the fork already
is open.
The issue was made worse by us not caching negative SMGR key lookups but
that is the subject of another commit.
This is just a conversion of the google doc into markdown, with
actualizing some of the outdated details in the document.
The last section (researc/investigation topics) is left out, as that
doesn't make much sense in a public documentation.
By using SPI and a SQL language function to force the rewrite of
sequences we made life unnecessarily hard for ourselves and also
intoduced a bug where ALTER TABLE could break if the pg_tde extension's
functions were not in the search path.
Instead we re-use the code for updating sequence persistence whe table
persistance is changed. And we only need to look at the table's
presistance since it is always the same as the one of the sequences.
It's not clear to someone new to the code that "key provider" in these
files refers to what's called "key provider type" elsewhere.
Rename these to make it easier for the next person.
The comment about all-zero pages created by smgrzeroextend() sounded
much scarier than the reality. In fact trying to encrypt these
all-zero pages might not only be a waste of CPU cycles but also could
decrease security by making us re-use the same IV first with all zeros
and then with the actual data. And the extra amount of protection
we gain from encrypting them is minuscule since they are only added
at the end of the table, soon overwritten and only gives the attacker
a very slightly more accurate table size.
The end LSN of the current buffer to decript was exclusive while the end
LSN of the key was inclusive which had led to confusion and an
off-by-one bug.
Also add a simple test case for the WAL encryption using the logical
decoding's test plugin.
We already had protection against decrypting relation keys with the wrong
principal key but to properly protect us against new relation keys being
encrypted with the wrong principal key we need to also verify that the
principal key was correct when we fetch the principal key from the key
provider. We do so by signing the principal key info header of the key map
file using AES-128-GCM.
This way we cannot get a jumbled mess of relation keys encrypted with
multiple different principal keys.
While this does not merge the two similar functions it removes a lot
of cruft from pg_tde_read_one_map_entry2() making the functions more
similar and the code easier to read.
Same as last commit but for the WAL encryption.
Rewrote the calculation in a way gcc's vectorizer likes, as verified
with Godbolt. The code generated by clang is ok and branch free but it
fails to properly vectorize both before and after.
We might as well increase the entropy by adding a random base value to
the IVs used used when encrypting relations. But since for now the pair
of key + IV is generated and used together it adds little extra security
over what we already have.
We add the IV with XOR since that is a cheap and easy operation which
uases no extra collissions.
If we look up something about a relation in the command start event
trigger we need to hold onto that lock until the end of the command
and not release it so no other commd is able to change the data we
looked at between now and when the DDL command itself actually locks
the relation.
The oids worked as-is, in general any reserved catalog oid which is
frefers to an object of another type will work as a special magical
oid. Here we decide to use tablespce oids for special database
values and a database oid, 1, for the relation value. But in the
latter case anything would have worked since we are guarnteed to
have no collissions due to the database oid not being and actual
database.
The old solution which use a mix worked too but this is more
consistent.
The previous 8192 * 100 initial allocation was arbitrary and bigger than
necessary. We pick something bassically as arbitrary but now a multiple of
the actual page size of DSAs, 4 KB.
This comment is pretty strange since we actually do verify, badly, that
there are no encrypted tables.
Additionally we merge two small functions to make the code easier to
understand.
There is no reason to waste memory when WAL encryption is not enabled
sd long as we do not support changing the WAL encryption status while
running. And if we add support for this we can revist this code.
All callers call pg_tde_free_key_map_entry() with MAP_ENTRY_VALID so
we can move it down to the place where it is used instead of passing
it as an argument.
Make sure that we lock the key file before adding a WAL key. It should
be harmless since this happens very early in the start but if we ever
add the ability to rotate WAL keys while running this will become an
issue.
Additionally we add asserts to make sure we hold the right lock level
when we open the key map file.
Andreas Karlsson