* Remove 'percona_tde.pg_tde_key_provider' user catalog and introduce a provider info file for key providers
This commit removes the 'percona_tde.pg_tde_key_provider' user catalog and
replaces it with a provider info file to save key providers.
This change ensures that the key provider information can be accessed
during recovery, the user catalogs cannot be relied upon in such scenarios.
The commit maintains the current API functions, so callers will not experience
any differences in functionality or usage after this change.
Additionally, the commit adjusts how the shared memory manager retrieves
information about the number of LWLocks required by the extension, optimizing
the process.
TODO: Implement xlog message for cleaning up the provider info file during
recovery operations to ensure consistency and avoid potential issues.
Add key management for WAL
Make the *.map *.dat processing code aware of custom databases and
table spaces
Add XLog GUC and init the keyring based on that. Only FS for now
Make the internal/external key infrastructure work with custom
(not stored in the database) keyrings.
Check and create an internal key for XLog during the server start.
If the key is created (not the first start with the EncryptWAL), then
upload it into the cache. We can't read the key from files while
writing the XLog to the disk as it happens in the critical section and
no palloc is allowed.
Create a custom cache for the global catalog external key as we can't
use PG's hashmap during the (again, no pallocs in critical section).
During the server start, when pg_tde module is loading and it needs to
read *.map, *.dat file, InitFileAccess is yet to be called, hence Vfd
isn't ready to use. The same gonna happen during recovery. So use raw
pread/pwrite calls istead.
During the server start, when pg_tde module is loading and it needs to
read *.map, *.dat file, InitFileAccess is yet to be called, hence Vfd
isn't ready to use. The same gonna happen during recovery. So use raw
pread/pwrite calls istead.
* Check and create an internal key for XLog during the server start.
If the key is created (not the first start with the EncryptWAL), then
upload it into the cache. We can't read the key from files while
writing the XLog to the disk as it happens in the critical section and
no palloc is allowed.
* Create a custom cache for the global catalog external key as we can't
use PG's hashmap during the (again, no pallocs in critical section).
* Make the *.map *.dat processing code aware of custom databases and
table spaces
* Add XLog GUC and init the keyring based on that. Only FS for now
* Make the internal/external key infrastructure work with custom
(not stored in the database) keyrings.
* TDE TupleTableSlot for storing decrypted tuple along with the buffer tuple
Tuple data in the shared buffer is encrypted. To store the tuple in the
tupleTableslot, the tuple data is decrypted into allocated memory. This memory
needs to be properly cleaned up. However, with the existing
BufferHeapTupleTableSlot, there is no way to free this memory until the end of
the current query executor cycle.
To address this, the commit introduces TDEBufferHeapTupleTableSlot, a clone of
BufferHeapTupleTableSlot that keeps a reference to the allocated decrypted tuple
and frees it when the tuple slot is cleared. Most of the code is borrowed from
the BufferHeapTupleTableSlot implementation, ensuring that
TDEBufferHeapTupleTableSlot can be cast to BufferHeapTupleTableSlot
Apart from the above, a workaround to clear the decrypted tuple pointer
is added to the TDEBufferHeapTupleTableSlot for cases when the
slot is reused while the previously decrypted tuple was cleared out by
MemoryContext deletion, instead of through the slot cleanup callback.
* Introduces `pg_tde2` access method
* New access method uses the event trigger changes from #196
* Keys are now loaded from the keyring
* This required changes to the map file / master key infrastructure
* This commit only modifies/fixes those as little as required for simplicity.
More refactoring/changes coming in separate commits / PRs
* Removes reliance from MyDatabaseId, as things now have to work with multiple databases in the checkpointer
* Removes some error reports, where functions should work even without a configured keyring
* Fixes some bugs in the map file functions
* Map file functions now work with multiple databases in a single process, but this is a hackish solution, global state needs a proper refactoring
* Contains anti-recursion hack in the new SMGR code, which is needed until we store the metadata in the catalog
This commit implements ddl-start and ddl-end event triggers to identify index
creation operations on encrypted tables. Upon creating an index on an encrypted
table, the trigger function updates the global state, which can be accessed by
the storage manager (mgr) to decide if smgr_create needs to do encryption or not.
The start-ddl function analyzes the CREATE TABLE and CREATE INDEX statements
and identifies if the table uses the pg_tde access method. When the table is
created or the one on which the index is being created utilizes the
pg_tde access method, the start-ddl trigger function populates relevant
information about the encrypted table into a global structure.
This structure can be accessed using the GetCurrentTdeCreateEvent() function.
After the execution of the current DDL command finishes, the end-ddl
function clears out this structure.
* Encrypts XLog pages while writing them to the segment file.
* Add WAL ecryption GUC.
* Both streaming and logical replication are working.
* TODO: needs key management.
Updated logo and favicon
Added extra ref to doc homepage
new file: documentation/_resource/overrides/partials/header.html
deleted: documentation/docs/_images/percona-favicon.ico
deleted: documentation/docs/_images/percona-logo.svg
new file: documentation/docs/_images/postgresql-fav.svg
new file: documentation/docs/_images/postgresql-mark.svg
modified: documentation/docs/css/design.css
new file: documentation/docs/css/landing.css
new file: documentation/docs/css/postgresql.css
modified: documentation/mkdocs.yml
Because the current smgr patchset is based on the PG17 branch, smgr
development needs to be based on that. But our current copied PG16
code doesn't compile with PG17.
This commit doesn't try to make the pg_tde AM work with PG17, only
"comments out" with conditional commands parts that do not compile.
PG16 builds work as before, without any changes.
PG17 at least compiles with missing code, probably doesn't really work.
* Fix issue-153: Server crash and database corruption
We can't use the Tuple CID as an IV because it changes when the tuple is deleted.
If we have a trigger function that needs the deleted tuple, it will get the
wrong IV when decrypting. This happens because the CID used to encrypt the tuple
(during INSERT/UPDATE) is different from the CID passed to the decryption
function (during delete).
To fix this, we need to stop using the CID for IV calculation.
* Update test case to produce same result on all environment
* Updating documentation with configuration changes
* Minor updates to improve readability
* Added link to test.md for setup doc
* Added documentation about key rotation and remote parameters
---------
Co-authored-by: Anastasia Alexadrova <anastasia.alexandrova@percona.com>
Muhammad Usama