Documentation update (#149)

* Updating documentation with configuration changes

* Minor updates to improve readability

* Added link to test.md for setup doc

* Added documentation about key rotation and remote parameters

---------

Co-authored-by: Anastasia Alexadrova <anastasia.alexandrova@percona.com>

documentation/docs/functions.md

@@ -0,0 +1,116 @@
+# Functions
+
+The `pg_tde` extension provides the following functions:
+
+## pg_tde_add_key_provider_file
+
+Creates a new key provider for the database using a local file.
+
+This function is intended for development, and stores the keys unencrypted in the specified data file.
+
+```sql
+SELECT pg_tde_add_key_provider_file('provider-name','/path/to/the/keyring/data.file');
+```
+
+All parameters can be either strings, or JSON objects referencing remote parameters.
+
+## pg_tde_add_key_provider_vault_v2
+
+Creates a new key provider for the database using a remote HashiCorp Vault server.
+
+The specified access parameters require permission to read and write keys at the location.
+
+```sql
+SELECT pg_tde_add_key_provider_vault_v2('provider-name',:'secret_token','url','mount','ca_path');
+```
+
+where:
+
+* `url` is the URL of the Vault server
+* `mount` is the mount point where the keyring should store the keys
+* `secret_token` is an access token with read and write access to the above mount point
+* [optional] `ca_path` is the path of the CA file used for SSL verification
+
+All parameters can be either strings, or JSON objects referencing remote parameters.
+
+## pg_tde_set_master_key
+
+Sets the master key for the database using the specified key provider.
+
+The master key name is also used for constructing the name in the provider, for example on the remote Vault server.
+
+You can use this function only to a master key. For changes in the master key, use the [`pg_tde_rotate_key`](#pg_tde_rotate_key) function.
+
+```sql
+SELECT pg_tde_set_master_key('name-of-the-master-key', 'provider-name');
+```
+
+## pg_tde_rotate_key
+
+Creates a new version of the specified master key, and updates the database so that it uses the new master key version.
+
+It can be used without any parameters, which will just create a new version of the current database
+master key, using the same provider:
+
+```sql
+SELECT pg_tde_rotate_key();
+```
+
+Or alternatively it can be used with two parameters, specifying both a new key name and a new provider
+name:
+
+```sql
+SELECT pg_tde_rotate_key('name-of-the-new-master-key', 'name-of-the-new-provider');
+```
+
+In this case, both parameters support the `NULL` value, which means that parameter won't be changed:
+
+```sql
+-- creates  new master key on the same provider as before
+SELECT pg_tde_rotate_key('name-of-the-new-master-key', NULL);
+
+-- copies the current master key to a new provider
+SELECT pg_tde_rotate_key(NULL, 'name-of-the-new-provider');
+```
+
+## pg_tde_is_encrypted
+
+Tells if a table is using the `pg_tde` access method or not.
+
+```sql
+SELECT pg_tde_is_encrypted('table_name');
+```
+
+
+# JSON objects as remote parameters
+
+To allow storing secrets, or any other parameters in a more secure, external location, `pg_tde`
+allows users to specify an external reference instead of hardcoded parameters.
+
+Currently `pg_tde` supports two external storage methods:
+
+* `file`, which just stores the data in a simple file specified by a `path`. The file should be
+readable to the postgres process.
+* `remote`, which uses a HTTP request to retrieve the parameter from the specified `url`.
+
+As an example, to use the file provider with a file location specified by the `remote` method,
+use the following command:
+
+```sql
+SELECT pg_tde_add_key_provider_file(
+    'file-provider', 
+    json_object( 'type' VALUE 'remote', 'url' VALUE 'http://localhost:8888/hello' )
+    );"
+```
+
+Or to use the `file` method, use the following command:
+
+```sql
+SELECT pg_tde_add_key_provider_file(
+    'file-provider', 
+    json_object( 'type' VALUE 'remote', 'path' VALUE '/tmp/datafile-location' )
+    );"
+```
+
+Any parameter specified to the `add_key_provider` functions can be a json_object instead of the string,
+similar to the above examples.

documentation/docs/index.md

@@ -1,10 +1,10 @@
-# `pg_tde` documentation
+# `pg_tde` documentation [Tech preview]
 
-`pg_tde` is the extension that brings in [Transparent Data Encryption (TDE)](tde.md) to PostgreSQL and enables users to keep sensitive data safe and secure. 
+`pg_tde` is the extension that brings in [Transparent Data Encryption (TDE)](tde.md) to PostgreSQL and enables users to keep sensitive data safe and secure. It enables users to configure encryption differently for each database, encrypting specific tables in some databases with different encryption keys, while keeping others non encrypted. 
 
 !!! important 
 
-    This is the MVP version of the extension and is not meant for production use yet.
+    This is the tech preview version of the extension and is not meant for production use yet.
 
 [What's new](release-notes/tech-preview.md){.md-button}
 
@@ -22,7 +22,7 @@
 * Keys in the local keyfile are stored unencrypted.
 * Indexes and `NULL` bitmaps of tuples are currently not encrypted. 
 
-<i warning>:material-alert: Warning:</i> Note that introducing encryption/decryption affects performance. Our benchmark tests show less than 10% performance overhead.
+<i warning>:material-alert: Warning:</i> Note that introducing encryption/decryption affects performance. Our benchmark tests show less than 10% performance overhead for most situations. However, in some specific applications such as those using JSONB operations, performance degradation might be higher.
 
 [Get started](install.md){.md-button}
 
@@ -35,8 +35,6 @@
 The following is planned for future releases of `pg_tde`:
 
 * Encryption of indexes and `NULL` bitmaps of tuples
-* Master key rotation
-* Multi-tenancy support
 * Logical replication support
 
 

documentation/docs/setup.md

@@ -1,4 +1,4 @@
-# Setup
+# Set up `pg_tde`
 
 Load the `pg_tde` at the start time. The extension requires additional shared memory; therefore,  add the `pg_tde` value for the `shared_preload_libraries` parameter and restart the `postgresql` instance.
 
@@ -38,68 +38,83 @@ Load the `pg_tde` at the start time. The extension requires additional shared me
         psql -d template1 -c 'CREATE EXTENSION pg_tde;'
         ```
 
-4. Set the location of the keyring configuration file in postgresql.conf: `pg_tde.keyringConfigFile = '/where/to/put/the/keyring.json'`
-5. Create the [keyring configuration file](#keyring-configuration)
-6. Start or restart the `postgresql` instance to apply the changes.
+4. Set up a key provider for the database where you have enabled the extension
 
-    * On Debian and Ubuntu:    
+    === "With HaschiCorp Vault"
 
-       ```sh
-       sudo systemctl restart postgresql.service
-       ```
-    
-    * On RHEL and derivatives
+        ```sql
+        SELECT pg_tde_add_key_provider_vault_v2('provider-name',:'secret_token','url','mount','ca_path');
+        ``` 
 
-       ```sh
-       sudo systemctl restart postgresql-16
-       ```
+        where: 
 
-7. You are all set to create encrypted tables. For that, specify `USING pg_tde` in the `CREATE TABLE` statement.
-**For example**:
-```sql
-CREATE TABLE albums (
-    album_id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
-    artist_id INTEGER,
-    title TEXT NOT NULL,
-    released DATE NOT NULL
-) USING pg_tde;
-```
+        * `url` is the URL of the Vault server
+        * `mount` is the mount point where the keyring should store the keys
+        * `secret_token` is an access token with read and write access to the above mount point
+        * [optional] `ca_path` is the path of the CA file used for SSL verification
 
-## Keyring configuration
 
-Create the keyring configuration file with the following contents:
+    === "With keyring file"
 
-=== "HashiCorp Vault"
+        This setup is intended for development and stores the keys unencrypted in the specified data file.    
 
-     ```json
-     {
-             "provider": "vault-v2",
-             "token": "ROOT_TOKEN",
-             "url": "http://127.0.0.1:8200",
-             "mountPath": "secret",
-             "caPath": "<path/to/caFile>"
-     }
-     ```
+        ```sql
+        SELECT pg_tde_add_key_provider_file('provider-name','/path/to/the/keyring/data.file');
+        ```
+       
+       
+5. Add a master key
 
-     where:
+    ```sql
+    SELECT pg_tde_set_master_key('name-of-the-master-key', 'provider-name');
+    ```
+## Optional: use external parameters
+
+When configuring `pg_tde` with the steps described above, the provider configuration specified in
+step 4 is stored in the database catalog, in an unencrypted table. This is not secure to store
+sensitive parameters, such as vault secrets.
+
+To allow storing secrets, or any other parameters in a more secure, external location, `pg_tde`
+allows users to specify an external reference instead of hardcoded parameters.
+
+Currently `pg_tde` supports two external storage methods:
+
+* `file`, which just stores the data in a simple file specified by a `path`. The file should be
+readable to the postgres process.
+* `remote`, which uses a HTTP request to retrieve the parameter from the specified `url`.
 
-     * `provider` is set to `vault-v2` since only the version 2 of the KV secrets engine is supported
-     * `url` is the URL of the Vault server
-     * `mountPath` is the mount point where the keyring should store the keys
-     * `token` is an access token with read and write access to the above mount point
-     * [optional] `caPath` is the path of the CA file used for SSL verification
+As an example, to use the file provider with a file location specified by the `remote` method,
+use the following command:
 
-=== "Local keyfile"
+```sql
+SELECT pg_tde_add_key_provider_file(
+    'file-provider', 
+    json_object( 'type' VALUE 'remote', 'url' VALUE 'http://localhost:8888/hello' )
+    );"
+```
+
+Or to use the `file` method, use the following command:
 
-     ```json
-     {
-             "provider": "file",
-             "datafile": "/tmp/pgkeyring"
-     }
-     ```     
+```sql
+SELECT pg_tde_add_key_provider_file(
+    'file-provider', 
+    json_object( 'type' VALUE 'remote', 'path' VALUE '/tmp/datafile-location' )
+    );"
+```
 
-     This keyring configuration has the file provider, with a single datafile parameter.     
+Any parameter specified to the `add_key_provider` functions can be a json_object instead of the string,
+similar to the above examples.
+
+6. You are all set to create encrypted tables. For that, specify `USING pg_tde` in the `CREATE TABLE` statement.
+**For example**:
+```sql
+CREATE TABLE albums (
+    album_id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+    artist_id INTEGER,
+    title TEXT NOT NULL,
+    released DATE NOT NULL
+) USING pg_tde;
 
-     This datafile is created and managed by PostgreSQL, the only requirement is that `postgres` should be able to write to the specified path.     
+## Next steps
 
-     This setup is intended for development, and stores the keys unencrypted in the specified data file.
+[Test TDE](test.md){.md-button}

documentation/docs/test.md

@@ -19,7 +19,17 @@ To check if the data is encrypted, do the following:
 2. Run the following function:
 
     ```sql
-    select pgtde_is_encrypted('table_name');
+    SELECT pg_tde_is_encrypted('table_name');
     ```
 
-    The function returns `t` if the table is encrypted and `f` - if not.
+    The function returns `t` if the table is encrypted and `f` - if not.
+
+3. Rotate the master key when needed:
+
+    ```sql
+    SELECT pg_tde_rotate_key(); -- uses automatic key versionin
+    -- or
+    SELECT pg_tde_rotate_key('new-master-key', NULL); -- specify new key name
+    -- or
+    SELECT pg_tde_rotate_key('new-master-key', 'new-provider'); -- change provider
+    ```

documentation/mkdocs.yml

@@ -144,6 +144,9 @@ nav:
     - "Install": "install.md"
     - "Set up": "setup.md"
     - Test TDE: "test.md"
+  - functions.md
+  - How to:
+    - Configure streaming replication: replication.md
   - Release notes:
     - "pg_tde tech preview": release-notes/release-notes.md
   - uninstall.md