PG-1241 Documented KMIP integration and setup (#368)

PG-1241 Documented KMIP integration and setup

 modified:   documentation/docs/_images/tde-flow.png
        modified:   documentation/docs/setup.md
        modified:   documentation/docs/tde.md

documentation/docs/functions.md

@@ -21,7 +21,7 @@ Creates a new key provider for the database using a remote HashiCorp Vault serve
 The specified access parameters require permission to read and write keys at the location.
 
 ```
-SELECT pg_tde_add_key_provider_vault_v2('provider-name',:'secret_token','url','mount','ca_path');
+SELECT pg_tde_add_key_provider_vault_v2('provider-name','secret_token','url','mount','ca_path');
 ```
 
 where:
@@ -33,6 +33,24 @@ where:
 
 All parameters can be either strings, or JSON objects [referencing remote parameters](external-parameters.md).
 
+## pg_tde_add_key_provider_kmip
+
+Creates a new key provider for the database using a remote KMIP server.
+
+The specified access parameters require permission to read and write keys at the server.
+
+```
+SELECT pg_tde_add_key_provider_kmip('provider-name','kmip-IP', 5696, '/path_to/server_certificate.pem', '/path_to/client_key.pem');
+```
+
+where:
+
+* `provider-name` is the name of the provider. You can specify any name, it's for you to identify the provider.
+* `kmip-IP` is the IP address of a domain name of the KMIP server
+* The port to communicate with the KMIP server. The default port is `5696`.
+* `server-certificate` is the path to the certificate file for the KMIP server.
+* `client key` is the path to the client key.
+
 ## pg_tde_set_principal_key
 
 Sets the principal key for the database using the specified key provider.

documentation/docs/setup.md

@@ -4,7 +4,7 @@
 
 Load the `pg_tde` at the start time. The extension requires additional shared memory; therefore,  add the `pg_tde` value for the `shared_preload_libraries` parameter and restart the `postgresql` instance.
 
-1. Use the [ALTER SYSTEM](https://www.postgresql.org/docs/current/sql-altersystem.html) command from `psql` terminal to modify the `shared_preload_libraries` parameter.
+1. Use the [ALTER SYSTEM](https://www.postgresql.org/docs/current/sql-altersystem.html) command from `psql` terminal to modify the `shared_preload_libraries` parameter. This requires superuser privileges. 
 
     ```
     ALTER SYSTEM SET shared_preload_libraries = 'pg_tde';
@@ -14,14 +14,14 @@ Load the `pg_tde` at the start time. The extension requires additional shared me
 
     * On Debian and Ubuntu:    
 
-       ```{.bash data-prompt="$"}
-       $ sudo systemctl restart postgresql.service
+       ```sh
+       sudo systemctl restart postgresql.service
        ```
     
     * On RHEL and derivatives
 
-       ```{.bash data-prompt="$"}
-       $ sudo systemctl restart postgresql-17
+       ```sh
+       sudo systemctl restart postgresql-17
        ```
 
 3. Create the extension using the [CREATE EXTENSION](https://www.postgresql.org/docs/current/sql-createextension.html) command. You must have the privileges of a superuser or a database owner to use this command. Connect to `psql` as a superuser for a database and run the following command:
@@ -36,7 +36,7 @@ Load the `pg_tde` at the start time. The extension requires additional shared me
 
         You can have the `pg_tde` extension automatically enabled for every newly created database. Modify the template `template1` database as follows: 
 
-        ```
+        ```sh
         psql -d template1 -c 'CREATE EXTENSION pg_tde;'
         ```
 
@@ -44,65 +44,95 @@ Load the `pg_tde` at the start time. The extension requires additional shared me
 
 1. Set up a key provider for the database where you have enabled the extension.
 
-    === "With HashiCorp Vault"
+    === "With KMIP server"
+
+        Make sure you have obtained the root certificate for the KMIP server and the keypair for the client. The client key needs permissions to create / read keys on the server. Find the [configuration guidelines for the HashiCorp Vault Enterprise KMIP Secrets Engine](https://developer.hashicorp.com/vault/tutorials/enterprise/kmip-engine).
+        
+        For testing purposes, you can use the PyKMIP server which enables you to set up required certificates. To use a real KMIP server, make sure to obtain the valid certificates issued by the key management appliance. 
+
+        ```
+        SELECT pg_tde_add_key_provider_kmip('provider-name','kmip-IP', 5696, '/path_to/server_certificate.pem', '/path_to/client_key.pem');
+        ```
+
+        where:
+
+        * `provider-name` is the name of the provider. You can specify any name, it's for you to identify the provider.
+        * `kmip-IP` is the IP address of a domain name of the KMIP server
+        * `port` is the port to communicate with the KMIP server. Typically used port is 5696.
+        * `server-certificate` is the path to the certificate file for the KMIP server.
+        * `client key` is the path to the client key.
+
+        <i warning>:material-information: Warning:</i> This example is for testing purposes only:
 
         The Vault server setup is out of scope of this document.
 
         ```
-        SELECT pg_tde_add_key_provider_vault_v2('provider-name',:'secret_token','url','mount','ca_path');
+        SELECT pg_tde_add_key_provider_kmip('kmip','127.0.0.1', 5696, '/tmp/server_certificate.pem', '/tmp/client_key_jane_doe.pem');
+        ```
+
+    === "With HashiCorp Vault"
+
+        The Vault server setup is out of scope of this document.
+
+        ```sql
+        SELECT pg_tde_add_key_provider_vault_v2('provider-name','root_token','url','mount','ca_path');
         ``` 
 
         where: 
 
         * `url` is the URL of the Vault server
         * `mount` is the mount point where the keyring should store the keys
-        * `secret_token` is an access token with read and write access to the above mount point
+        * `root_token` is an access token with read and write access to the above mount point
         * [optional] `ca_path` is the path of the CA file used for SSL verification
 
+        <i warning>:material-information: Warning:</i> This example is for testing purposes only:
 
-    === "With keyring file"
+	    ```
+	    SELECT pg_tde_add_key_provider_file_vault_v2('my-vault','https://vault.example.com','secret/data','hvs.zPuyktykA...example...ewUEnIRVaKoBzs2', NULL);
+	    ```
+
+    === "With a keyring file"
 
         This setup is intended for development and stores the keys unencrypted in the specified data file.    
 
-        ```
+        ```sql
         SELECT pg_tde_add_key_provider_file('provider-name','/path/to/the/keyring/data.file');
         ```
 
-	<i warning>:material-information: Warning:</i> This example is for testing purposes only:
+	    <i warning>:material-information: Warning:</i> This example is for testing purposes only:
 
-	```
-	SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_local_keyring.per');
-	```
+	    ```sql
+	    SELECT pg_tde_add_key_provider_file('file-keyring','/tmp/pg_tde_test_local_keyring.per');
+	    ```
        
        
 2. Add a principal key
 
-    ```
+    ```sql
     SELECT pg_tde_set_principal_key('name-of-the-principal-key', 'provider-name');
     ```
 
     <i warning>:material-information: Warning:</i> This example is for testing purposes only:
 
-    ```
+    ```sql
     SELECT pg_tde_set_principal_key('test-db-master-key','file-vault');
     ```
 
     The key is auto-generated.
 
-   <i info>:material-information: Info:</i> The key provider configuration is stored in the database catalog in an unencrypted table. See [how to use external reference to parameters](external-parameters.md) to add an extra security layer to your setup.
 
+   <i info>:material-information: Info:</i> The key provider configuration is stored in the database catalog in an unencrypted table. See [how to use external reference to parameters](external-parameters.md) to add an extra security layer to your setup.
 
-## WAL encryption configuration (tech preview)
 
-After you [enabled `pg_tde`](#enable-extension) and started the Percona Server for PostgreSQL, a principal key and internal keys for WAL encryption are created. They are stored in the data directory so that after WAL encryption is enabled, any process that requires access to WAL (a recovery or a checkpointer) can use them for decryption.
+## WAL encryption configuration
 
-Now you need to instruct `pg_tde ` to encrypt WAL files by configuring WAL encryption. Here's how to do it:
+After you [enabled `pg_tde`](#enable-extension) and started the Percona Server for PostgreSQL, a principal key and a keyring for WAL are created. Now you need to instruct `pg_tde ` to encrypt WAL files by configuring WAL encryption. 
 
-### Enable WAL level encryption
+Here's how to do it:
 
-1.  Use the `ALTER SYSTEM SET` command. You need the privileges of the superuser to run this command:
+1. Enable WAL level encryption using the `ALTER SYSTEM SET` command. You need the privileges of the superuser to run this command:
 
-    ```
+    ```sql
     ALTER SYSTEM set pg_tde.wal_encrypt = on;
     ```
 
@@ -110,29 +140,23 @@ Now you need to instruct `pg_tde ` to encrypt WAL files by configuring WAL encry
 
     * On Debian and Ubuntu:    
 
-       ```{.bash data-prompt="$"}
-       $ sudo systemctl restart postgresql.service
+       ```sh
+       sudo systemctl restart postgresql.service
        ```
     
     * On RHEL and derivatives
 
-       ```{.bash data-prompt="$"}
-       $ sudo systemctl restart postgresql-17
+       ```sh
+       sudo systemctl restart postgresql-17
        ```
 
-On the server start 
+3. We highly recommend you to create your own keyring and rotate the principal key. This is because the default principal key is created from the local keyfile and is stored unencrypted. 
 
-### Rotate the principal key
-
-We highly recommend you to create your own keyring and rotate the principal key. This is because the default principal key is created from the local keyfile and is stored unencrypted. 
-
-Rotating the principal key means re-encrypting internal keys used for WAL encryption with the new principal key. This process doesn't stop the database operation meaning that reads and writes can take place as usual during key rotation. 
-
-1. Set up the key provider for WAL encryption
+    Set up the key provider for WAL encryption
 
     === "With HashiCorp Vault"
     
-        ```
+        ```sql
         SELECT pg_tde_add_key_provider_vault_v2('PG_TDE_GLOBAL','provider-name',:'secret_token','url','mount','ca_path');
         ``` 
 
@@ -150,32 +174,17 @@ Rotating the principal key means re-encrypting internal keys used for WAL encryp
 
         This setup is intended for development and stores the keys unencrypted in the specified data file.    
 
-        ```
+        ```sql
         SELECT pg_tde_add_key_provider_file('provider-name','/path/to/the/keyring/data.file');
         ```
 
-2. Rotate the principal key. Don't forget to specify the `PG_TDE_GLOBAL` constant to rotate only the principal key for WAL.
+4. Rotate the principal key. Don't forget to specify the `PG_TDE_GLOBAL` constant to rotate only the principal key for WAL.
 
-    ```
+    ```sql
     SELECT pg_tde_rotate_principal_key('PG_TDE_GLOBAL', 'new-principal-key', 'provider-name');
     ```
 
-    Now all WAL files are encrypted for both encrypted and unencrypted tables. 
-   
-3. Verify the encryption by checking the `pg_tde.wal_encrypt` GUC (Grand Unified Configuration) parameter as follows: 
-
-    ```
-    SELECT name, setting FROM pg_settings WHERE name = 'pg_tde.wal_encrypt';
-    ```
-
-    ??? example "Sample output"
-
-        ```{.text .no-copy}
-
-                name        | setting
-        --------------------+---------
-         pg_tde.wal_encrypt | on
-        ```
+Now all WAL files are encrypted for both encrypted and unencrypted tables.
 
 ## Next steps
 

documentation/docs/tde.md

@@ -6,10 +6,13 @@ Transparent Data Encryption is a technology to protect data at rest. The encrypt
 
 To encrypt the data, two types of keys are used:
 
-* Table encryption keys (TEK) to encrypt user data. These keys are stored internally, near the data that they encrypt.
-* The principal key to encrypt table keys. It is kept separately from the table keys and is managed externally. 
+* Internal encryption keys to encrypt user data. They are stored internally, near the data that they encrypt.
+* The principal key to encrypt database keys. It is kept separately from the database keys and is managed externally in the key management store. 
 
-`pg_tde` is integrated with HashiCorp Vault server to store and manage principal keys. Only the back end KV Secrets Engine - Version 2 (API) is supported.
+You have the following options to store and manage principal keys externally:
+
+* Use the HashiCorp Vault server. Only the back end KV Secrets Engine - Version 2 (API) is supported.
+* Use the KMIP-compatible server. `pg_tde` has been tested with the [PyKMIP](https://pykmip.readthedocs.io/en/latest/server.html) server and [the HashiCorp Vault Enterprise KMIP Secrets Engine](https://www.vaultproject.io/docs/secrets/kmip).
 
 The encryption process is the following:
 
@@ -17,7 +20,7 @@ The encryption process is the following:
 
 When a user creates an encrypted table using `pg_tde`, a new random key is generated for that table using the AES128 (AES-ECB) cipher algorithm. This key is used to encrypt all data the user inserts in that table. Eventually the encrypted data gets stored in the underlying storage. 
 
-The table itself is encrypted using the principal key. The principal key is stored externally in the Vault key management store. 
+The table itself is encrypted using the principal key. The principal key is stored externally in the key management store. 
 
 Similarly when the user queries the encrypted table, the principal key is retrieved from the key store to decrypt the table. Then the same unique internal key for that table is used to decrypt the data, and unencrypted data gets returned to the user. So, effectively, every TDE table has a unique key, and each table key is encrypted using the principal key.
 

documentation/mkdocs.yml

@@ -164,7 +164,6 @@ nav:
   - functions.md
   - Concepts:
     - "What is TDE": tde.md
-  #  - wal-encryption.md
     - table-access-method.md
   - How to:
     - Use reference to external parameters: external-parameters.md