open-dsi
/
postgres
mirror of https://github.com/postgres/postgres
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
194 Commits
38 Branches
662 Tags
732 MiB
Tag: Branch: Tree: 8a2b8ec3e5
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '8a2b8ec3e5'
${ noResults }
postgres/sql/toast_extended_storage.sql

39 lines
37 KiB
Raw Normal View History Unescape

Repository rename: using the new name everywhere (#114)
2 years ago
-- test https://github.com/Percona-Lab/pg_tde/issues/63
Fix compressed TOASTs encryption/decryption (#76) If TOAST data gets compressed, it has an extended header containing compression info. We used to encrypt this header along with the actual data which in turn caused a crash as PG needs this data in later stages. So it should be taken into account while encrypting data during externalisation. Then, during detoasting, we should not decrypt this compression header as it is being extracted with the data with the first TOAST chunk. So, copy the first N bytes (now it is 4 bytes) of the first chunk as it is and decrypt the rest of the data. Fixes https://github.com/Percona-Lab/postgres-tde-ext/issues/63
2 years ago
CREATE EXTENSION pg_tde;
Framework for multi-tenancy support (#121) * Introducing catalog table for managing key providers This commit introduces a user catalog table, percona_tde.pg_tde_key_provider, within the percona_tde schema, as part of the pg_tde extension. The purpose of this table is to store essential provider information. The catalog accommodates various key providers, present and future, utilizing a JSON type options field to capture provider-specific details. To facilitate the creation of key providers, the commit introduces new SQL interfaces: - pg_tde_add_key_provider(provider_type VARCHAR(10), provider_name VARCHAR(128), options JSON) - pg_tde_add_key_provider_file(provider_name VARCHAR(128), file_path TEXT) - pg_tde_add_key_provider_vault_v2(provider_name VARCHAR(128), vault_token TEXT, vault_url TEXT, vault_mount_path TEXT, vault_ca_path TEXT) Additionally, the commit implements the C interface for catalog interaction, detailed in the 'tde_keyring.h' file. These changes lay the foundation for implementing multi-tenancy in pg_tde by eliminating the necessity of a 'keyring.json' file for configuring a cluster-wide key provider. With this enhancement, each database can have its dedicated key provider, added via SQL interface, removing the need for DBA intervention in TDE setup." * Establishing a Framework for Master Key and Shared Cache Management Up until now, pg_tde relied on a hard-coded master key name, primarily for proof-of-concept purposes. This commit introduces a more robust infrastructure for configuring the master key and managing a dynamic shared memory-based master-key cache to enhance accessibility. For user interaction, a new SQL interface is provided: - pg_tde_set_master_key(master_key_name VARCHAR(255), provider_name VARCHAR(255)); This interface enables users to set a master key for a specific database and make further enhancements toward implementing the multi-tenancy. In addition to the public SQL interface, the commit optimizes the internal master-key API. It introduces straightforward Get and Set functions, handling locking, retrieval, caching, and seamlessly assigning a master key for a database. The commit also introduces a unified internal interface for requesting and utilizing shared memory, contributing to a more cohesive and efficient master key and cache management system. * Revamping the Keyring API Interface and Integrating Master Key This commit unifies the master-key and key-provider modules with the core of pg_tde, marking a significant evolution in the architecture. As part of this integration, the keyring API undergoes substantial changes to enhance flexibility and remove unnecessary components such as the key cache. As a result of the keyring refactoring, the file keyring is also rewritten, offering a template for implementing additional key providers for the extension. The modifications make the keyring API more pluggable, streamlining interactions and paving the way for future enhancements. * An Interface for Informing the Shared Memory Manager about Lock Requirements This commit addresses PostgreSQL core's requirement for upfront information regarding the number of locks the extension needs. Given the connection between locks and the shared memory interface, a new callback routine is introduced. This routine allows modules to specify the number of locks they require. In addition to this functionality, the commit includes code cleanups and adjustments to nomenclature for improved clarity and consistency. * Adjusting test cases * Extension Initialization and Cleanup Mechanism This commit enhances the extension by adding a new mechanism to facilitate cleanup or setup procedures when the extension is installed in a database. The core addition is a function "pg_tde_extension_initialize" invoked upon executing the database's 'CREATE EXTENSION' command. The commit introduces a callback registration mechanism to streamline future development and ensure extensibility. This enables any module to specify a callback function (registered using on_ext_install() ) to be invoked during extension creation. As of this commit, the callback functionality is explicitly utilized by the master key module to handle the cleanup of the master key information file. This file might persist in the database directory if the extension had been previously deleted in the same database. This enhancement paves the way for a more modular and maintainable extension architecture, allowing individual modules to manage their specific setup and cleanup tasks seamlessly." * Adjusting Vault-V2 key provider to use new keyring architecture
2 years ago
SELECT pg_tde_add_key_provider_file('file-vault','/tmp/pg_tde_test_keyring.per');
Renamed master key to principal key (#228) This commit contains lots of changes, but it's just a repeated execution of find <...> -exec sed <...>, so everything should work as before.
1 year ago
SELECT pg_tde_set_principal_key('test-db-principal-key','file-vault');
Framework for multi-tenancy support (#121) * Introducing catalog table for managing key providers This commit introduces a user catalog table, percona_tde.pg_tde_key_provider, within the percona_tde schema, as part of the pg_tde extension. The purpose of this table is to store essential provider information. The catalog accommodates various key providers, present and future, utilizing a JSON type options field to capture provider-specific details. To facilitate the creation of key providers, the commit introduces new SQL interfaces: - pg_tde_add_key_provider(provider_type VARCHAR(10), provider_name VARCHAR(128), options JSON) - pg_tde_add_key_provider_file(provider_name VARCHAR(128), file_path TEXT) - pg_tde_add_key_provider_vault_v2(provider_name VARCHAR(128), vault_token TEXT, vault_url TEXT, vault_mount_path TEXT, vault_ca_path TEXT) Additionally, the commit implements the C interface for catalog interaction, detailed in the 'tde_keyring.h' file. These changes lay the foundation for implementing multi-tenancy in pg_tde by eliminating the necessity of a 'keyring.json' file for configuring a cluster-wide key provider. With this enhancement, each database can have its dedicated key provider, added via SQL interface, removing the need for DBA intervention in TDE setup." * Establishing a Framework for Master Key and Shared Cache Management Up until now, pg_tde relied on a hard-coded master key name, primarily for proof-of-concept purposes. This commit introduces a more robust infrastructure for configuring the master key and managing a dynamic shared memory-based master-key cache to enhance accessibility. For user interaction, a new SQL interface is provided: - pg_tde_set_master_key(master_key_name VARCHAR(255), provider_name VARCHAR(255)); This interface enables users to set a master key for a specific database and make further enhancements toward implementing the multi-tenancy. In addition to the public SQL interface, the commit optimizes the internal master-key API. It introduces straightforward Get and Set functions, handling locking, retrieval, caching, and seamlessly assigning a master key for a database. The commit also introduces a unified internal interface for requesting and utilizing shared memory, contributing to a more cohesive and efficient master key and cache management system. * Revamping the Keyring API Interface and Integrating Master Key This commit unifies the master-key and key-provider modules with the core of pg_tde, marking a significant evolution in the architecture. As part of this integration, the keyring API undergoes substantial changes to enhance flexibility and remove unnecessary components such as the key cache. As a result of the keyring refactoring, the file keyring is also rewritten, offering a template for implementing additional key providers for the extension. The modifications make the keyring API more pluggable, streamlining interactions and paving the way for future enhancements. * An Interface for Informing the Shared Memory Manager about Lock Requirements This commit addresses PostgreSQL core's requirement for upfront information regarding the number of locks the extension needs. Given the connection between locks and the shared memory interface, a new callback routine is introduced. This routine allows modules to specify the number of locks they require. In addition to this functionality, the commit includes code cleanups and adjustments to nomenclature for improved clarity and consistency. * Adjusting test cases * Extension Initialization and Cleanup Mechanism This commit enhances the extension by adding a new mechanism to facilitate cleanup or setup procedures when the extension is installed in a database. The core addition is a function "pg_tde_extension_initialize" invoked upon executing the database's 'CREATE EXTENSION' command. The commit introduces a callback registration mechanism to streamline future development and ensure extensibility. This enables any module to specify a callback function (registered using on_ext_install() ) to be invoked during extension creation. As of this commit, the callback functionality is explicitly utilized by the master key module to handle the cleanup of the master key information file. This file might persist in the database directory if the extension had been previously deleted in the same database. This enhancement paves the way for a more modular and maintainable extension architecture, allowing individual modules to manage their specific setup and cleanup tasks seamlessly." * Adjusting Vault-V2 key provider to use new keyring architecture
2 years ago
Access method renaming and conditions (#231) * Renamed pg_tde -> pg_tde_basic * The "fork" extension is only visible when it's actally built * Update sysbench tests to use pg_tde_basic
1 year ago
CREATE TEMP TABLE src (f1 text) USING pg_tde_basic;
Fix compressed TOASTs encryption/decryption (#76) If TOAST data gets compressed, it has an extended header containing compression info. We used to encrypt this header along with the actual data which in turn caused a crash as PG needs this data in later stages. So it should be taken into account while encrypting data during externalisation. Then, during detoasting, we should not decrypt this compression header as it is being extracted with the data with the first TOAST chunk. So, copy the first N bytes (now it is 4 bytes) of the first chunk as it is and decrypt the rest of the data. Fixes https://github.com/Percona-Lab/postgres-tde-ext/issues/63
2 years ago
-- Crash on INSERT
INSERT INTO src
VALUES('0.55859909742449630.44658969494913570.54075930161272720.173117157913014630.61483029376206380.65764492874377220.341317552838924730.367982528684053230.175345977931963270.168412839608874880.00154803678245296620.82706532396263290.74748634462447190.090831815264683650.390919315685386960.99951082699941550.9977981693287330.6988579613645320.310754450662202640.90325242484683190.75374800591547490.26680100643896230.55751915566773990.57902456214791110.36183153154123460.63524053266029830.78389378855711180.19584445869629020.199333924650425760.82155191593829560.7371944732869880.183910466357891660.0147813222233452720.3747022411129810.49101561236565550.95483453706535880.35594888092451550.43381965349401440.46361549602747920.50604155870332760.86586716524835540.63478889357891990.77509493569207090.86665305443338790.64852060828658550.50280760242256580.21800585609741340.096173392125813220.0261400320036884180.33800342276157360.485498510272187160.69492885593869610.14719438590370170.57633710730539910.6854376608363930.162803430883830650.28902094699378880.93884121928877070.4124819510126210.69895400258256470.61386295568035320.019902272612943640.85235316437206570.0940431968488260.272794218757168140.61549039934229780.422575394607501930.67002314675933960.465323258961145570.163191821055387760.0126060416991824460.40893698240906830.31893797439819460.15469713662310670.55528689194077320.66788769570588440.71025660771475390.38117379415620990.0220335908759561330.107673951160519810.71950609969184590.54341042925206180.024053937929693570.74203099973156790.035064651259838710.86887319172737380.335093303911782050.7483180995321150.97612135845236070.084654394261215680.76508793255901520.68191364158327270.64505339286832350.448618338317764650.335092422718133550.55149498651635520.9413160253094210.9168195414285170.98684856309613790.60400653549636750.85646368669913330.58837858335250370.9799739681795840.48079146876587030.218616079813109820.9302335200895790.4780449500011730.424050492872935840.063479437634682330.98094393207488960.335273138834828230.48560551700566790.139203310225991970.62595627061874380.71122415168232940.152848330691444540.89199132936279750.27800941859127290.95439564372772280.84837555269067490.75100083734460510.362767538265398760.235976384421268110.61187422491548470.9495830853409060.89514971758309940.26872924068443680.74626444803809160.139587450203541460.302395254482703770.78411327172089250.38966191620694680.198917136896949340.64377926785777470.293260719678428260.44648764937475450.37420519795286180.92220518748025750.0073108929093146370.93143459249930790.61565949670551650.95409849589104280.70559701135921380.80911223952124960.78688763724234260.0143429787186462040.47314838377612680.220085013432371750.74895079799389160.34241785580036590.377476662711184960.55856596798903090.72300200663394070.93372512510565420.48213601319131170.98122442024471220.346628953420145660.74820202768550950.36134870838736320.53096217018068880.91813333111021930.16912775074741670.37503790891484610.9532471869686030.125924225709406650.481359293779658250.49808987733380960.292530386328931250.87891128070888010.190872215535672130.8880398891158570.312849610687170760.83382258936561130.88683286703304340.54819728672968980.55198306885689990.94518489093830830.82370179536934040.39422249429194810.88920643110698830.100781813305392380.156896688655811630.176728786940858470.418953555096873260.49179322828441930.6482244643731250.50630017133792920.96824089562929780.48649962422895390.224128640753047840.65318604085187480.0277597024087572470.269592268429819760.078229807252904630.12959218454427490.76024146340840760.53005245019718040.208874546770384530.55257426353213910.5937585938899110.80002298982932360.176800500254526760.80793461098073150.73215202402765760.89330730727462560.0317516822834684740.160090174689148550.51532774354845980.70921991745912830.138735433408188950.57752467002560.403150487295366840.40749394747573110.66251587358165880.35063881167575020.9599596099677250.445932818660210060.287245889223862740.8257369856611840.70400052356170930.30353378134511090.393039351875958730.362370380599912560.38847117294574
SELECT * FROM src;
DROP TABLE src;
Access method renaming and conditions (#231) * Renamed pg_tde -> pg_tde_basic * The "fork" extension is only visible when it's actally built * Update sysbench tests to use pg_tde_basic
1 year ago
CREATE TABLE src2 (f1 TEXT) USING pg_tde_basic;
Fix compressed TOASTs encryption/decryption (#76) If TOAST data gets compressed, it has an extended header containing compression info. We used to encrypt this header along with the actual data which in turn caused a crash as PG needs this data in later stages. So it should be taken into account while encrypting data during externalisation. Then, during detoasting, we should not decrypt this compression header as it is being extracted with the data with the first TOAST chunk. So, copy the first N bytes (now it is 4 bytes) of the first chunk as it is and decrypt the rest of the data. Fixes https://github.com/Percona-Lab/postgres-tde-ext/issues/63
2 years ago
INSERT INTO src2
VALUES('0.55859909742449630.44658969494913570.54075930161272720.173117157913014630.61483029376206380.65764492874377220.341317552838924730.367982528684053230.175345977931963270.168412839608874880.00154803678245296620.82706532396263290.74748634462447190.090831815264683650.390919315685386960.99951082699941550.9977981693287330.6988579613645320.310754450662202640.90325242484683190.75374800591547490.26680100643896230.55751915566773990.57902456214791110.36183153154123460.63524053266029830.78389378855711180.19584445869629020.199333924650425760.82155191593829560.7371944732869880.183910466357891660.0147813222233452720.3747022411129810.49101561236565550.95483453706535880.35594888092451550.43381965349401440.46361549602747920.50604155870332760.86586716524835540.63478889357891990.77509493569207090.86665305443338790.64852060828658550.50280760242256580.21800585609741340.096173392125813220.0261400320036884180.33800342276157360.485498510272187160.69492885593869610.14719438590370170.57633710730539910.6854376608363930.162803430883830650.28902094699378880.93884121928877070.4124819510126210.69895400258256470.61386295568035320.019902272612943640.85235316437206570.0940431968488260.272794218757168140.61549039934229780.422575394607501930.67002314675933960.465323258961145570.163191821055387760.0126060416991824460.40893698240906830.31893797439819460.15469713662310670.55528689194077320.66788769570588440.71025660771475390.38117379415620990.0220335908759561330.107673951160519810.71950609969184590.54341042925206180.024053937929693570.74203099973156790.035064651259838710.86887319172737380.335093303911782050.7483180995321150.97612135845236070.084654394261215680.76508793255901520.68191364158327270.64505339286832350.448618338317764650.335092422718133550.55149498651635520.9413160253094210.9168195414285170.98684856309613790.60400653549636750.85646368669913330.58837858335250370.9799739681795840.48079146876587030.218616079813109820.9302335200895790.4780449500011730.424050492872935840.063479437634682330.98094393207488960.335273138834828230.48560551700566790.139203310225991970.62595627061874380.71122415168232940.152848330691444540.89199132936279750.27800941859127290.95439564372772280.84837555269067490.75100083734460510.362767538265398760.235976384421268110.61187422491548470.9495830853409060.89514971758309940.26872924068443680.74626444803809160.139587450203541460.302395254482703770.78411327172089250.38966191620694680.198917136896949340.64377926785777470.293260719678428260.44648764937475450.37420519795286180.92220518748025750.0073108929093146370.93143459249930790.61565949670551650.95409849589104280.70559701135921380.80911223952124960.78688763724234260.0143429787186462040.47314838377612680.220085013432371750.74895079799389160.34241785580036590.377476662711184960.55856596798903090.72300200663394070.93372512510565420.48213601319131170.98122442024471220.346628953420145660.74820202768550950.36134870838736320.53096217018068880.91813333111021930.16912775074741670.37503790891484610.9532471869686030.125924225709406650.481359293779658250.49808987733380960.292530386328931250.87891128070888010.190872215535672130.8880398891158570.312849610687170760.83382258936561130.88683286703304340.54819728672968980.55198306885689990.94518489093830830.82370179536934040.39422249429194810.88920643110698830.100781813305392380.156896688655811630.176728786940858470.418953555096873260.49179322828441930.6482244643731250.50630017133792920.96824089562929780.48649962422895390.224128640753047840.65318604085187480.0277597024087572470.269592268429819760.078229807252904630.12959218454427490.76024146340840760.53005245019718040.208874546770384530.55257426353213910.5937585938899110.80002298982932360.176800500254526760.80793461098073150.73215202402765760.89330730727462560.0317516822834684740.160090174689148550.51532774354845980.70921991745912830.138735433408188950.57752467002560.403150487295366840.40749394747573110.66251587358165880.35063881167575020.9599596099677250.445932818660210060.287245889223862740.8257369856611840.70400052356170930.30353378134511090.393039351875958730.362370380599912560.38847117294574
SELECT * FROM src2;
DROP TABLE src2;
Test case update for Issue:82 (#142)
2 years ago
-- https://github.com/Percona-Lab/pg_tde/issues/82
Access method renaming and conditions (#231) * Renamed pg_tde -> pg_tde_basic * The "fork" extension is only visible when it's actally built * Update sysbench tests to use pg_tde_basic
1 year ago
CREATE TABLE indtoasttest(descr text, cnt int DEFAULT 0, f1 text, f2 text) using pg_tde_basic;
Test case update for Issue:82 (#142)
2 years ago
INSERT INTO indtoasttest(descr, f1, f2) VALUES('two-compressed', repeat('1234567890',1000), repeat('1234567890',1000));
INSERT INTO indtoasttest(descr, f1, f2) VALUES('two-toasted', repeat('1234567890',30000), repeat('1234567890',50000));
INSERT INTO indtoasttest(descr, f1, f2) VALUES('one-compressed,one-null', NULL, repeat('1234567890',1000));
INSERT INTO indtoasttest(descr, f1, f2) VALUES('one-toasted,one-null', NULL, repeat('1234567890',50000));
UPDATE indtoasttest SET cnt = cnt +1 RETURNING substring(indtoasttest::text, 1, 200);
UPDATE indtoasttest SET cnt = cnt +1, f1 = f1 RETURNING substring(indtoasttest::text, 1, 200);
UPDATE indtoasttest SET cnt = cnt +1, f1 = f1||'' RETURNING substring(indtoasttest::text, 1, 200);
UPDATE indtoasttest SET cnt = cnt +1, f1 = f1||'' RETURNING substring(indtoasttest::text, 1, 200);
UPDATE indtoasttest SET f2 = '+'||f2||'-' ;
DROP TABLE indtoasttest;
Fix compressed TOASTs encryption/decryption (#76) If TOAST data gets compressed, it has an extended header containing compression info. We used to encrypt this header along with the actual data which in turn caused a crash as PG needs this data in later stages. So it should be taken into account while encrypting data during externalisation. Then, during detoasting, we should not decrypt this compression header as it is being extracted with the data with the first TOAST chunk. So, copy the first N bytes (now it is 4 bytes) of the first chunk as it is and decrypt the rest of the data. Fixes https://github.com/Percona-Lab/postgres-tde-ext/issues/63
2 years ago
DROP EXTENSION pg_tde;