You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Amit Kapila 0434033e8b Fix data loss in logical replication. ... Data loss can happen when the DDLs like ALTER PUBLICATION ... ADD TABLE ... or ALTER TYPE ... that don't take a strong lock on table happens concurrently to DMLs on the tables involved in the DDL. This happens because logical decoding doesn't distribute invalidations to concurrent transactions and those transactions use stale cache data to decode the changes. The problem becomes bigger because we keep using the stale cache even after those in-progress transactions are finished and skip the changes required to be sent to the client. This commit fixes the issue by distributing invalidation messages from catalog-modifying transactions to all concurrent in-progress transactions. This allows the necessary rebuild of the catalog cache when decoding new changes after concurrent DDL. We observed performance regression primarily during frequent execution of *publication DDL* statements that modify the published tables. The regression is minor or nearly nonexistent for DDLs that do not affect the published tables or occur infrequently, making this a worthwhile cost to resolve a longstanding data loss issue. An alternative approach considered was to take a strong lock on each affected table during publication modification. However, this would only address issues related to publication DDLs (but not the ALTER TYPE ...) and require locking every relation in the database for publications created as FOR ALL TABLES, which is impractical. The bug exists in all supported branches, but we are backpatching till 14. The fix for 13 requires somewhat bigger changes than this fix, so the fix for that branch is still under discussion. Reported-by: hubert depesz lubaczewski <depesz@depesz.com> Reported-by: Tomas Vondra <tomas.vondra@enterprisedb.com> Author: Shlok Kyal <shlok.kyal.oss@gmail.com> Author: Hayato Kuroda <kuroda.hayato@fujitsu.com> Reviewed-by: Zhijie Hou <houzj.fnst@fujitsu.com> Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com> Reviewed-by: Amit Kapila <amit.kapila16@gmail.com> Tested-by: Benoit Lobréau <benoit.lobreau@dalibo.com> Backpatch-through: 14 Discussion: https://postgr.es/m/de52b282-1166-1180-45a2-8d8917ca74c6@enterprisedb.com Discussion: https://postgr.es/m/CAD21AoAenVqiMjpN-PvGHL1N9DWnHSq673bfgr6phmBUzx=kLQ@mail.gmail.com		10 months ago
..
binary.out	Message style improvements	10 years ago
catalog_change_snapshot.out	Fix catalog lookup due to wrong snapshot for subtransactions during decoding.	2 years ago
concurrent_ddl_dml.out	Improve display of query results in isolation tests.	5 years ago
concurrent_stream.out	Improve display of query results in isolation tests.	5 years ago
ddl.out	Make test_decoding ddl.out shorter	3 years ago
decoding_in_xact.out	Introduce xid8-based functions to replace txid_XXX.	6 years ago
decoding_into_rel.out	Fix crash when logical decoding is invoked from a PL function.	8 years ago
delayed_startup.out	Improve display of query results in isolation tests.	5 years ago
invalidation_distrubution.out	Fix data loss in logical replication.	10 months ago
messages.out	Add required database and origin filtering for logical messages.	10 years ago
mxact.out	Improve display of query results in isolation tests.	5 years ago
oldest_xmin.out	Improve display of query results in isolation tests.	5 years ago
ondisk_startup.out	Improve display of query results in isolation tests.	5 years ago
permissions.out	Replace AclObjectKind with ObjectType	8 years ago
prepared.out	Add skip-empty-xacts option to test_decoding for use in the regression tests.	12 years ago
replorigin.out	Remove assertion for replication origins in PREPARE TRANSACTION	4 years ago
rewrite.out	Fix regression tests to use only global names beginning with "regress_".	7 years ago
skip_snapshot_restore.out	Fix possibility of logical decoding partial transaction changes.	2 years ago
slot.out	Message improvement	5 years ago
snapshot_transfer.out	Improve display of query results in isolation tests.	5 years ago
spill.out	Make contrib/test_decoding regression tests safe for CZ locale.	9 years ago
stats.out	Fix tests for replication slots stats.	5 years ago
stream.out	Stabilize the test added by commit 022564f60c.	1 year ago
subxact_without_top.out	Improve display of query results in isolation tests.	5 years ago
time.out	logical decoding: fix decoding of a commit's commit time.	10 years ago
toast.out	WAL log unchanged toasted replica identity key attributes.	4 years ago
truncate.out	Implement streaming mode in ReorderBuffer.	6 years ago
twophase.out	Stabilize the test added by commit 022564f60c.	1 year ago
twophase_snapshot.out	Improve display of query results in isolation tests.	5 years ago
twophase_stream.out	Add option to enable two_phase commits via pg_create_logical_replication_slot.	5 years ago
xact.out	Don't propagate a null subtransaction snapshot up to parent transaction.	10 years ago