27 Commits (ffd398021c296e55b59ab848ac71a1769ddff93f)

Author	SHA1	Message	Date
Michael Paquier	9555cc8d2b	Revert hooks for session start and end, take two ... The location of the session end hook has been chosen so as it is possible to allow modules to do their own transactions, however any trying to any any subsystem which went through before_shmem_exit() would cause issues, limiting the pluggability of the hook. Per discussion with Tom Lane and Andres Freund. Discussion: https://postgr.es/m/18722.1569906636@sss.pgh.pa.us	6 years ago
Michael Paquier	e788bd924c	Add hooks for session start and session end, take two ... These hooks can be used in loadable modules. A simple test module is included. The first attempt was done with cd8ce3a but we lacked handling for NO_INSTALLCHECK in the MSVC scripts (problem solved afterwards by 431f1599) so the buildfarm got angry. This also fixes a couple of issues noticed upon review compared to the first attempt, so the code has slightly changed, resulting in a more simple test module. Author: Fabrízio de Royes Mello, Yugo Nagata Reviewed-by: Andrew Dunstan, Michael Paquier, Aleksandr Parfenov Discussion: https://postgr.es/m/20170720204733.40f2b7eb.nagata@sraoss.co.jp Discussion: https://postgr.es/m/20190823042602.GB5275@paquier.xyz	6 years ago
Michael Paquier	640c19869f	Add dummy_index_am to src/test/modules/ ... This includes more tests dedicated to relation options, bringing the coverage of this code close to 100%, and the module can be used for other purposes, like a base template for an index AM implementation. Author: Nikolay Sharplov, Michael Paquier Reviewed-by: Álvaro Herrera, Dent John Discussion: https://postgr.es/m/17071942.m9zZutALE6@x200m	6 years ago
Tom Lane	db43831899	Avoid using INFO elevel for what are fundamentally debug messages. ... Commit 6f6b99d13 stuck an INFO message into the fast path for checking partition constraints, for no very good reason except that it made it easy for the regression tests to verify that that path was taken. Assorted later patches did likewise, increasing the unsuppressable-chatter level from ALTER TABLE even more. This isn't good for the user experience, so let's drop these messages down to DEBUG1 where they belong. So as not to have a loss of test coverage, create a TAP test that runs the relevant queries with client_min_messages = DEBUG1 and greps for the expected messages. This testing method is a bit brute-force --- in particular, it duplicates the execution of a fair amount of the core create_table and alter_table tests. We experimented with other solutions, but running any significant amount of standard testing with client_min_messages = DEBUG1 seems to have a lot of output-stability pitfalls, cf commits bbb96c370 and 5655565c0. Possibly at some point we'll look into whether we can reduce the amount of test duplication. Backpatch into v12, because some of these messages are new in v12 and we don't really want to ship it that way. Sergei Kornilov Discussion: https://postgr.es/m/81911511895540@web58j.yandex.ru Discussion: https://postgr.es/m/4859321552643736@myt5-02b80404fd9e.qloud-c.yandex.net	6 years ago
Heikki Linnakangas	bde7493d10	Fix overflow check and comment in GIN posting list encoding. ... The comment did not match what the code actually did for integers with the 43rd bit set. You get an integer like that, if you have a posting list with two adjacent TIDs that are more than 2^31 blocks apart. According to the comment, we would store that in 6 bytes, with no continuation bit on the 6th byte, but in reality, the code encodes it using 7 bytes, with a continuation bit on the 6th byte as normal. The decoding routine also handled these 7-byte integers correctly, except for an overflow check that assumed that one integer needs at most 6 bytes. Fix the overflow check, and fix the comment to match what the code actually does. Also fix the comment that claimed that there are 17 unused bits in the 64-bit representation of an item pointer. In reality, there are 64-32-11=21. Fitting any item pointer into max 6 bytes was an important property when this was written, because in the old pre-9.4 format, item pointers were stored as plain arrays, with 6 bytes for every item pointer. The maximum of 6 bytes per integer in the new format guaranteed that we could convert any page from the old format to the new format after upgrade, so that the new format was never larger than the old format. But we hardly need to worry about that anymore, and running into that problem during upgrade, where an item pointer is expanded from 6 to 7 bytes such that the data doesn't fit on a page anymore, is implausible in practice anyway. Backpatch to all supported versions. This also includes a little test module to test these large distances between item pointers, without requiring a 16 TB table. It is not backpatched, I'm including it more for the benefit of future development of new posting list formats. Discussion: https://www.postgresql.org/message-id/33bfc20a-5c86-f50c-f5a5-58e9925d05ff%40iki.fi Reviewed-by: Masahiko Sawada, Alexander Korotkov	6 years ago
Tom Lane	c91504b958	Move rolenames test out of the core regression tests. ... This test script is unsafe to run in "make installcheck" mode for (at least) two reasons: it creates and destroys some role names that don't follow the "regress_xxx" naming convention, and it sets and then resets the application_name GUC attached to every existing role. While we've not had complaints, these surely are not good things to do within a production installation, and regress.sgml pretty clearly implies that we won't do them. Rather than lose test coverage altogether, let's just move this script somewhere where it will get run by "make check" but not "make installcheck". src/test/modules/ already has that property. Since it seems likely that we'll want other regression tests in future that also exceed the constraints of "make installcheck", create a generically-named src/test/modules/unsafe_tests/ directory to hold them. Discussion: https://postgr.es/m/16638.1468620817@sss.pgh.pa.us	6 years ago
Heikki Linnakangas	df816f6ad5	Add IntegerSet, to hold large sets of 64-bit ints efficiently. ... The set is implemented as a B-tree, with a compact representation at leaf items, using Simple-8b algorithm, so that clusters of nearby values use less memory. The IntegerSet isn't used for anything yet, aside from the test code, but we have two patches in the works that would benefit from this: A patch to allow GiST vacuum to delete empty pages, and a patch to reduce heap VACUUM's memory usage, by storing the list of dead TIDs more efficiently and lifting the 1 GB limit on its size. This includes a unit test module, in src/test/modules/test_integerset. It can be used to verify correctness, as a regression test, but if you run it manully, it can also print memory usage and execution time of some of the tests. Author: Heikki Linnakangas, Andrey Borodin Reviewed-by: Julien Rouhaud Discussion: https://www.postgresql.org/message-id/b5e82599-1966-5783-733c-1a947ddb729f@iki.fi	7 years ago
Tom Lane	3b8f6e75f3	Fix partial-build problems introduced by having more generated headers. ... Commit 372728b0d created some problems for usages like building a subdirectory without having first done "make all" at the top level, or for proceeding directly to "make install" without "make all". The only reasonably clean way to fix this seems to be to force the submake-generated-headers rule to fire in *any* "make all" or "make install" command anywhere in the tree. To avoid lots of redundant work, as well as parallel make jobs possibly clobbering each others' output, we still need to be sure that the rule fires only once in a recursive build. For that, adopt the same MAKELEVEL hack previously used for "temp-install". But try to document it a bit better. The submake-errcodes mechanism previously used in src/port/ and src/common/ is subsumed by this, so we can get rid of those special cases. It was inadequate for src/common/ anyway after the aforesaid commit, and it always risked parallel attempts to build errcodes.h. Discussion: https://postgr.es/m/E1f5FAB-0006LU-MB@gemulon.postgresql.org	8 years ago
Andres Freund	51bc271790	Add Bloom filter implementation. ... A Bloom filter is a space-efficient, probabilistic data structure that can be used to test set membership. Callers will sometimes incur false positives, but never false negatives. The rate of false positives is a function of the total number of elements and the amount of memory available for the Bloom filter. Two classic applications of Bloom filters are cache filtering, and data synchronization testing. Any user of Bloom filters must accept the possibility of false positives as a cost worth paying for the benefit in space efficiency. This commit adds a test harness extension module, test_bloomfilter. It can be used to get a sense of how the Bloom filter implementation performs under varying conditions. This is infrastructure for the upcoming "heapallindexed" amcheck patch, which verifies the consistency of a heap relation against one of its indexes. Author: Peter Geoghegan Reviewed-By: Andrey Borodin, Michael Paquier, Thomas Munro, Andres Freund Discussion: https://postgr.es/m/CAH2-Wzm5VmG7cu1N-H=nnS57wZThoSDQU+F5dewx3o84M+jY=g@mail.gmail.com	8 years ago
Tom Lane	44468f49bb	Add test scaffolding for exercising optimizer's predicate-proof logic. ... The predicate-proof code in predtest.c is a bit hard to test as-is: you have to construct a query whose plan changes depending on the success of a test, and in tests that have been around for awhile, it's always possible that the plan shape is now being determined by some other factor. Our existing regression tests aren't doing real well at providing full code coverage of predtest.c, either. So, let's add a small test module that allows directly inspecting the results of predicate_implied_by() and predicate_refuted_by() for arbitrary boolean expressions. I chose the set of tests committed here in order to get reasonably complete code coverage of predtest.c just from running this test module, and to cover some cases called out as being interesting in the existing comments. We might want to add more later. But this set already shows a few cases where perhaps things could be improved. Indeed, this exercise proves that predicate_refuted_by() is buggy for the case of clause_is_check = true, though fortunately we aren't using that case anywhere yet. I'll look into doing something about that in a separate commit. For now, just memorialize the current behavior. Discussion: https://postgr.es/m/5983.1520487191@sss.pgh.pa.us	8 years ago
Andrew Dunstan	98d54bb779	Back out the session_start and session_end hooks feature. ... It's become apparent during testing that there are problems with at least the testing regime. I don't think we should have it without a working test regime, and the difficulties might indicate implementation problems anyway, so I'm backing out the whole thing until that's sorted out. This reverts commits 7459484 9989f92 cd8ce3a	8 years ago
Andrew Dunstan	745948422c	Disable installcheck tests for test_session_hooks ... The module requires a preloaded library and the defect can't be cured by a LOAD instruction in the test script. To achieve this we override the installcheck target in the module's Makefile, and exclude ithe module in vcregress.pl. Along the way, revert commit 9989f92aab.	8 years ago
Andrew Dunstan	9989f92aab	Disable test_session_hooks test module until buildfarm issues are sorted out	8 years ago
Andrew Dunstan	cd8ce3a22c	Add hooks for session start and session end ... These hooks can be used in loadable modules. A simple test module is included. Discussion: https://postgr.es/m/20170720204733.40f2b7eb.nagata@sraoss.co.jp Fabrízio de Royes Mello and Yugo Nagata Reviewed by Michael Paquier and Aleksandr Parfenov	8 years ago
Tom Lane	610bbdd8ac	Add a test harness for the red-black tree code. ... This improves the regression tests' coverage of rbtree.c from pretty awful (because some of the functions aren't used yet) to basically 100%. Victor Drobny, reviewed by Aleksander Alekseev and myself Discussion: https://postgr.es/m/c9d61310e16e75f8acaf6cb1c48b7b77@postgrespro.ru	8 years ago
Tom Lane	548af97fce	Provide and use a makefile target to build all generated headers. ... As of 9.6, pg_regress doesn't build unless storage/lwlocknames.h has been created; but there was nothing forcing that to happen if you just went into src/test/regress/ and built there. We previously had a similar complaint about plpython. To fix in a way that won't break next time we invent a generated header, make src/backend/Makefile expose a phony target for updating all the include files it builds, and invoke that before building pg_regress or plpython. In principle, maybe we ought to invoke that everywhere; but it would add a lot of usually-useless make cycles, so let's just do it in the places where people have complained. I made a couple of cosmetic adjustments in src/backend/Makefile as well, to deal with the generated headers in consistent orders. Michael Paquier and Tom Lane Report: <31398.1467036827@sss.pgh.pa.us> Report: <20150916200959.GB32090@msg.df7cb.de>	9 years ago
Stephen Frost	6bd356c33a	Add TAP tests for pg_dump ... This TAP test suite will create a new cluster, populate it based on the 'create_sql' values in the '%tests' hash, run all of the runs defined in the '%pgdump_runs' hash, and then for each test in the '%tests' hash, compare each run's output the the regular expression defined for the test under the 'like' and 'unlike' functions, as appropriate. While this test suite covers a fair bit of ground (67% of pg_dump.c and quite a bit of the other files in src/bin/pg_dump), there is still quite a bit which remains to be added to provide better code coverage. Still, this is quite a bit better than we had, and has found a few bugs already (note that the CREATE TRANSFORM test is commented out, as it is currently failing). Idea for using the TAP system from Tom, though all of the code is mine.	9 years ago
Alvaro Herrera	c1543a81a7	Revert timeline following in replication slots ... This reverts commits f07d18b6e9, 82c83b3372, 3a3b309041, and 24c5f1a103. This feature has shown enough immaturity that it was deemed better to rip it out before rushing some more fixes at the last minute. There are discussions on larger changes in this area for the next release.	9 years ago
Kevin Grittner	848ef42bb8	Add the "snapshot too old" feature ... This feature is controlled by a new old_snapshot_threshold GUC. A value of -1 disables the feature, and that is the default. The value of 0 is just intended for testing. Above that it is the number of minutes a snapshot can reach before pruning and vacuum are allowed to remove dead tuples which the snapshot would otherwise protect. The xmin associated with a transaction ID does still protect dead tuples. A connection which is using an "old" snapshot does not get an error unless it accesses a page modified recently enough that it might not be able to produce accurate results. This is similar to the Oracle feature, and we use the same SQLSTATE and error message for compatibility.	10 years ago
Alvaro Herrera	24c5f1a103	Enable logical slots to follow timeline switches ... When decoding from a logical slot, it's necessary for xlog reading to be able to read xlog from historical (i.e. not current) timelines; otherwise, decoding fails after failover, because the archives are in the historical timeline. This is required to make "failover logical slots" possible; it currently has no other use, although theoretically it could be used by an extension that creates a slot on a standby and continues to replay from the slot when the standby is promoted. This commit includes a module in src/test/modules with functions to manipulate the slots (which is not otherwise possible in SQL code) in order to enable testing, and a new test in src/test/recovery to ensure that the behavior is as expected. Author: Craig Ringer Reviewed-By: Oleksii Kliukin, Andres Freund, Petr Jelínek	10 years ago
Andres Freund	b67aaf21e8	Add CASCADE support for CREATE EXTENSION. ... Without CASCADE, if an extension has an unfullfilled dependency on another extension, CREATE EXTENSION ERRORs out with "required extension ... is not installed". That is annoying, especially when that dependency is an implementation detail of the extension, rather than something the extension's user can make sense of. In addition to CASCADE this also includes a small set of regression tests around CREATE EXTENSION. Author: Petr Jelinek, editorialized by Michael Paquier, Andres Freund Reviewed-By: Michael Paquier, Andres Freund, Jeff Janes Discussion: 557E0520.3040800@2ndquadrant.com	10 years ago
Alvaro Herrera	672e3ec0e9	Re-add BRIN isolation test ... This time, instead of using a core isolation test, put it on its own test module; this way it can require the pageinspect module to be present before running. The module's Makefile is loosely modeled after test_decoding's, so that it's easy to add further tests for either pg_regress or isolationtester later. Backpatch to 9.5.	10 years ago
Alvaro Herrera	b488c580ae	Allow on-the-fly capture of DDL event details ... This feature lets user code inspect and take action on DDL events. Whenever a ddl_command_end event trigger is installed, DDL actions executed are saved to a list which can be inspected during execution of a function attached to ddl_command_end. The set-returning function pg_event_trigger_ddl_commands can be used to list actions so captured; it returns data about the type of command executed, as well as the affected object. This is sufficient for many uses of this feature. For the cases where it is not, we also provide a "command" column of a new pseudo-type pg_ddl_command, which is a pointer to a C structure that can be accessed by C code. The struct contains all the info necessary to completely inspect and even reconstruct the executed command. There is no actual deparse code here; that's expected to come later. What we have is enough infrastructure that the deparsing can be done in an external extension. The intention is that we will add some deparsing code in a later release, as an in-core extension. A new test module is included. It's probably insufficient as is, but it should be sufficient as a starting point for a more complete and future-proof approach. Authors: Álvaro Herrera, with some help from Andres Freund, Ian Barwick, Abhijit Menon-Sen. Reviews by Andres Freund, Robert Haas, Amit Kapila, Michael Paquier, Craig Ringer, David Steele. Additional input from Chris Browne, Dimitri Fontaine, Stephen Frost, Petr Jelínek, Tom Lane, Jim Nasby, Steven Singer, Pavel Stěhule. Based on original work by Dimitri Fontaine, though I didn't use his code. Discussion: https://www.postgresql.org/message-id/m2txrsdzxa.fsf@2ndQuadrant.fr https://www.postgresql.org/message-id/20131108153322.GU5809@eldon.alvh.no-ip.org https://www.postgresql.org/message-id/20150215044814.GL3391@alvh.no-ip.org	10 years ago
Stephen Frost	0bf22e0c8b	RLS fixes, new hooks, and new test module ... In prepend_row_security_policies(), defaultDeny was always true, so if there were any hook policies, the RLS policies on the table would just get discarded. Fixed to start off with defaultDeny as false and then properly set later if we detect that only the default deny policy exists for the internal policies. The infinite recursion detection in fireRIRrules() didn't properly manage the activeRIRs list in the case of WCOs, so it would incorrectly report infinite recusion if the same relation with RLS appeared more than once in the rtable, for example "UPDATE t ... FROM t ...". Further, the RLS expansion code in fireRIRrules() was handling RLS in the main loop through the rtable, which lead to RTEs being visited twice if they contained sublink subqueries, which prepend_row_security_policies() attempted to handle by exiting early if the RTE already had securityQuals. That doesn't work, however, since if the query involved a security barrier view on top of a table with RLS, the RTE would already have securityQuals (from the view) by the time fireRIRrules() was invoked, and so the table's RLS policies would be ignored. This is fixed in fireRIRrules() by handling RLS in a separate loop at the end, after dealing with any other sublink subqueries, thus ensuring that each RTE is only visited once for RLS expansion. The inheritance planner code didn't correctly handle non-target relations with RLS, which would get turned into subqueries during planning. Thus an update of the form "UPDATE t1 ... FROM t2 ..." where t1 has inheritance and t2 has RLS quals would fail. Fix by making sure to copy in and update the securityQuals when they exist for non-target relations. process_policies() was adding WCOs to non-target relations, which is unnecessary, and could lead to a lot of wasted time in the rewriter and the planner. Fix by only adding WCO policies when working on the result relation. Also in process_policies, we should be copying the USING policies to the WITH CHECK policies on a per-policy basis, fix by moving the copying up into the per-policy loop. Lastly, as noted by Dean, we were simply adding policies returned by the hook provided to the list of quals being AND'd, meaning that they would actually restrict records returned and there was no option to have internal policies and hook-based policies work together permissively (as all internal policies currently work). Instead, explicitly add support for both permissive and restrictive policies by having a hook for each and combining the results appropriately. To ensure this is all done correctly, add a new test module (test_rls_hooks) to test the various combinations of internal, permissive, and restrictive hook policies. Largely from Dean Rasheed (thanks!): CAEZATCVmFUfUOwwhnBTcgi6AquyjQ0-1fyKd0T3xBWJvn+xsFA@mail.gmail.com Author: Dean Rasheed, though I added the new hooks and test module.	11 years ago
Alvaro Herrera	73c986adde	Keep track of transaction commit timestamps ... Transactions can now set their commit timestamp directly as they commit, or an external transaction commit timestamp can be fed from an outside system using the new function TransactionTreeSetCommitTsData(). This data is crash-safe, and truncated at Xid freeze point, same as pg_clog. This module is disabled by default because it causes a performance hit, but can be enabled in postgresql.conf requiring only a server restart. A new test in src/test/modules is included. Catalog version bumped due to the new subdirectory within PGDATA and a couple of new SQL functions. Authors: Álvaro Herrera and Petr Jelínek Reviewed to varying degrees by Michael Paquier, Andres Freund, Robert Haas, Amit Kapila, Fujii Masao, Jaime Casanova, Simon Riggs, Steven Singer, Peter Eisentraut	11 years ago
Alvaro Herrera	da34731bd3	Install kludges to fix check-world for src/test/modules ... check-world failed in a completely clean tree, because src/test/modules fail to build unless errcodes.h is generated first. To fix this, install a dependency in src/test/modules' Makefile so that the necessary file is generated. Even with this, running "make check" within individual module subdirs will still fail because the dependency is not considered there, but this case is less interesting and would be messier to fix. check-world still failed with the above fix in place, this time because dummy_seclabel used LOAD to load the dynamic library, which doesn't work because the @libdir@ (expanded by the makefile) is expanded to the final install path, not the temporary installation directory used by make check. To fix, tweak things so that CREATE EXTENSION can be used instead, which solves the problem because the library path is expanded by the backend, which is aware of the true libdir.	11 years ago
Alvaro Herrera	22dfd116a1	Move test modules from contrib to src/test/modules ... This is advance preparation for introducing even more test modules; the easy solution is to add them to contrib, but that's bloated enough that it seems a good time to think of something different. Moved modules are dummy_seclabel, test_shm_mq, test_parser and worker_spi. (test_decoding was also a candidate, but there was too much opposition to moving that one. We can always reconsider later.)	11 years ago