25 Commits (b3b4d8e68ae83f432f43f035c7eb481ef93e1583)

Author	SHA1	Message	Date
Andrew Dunstan	b3b4d8e68a	Move Perl test modules to a better namespace ... The five modules in our TAP test framework all had names in the top level namespace. This is unwise because, even though we're not exporting them to CPAN, the names can leak, for example if they are exported by the RPM build process. We therefore move the modules to the PostgreSQL::Test namespace. In the process PostgresNode is renamed to Cluster, and TestLib is renamed to Utils. PostgresVersion becomes simply PostgreSQL::Version, to avoid possible confusion about what it's the version of. Discussion: https://postgr.es/m/aede93a4-7d92-ef26-398f-5094944c2504@dunslane.net Reviewed by Erik Rijkers and Michael Paquier	4 years ago
Michael Paquier	5fcb23c18f	Remove some unused variables in TAP tests ... Author: Amul Sul Discussion: https://postgr.es/m/CAAJ_b96xuFh4JZE6p-zhLyDu7q=NbxJfb1z_yeAu6t-MqaBC+Q@mail.gmail.com	4 years ago
Michael Paquier	f7a9a3d4b2	Skip trailing whitespaces when parsing integer options ... strtoint(), via strtol(), would skip leading whitespaces but the same rule was not applied for trailing whitespaces, leading to an inconsistent behavior. Some tests are changed to cover more this area. Author: Michael Paquier Reviewed-by: Kyotaro Horiguchi Discussion: https://postgr.es/m/YP5Pv0d13Ct+03ve@paquier.xyz	4 years ago
Michael Paquier	6f164e6d17	Unify parsing logic for command-line integer options ... Most of the integer options for command-line binaries now make use of a single routine able to do the job, fixing issues with the detection of sloppy values caused for example by the use of atoi(), that fails on strings beginning with numerical characters with junk trailing characters. This commit cuts down the number of strings requiring translation by 26 per my count, switching the code to have two error types for invalid and out-of-range values instead. Much more could be done here, with float or even int64 options, but int32 was the most appealing case as it is possible to rely on strtol() to do the job reliably. Note that there are some exceptions for now, like pg_ctl or pg_upgrade that use their own logging logic. A couple of negative TAP tests required some adjustments for the new errors generated. pg_dump and pg_restore tracked the maximum number of parallel jobs within the option parsing. The code is refactored a bit to track that in the code dedicated to parallelism instead. Author: Kyotaro Horiguchi, Michael Paquier Reviewed-by: David Rowley, Álvaro Herrera Discussion: https://postgr.es/m/CALj2ACXqdG9WhqVoJ9zYf-iZt7sgK7Szv5USs=he6NnWQ2ofTA@mail.gmail.com	4 years ago
Andrew Dunstan	8fa6e6919c	Add a copyright notice to perl files lacking one.	4 years ago
Michael Paquier	64725728e7	Add more TAP tests for pg_dump options with range checks ... This adds two tests for --extra-float-digits and --rows-per-insert, similar to what exists for --compress. Author: Dong Wook Lee Discussion: https://postgr.es/m/CAAcByaJsgrB-qc-ALb0mALprRGLAdmcBap7SZxO4kCAU-JEHcQ@mail.gmail.com	5 years ago
Alvaro Herrera	2f9eb31320	pg_dump: Allow dumping data of specific foreign servers ... The new command-line switch --include-foreign-data=PATTERN lets the user specify foreign servers from which to dump foreign table data. This can be refined by further inclusion/exclusion switches, so that the user has full control over which tables to dump. A limitation is that this doesn't work in combination with parallel dumps, for implementation reasons. This might be lifted in the future, but requires shuffling some code around. Author: Luis Carril <luis.carril@swarm64.com> Reviewed-by: Daniel Gustafsson <daniel@yesql.se> Reviewed-by: Surafel Temesgen <surafel3000@gmail.com> Reviewed-by: vignesh C <vignesh21@gmail.com> Reviewed-by: Álvaro Herrera <alvherre@2ndQuadrant.com> Discussion: https://postgr.es/m/LEJPR01MB0185483C0079D2F651B16231E7FC0@LEJPR01MB0185.DEUPRD01.PROD.OUTLOOK.DE	6 years ago
Peter Eisentraut	1b29e990e3	Add missing serial commas	6 years ago
Tom Lane	db6e2b4c52	Initial pgperltidy run for v12. ... Make all the perl code look nice, too (for some value of "nice").	6 years ago
Alvaro Herrera	413ccaa74d	pg_restore: Require "-f -" to mean stdout ... The previous convention that stdout was selected by default when nothing is specified was just too error-prone. After a suggestion from Andrew Gierth. Author: Euler Taveira Reviewed-by: Yoshikazu Imai, José Arthur Benetasso Villanova Discussion: https://postgr.es/m/87sgwrmhdv.fsf@news-spur.riddles.org.uk	7 years ago
Peter Eisentraut	cc8d415117	Unified logging system for command-line programs ... This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com	7 years ago
Alvaro Herrera	7e413a0f82	pg_dump: allow multiple rows per insert ... This is useful to speed up loading data in a different database engine. Authors: Surafel Temesgen and David Rowley. Lightly edited by Álvaro. Reviewed-by: Fabien Coelho Discussion: https://postgr.es/m/CALAY4q9kumSdnRBzvRJvSRf2+BH20YmSvzqOkvwpEmodD-xv6g@mail.gmail.com	7 years ago
Andrew Dunstan	97b6f2eb75	Remove tests for pg_dumpall --exclude-database missing argument ... It turns out that different getopt implementations spell the error for missing arguments different ways. This test is of fairly marginal value, so instead of trying to keep up with the different error messages just remove the test.	7 years ago
Andrew Dunstan	f092de0503	Add --exclude-database option to pg_dumpall ... This option functions similarly to pg_dump's --exclude-table option, but for database names. The option can be given once, and the argument can be a pattern including wildcard characters. Author: Andrew Dunstan. Reviewd-by: Fabien Coelho and Michael Paquier Discussion: https://postgr.es/m/43a54a47-4aa7-c70e-9ca6-648f436dd6e6@2ndQuadrant.com	7 years ago
Andres Freund	578b229718	Remove WITH OIDS support, change oid catalog column visibility. ... Previously tables declared WITH OIDS, including a significant fraction of the catalog tables, stored the oid column not as a normal column, but as part of the tuple header. This special column was not shown by default, which was somewhat odd, as it's often (consider e.g. pg_class.oid) one of the more important parts of a row. Neither pg_dump nor COPY included the contents of the oid column by default. The fact that the oid column was not an ordinary column necessitated a significant amount of special case code to support oid columns. That already was painful for the existing, but upcoming work aiming to make table storage pluggable, would have required expanding and duplicating that "specialness" significantly. WITH OIDS has been deprecated since 2005 (commit ff02d0a05280e0). Remove it. Removing includes: - CREATE TABLE and ALTER TABLE syntax for declaring the table to be WITH OIDS has been removed (WITH (oids[ = true]) will error out) - pg_dump does not support dumping tables declared WITH OIDS and will issue a warning when dumping one (and ignore the oid column). - restoring an pg_dump archive with pg_restore will warn when restoring a table with oid contents (and ignore the oid column) - COPY will refuse to load binary dump that includes oids. - pg_upgrade will error out when encountering tables declared WITH OIDS, they have to be altered to remove the oid column first. - Functionality to access the oid of the last inserted row (like plpgsql's RESULT_OID, spi's SPI_lastoid, ...) has been removed. The syntax for declaring a table WITHOUT OIDS (or WITH (oids = false) for CREATE TABLE) is still supported. While that requires a bit of support code, it seems unnecessary to break applications / dumps that do not use oids, and are explicit about not using them. The biggest user of WITH OID columns was postgres' catalog. This commit changes all 'magic' oid columns to be columns that are normally declared and stored. To reduce unnecessary query breakage all the newly added columns are still named 'oid', even if a table's column naming scheme would indicate 'reloid' or such. This obviously requires adapting a lot code, mostly replacing oid access via HeapTupleGetOid() with access to the underlying Form_pg_*->oid column. The bootstrap process now assigns oids for all oid columns in genbki.pl that do not have an explicit value (starting at the largest oid previously used), only oids assigned later by oids will be above FirstBootstrapObjectId. As the oid column now is a normal column the special bootstrap syntax for oids has been removed. Oids are not automatically assigned during insertion anymore, all backend code explicitly assigns oids with GetNewOidWithIndex(). For the rare case that insertions into the catalog via SQL are called for the new pg_nextoid() function can be used (which only works on catalog tables). The fact that oid columns on system tables are now normal columns means that they will be included in the set of columns expanded by * (i.e. SELECT * FROM pg_class will now include the table's oid, previously it did not). It'd not technically be hard to hide oid column by default, but that'd mean confusing behavior would either have to be carried forward forever, or it'd cause breakage down the line. While it's not unlikely that further adjustments are needed, the scope/invasiveness of the patch makes it worthwhile to get merge this now. It's painful to maintain externally, too complicated to commit after the code code freeze, and a dependency of a number of other patches. Catversion bump, for obvious reasons. Author: Andres Freund, with contributions by John Naylor Discussion: https://postgr.es/m/20180930034810.ywp2c7awz7opzcfr@alap3.anarazel.de	7 years ago
Michael Paquier	c34bca9ea5	Consolidate cross-option checks in pg_restore ... This moves one check for conflicting options from the archive restore code to the main function where other similar checks are performed. Also reword the error message to be consistent with other messages. The only option combination impacted is --create specified with --single-transaction, and informing the caller at an early step saves from opening the archive worked on. A TAP test is added for this combination. Author: Daniel Gustafsson Reviewed-by: Fabien Coelho Discussion: https://postgr.es/m/616808BD-4B59-4E6C-97A9-7317F62D5570@yesql.se	7 years ago
Thomas Munro	387a5cfb94	Add pg_dump --on-conflict-do-nothing option. ... When dumping INSERT statements, optionally add ON CONFLICT DO NOTHING. Author: Surafel Temesgen Reviewed-by: Takeshi Ideriha, Nico Williams, Dilip Kumar Discussion: https://postgr.es/m/CALAY4q-PQ9cOEzs2%2BQHK5ObfF_4QbmBaYXbZx6BGGN66Q-n8FA%40mail.gmail.com	7 years ago
Peter Eisentraut	d4f16d5071	perltidy: Add option --nooutdent-long-quotes	7 years ago
Stephen Frost	805e8bc702	Adjust number of tests for pg_dump 001_basic.pl ... When removing a test, need to make sure the count of tests is adjusted when it isn't calculated.	9 years ago
Stephen Frost	5abda5a3e9	pg_dump: Remove "option requires an argument -- j" test ... This is really testing getopt more than pg_dump, and what getopt returns exactly appears to differ based on platform, so remove this test. Per buildfarm.	9 years ago
Stephen Frost	31a8b77a92	Improve pg_dump regression tests and code coverage ... These improvements bring the lines-of-code coverage of pg_dump.c up to 87.7% (at least using LCOV 1.12, 1.11 seems to differ slightly). Nearly every function is covered, three of the four which aren't are only called when talking to older PG instances. There is more which can, and should, be done here to improve the coverage but it's past time to see what the buildfarm thinks of this. What has been added: - Coverage for many more command-line options - Use command_fails_like instead of command_exit_is - Operator classes, operator families - Text search configuration, templates, parsers, dictionaries - FDWs, servers, foreign tables - Materialized views - Improved Publications / Subscriptions test (though this needs work, see PG10 open items and tests marked with XXX in 002_pg_dump.pl) - Unlogged tables - Partitioned tables - Additional ACL testing for various object types There is room for improvement, specifically: - Various type-based option (alignment, storage, etc) - Composite type collation - Extra Procedural language functions (inline, validator) - Different function options (SRF, Transform, config, security definer, cost, leakproof) - OpClass options (default, storage, order by, recheck) - OpFamily options (order by, recheck) - Aggregate functions (combinefunc, serialfunc, deserialfunc, etc) - Text Search parser 'headline' - Text Search template 'init' - FDW options (handler, validator, options) - Server options (type, version, options) - User mapping options - Default ACLs for sequences, types - Security labels - View circular dependencies (last function that needs coverage) - Toast table autovacuum options - Replica identity options - Independent indexes (plus marking them as clustered on) - Deferrable / initially deferred constraints - Independent domain constraints There's bits of extension pg_dump'ing also not covered, but those will need to go into test_pg_dump (such as having a filter for config tables). Last, but not least, this approximately halves the number of tests run with 'ok()' by removing the ok()-based checking of if all runs are covered by each test. Instead, 002_pg_dump.pl will just exit out in such a case (with a message in the log file). In general, when adding tests, cover all runs unless there is a very good reason not to (such as adding a 'catch-all' case). With these changes, the resulting output and number of "tests" run is actually reduced.	9 years ago
Stephen Frost	6af8b89adb	perltidy pg_dump TAP tests ... The pg_dump TAP tests have gotten pretty far from what perltidy thinks they should be, so fix that, and in passing use long-form argument names with arguments passed via "=" in a similar vein to 58da833. No functional changes here, just whitespace and changing runs from "-f" to "--file=", and similar.	9 years ago
Stephen Frost	9b815a8ff2	Add basic pg_dumpall/pg_restore TAP tests ... For reasons unknown, pg_dumpall and pg_restore managed to escape the basic set of TAP tests that were added for pg_dump in 6bd356c3, so let's get them added now. A few minor adjustments are also made to the dump/restore tests to improve code coverage for pg_restore/pg_dumpall.	9 years ago
Noah Misch	3be0a62ffe	Finish pgindent run for 9.6: Perl files.	9 years ago
Stephen Frost	6bd356c33a	Add TAP tests for pg_dump ... This TAP test suite will create a new cluster, populate it based on the 'create_sql' values in the '%tests' hash, run all of the runs defined in the '%pgdump_runs' hash, and then for each test in the '%tests' hash, compare each run's output the the regular expression defined for the test under the 'like' and 'unlike' functions, as appropriate. While this test suite covers a fair bit of ground (67% of pg_dump.c and quite a bit of the other files in src/bin/pg_dump), there is still quite a bit which remains to be added to provide better code coverage. Still, this is quite a bit better than we had, and has found a few bugs already (note that the CREATE TRANSFORM test is commented out, as it is currently failing). Idea for using the TAP system from Tom, though all of the code is mine.	9 years ago