29 Commits (012ee842590d7bb56f250c15e00a8611ba0ae1da)

Author	SHA1	Message	Date
Peter Eisentraut	012ee84259	Add a test for UCS_BASIC collation	3 years ago
Peter Eisentraut	37851a8b83	Database-level collation version tracking ... This adds to database objects the same version tracking that collation objects have. There is a new pg_database column datcollversion that stores the version, a new function pg_database_collation_actual_version() to get the version from the operating system, and a new subcommand ALTER DATABASE ... REFRESH COLLATION VERSION. This was not originally added together with pg_collation.collversion, since originally version tracking was only supported for ICU, and ICU on a database-level is not currently supported. But we now have version tracking for glibc (since PG13), FreeBSD (since PG14), and Windows (since PG13), so this is useful to have now. Reviewed-by: Julien Rouhaud <rjuju123@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/f0ff3190-29a3-5b39-a179-fa32eee57db6%40enterprisedb.com	4 years ago
Thomas Munro	ec48314708	Revert per-index collation version tracking feature. ... Design problems were discovered in the handling of composite types and record types that would cause some relevant versions not to be recorded. Misgivings were also expressed about the use of the pg_depend catalog for this purpose. We're out of time for this release so we'll revert and try again. Commits reverted: 1bf946bd: Doc: Document known problem with Windows collation versions. cf002008: Remove no-longer-relevant test case. ef387bed: Fix bogus collation-version-recording logic. 0fb0a050: Hide internal error for pg_collation_actual_version(<bad OID>). ff942057: Suppress "warning: variable 'collcollate' set but not used". d50e3b1f: Fix assertion in collation version lookup. f24b1569: Rethink extraction of collation dependencies. 257836a7: Track collation versions for indexes. cd6f479e: Add pg_depend.refobjversion. 7d1297df: Remove pg_collation.collversion. Discussion: https://postgr.es/m/CA%2BhUKGLhj5t1fcjqAu8iD9B3ixJtsTNqyCCD4V0aTO9kAKAjjA%40mail.gmail.com	5 years ago
Thomas Munro	7d1297df08	Remove pg_collation.collversion. ... This model couldn't be extended to cover the default collation, and didn't have any information about the affected database objects when the version changed. Remove, in preparation for a follow-up commit that will add a new mechanism. Author: Thomas Munro <thomas.munro@gmail.com> Reviewed-by: Julien Rouhaud <rjuju123@gmail.com> Reviewed-by: Peter Eisentraut <peter.eisentraut@2ndquadrant.com> Discussion: https://postgr.es/m/CAEepm%3D0uEQCpfq_%2BLYFBdArCe4Ot98t1aR4eYiYTe%3DyavQygiQ%40mail.gmail.com	5 years ago
Tom Lane	d677550493	Allow to_date/to_timestamp to recognize non-English month/day names. ... to_char() has long allowed the TM (translation mode) prefix to specify output of translated month or day names; but that prefix had no effect in input format strings. Now it does. to_date() and to_timestamp() will now recognize the same month or day names that to_char() would output for the same format code. Matching is case-insensitive (per the active collation's notion of what that means), just as it has always been for English month/day names without the TM prefix. (As per the discussion thread, there are lots of cases that this feature will not handle, such as alternate day names. But being able to accept what to_char() will output seems useful enough.) In passing, fix some shaky English and violations of message style guidelines in jsonpath errors for the .datetime() method, which depends on this code. Juan José Santamaría Flecha, reviewed and modified by me, with other commentary from Alvaro Herrera, Tomas Vondra, Arthur Zakirov, Peter Eisentraut, Mark Dilger. Discussion: https://postgr.es/m/CAC+AXB3u1jTngJcoC1nAHBf=M3v-jrEfo86UFtCqCjzbWS9QhA@mail.gmail.com	6 years ago
Thomas Munro	cce95a2f02	Remove obsolete collation test. ... The previous commit forgot to remove this test, which no longer holds on all systems.	6 years ago
Peter Eisentraut	f140007050	Run UTF8-requiring collation tests by default ... The tests collate.icu.utf8 and collate.linux.utf8 were previously only run when explicitly selected via EXTRA_TESTS. They require a UTF8 database, because the error messages in the expected files refer to that, and they use some non-ASCII characters in the tests. Since users can select any locale and encoding for the regression test run, it was not possible to include these tests automatically. To fix, use psql's \if facility to check various prerequisites such as platform and the server encoding and quit the tests at the very beginning if the configuration is not adequate. We then need to maintain alternative expected files for these tests, but they are very tiny and never need to change after this. These two tests are now run automatically as part of the regression tests. Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> Discussion: https://www.postgresql.org/message-id/flat/052295c2-a2e1-9a21-bd36-8fbff8686cf3%402ndquadrant.com	7 years ago
Tom Lane	7ad1cd31bf	Repair assorted issues in locale data extraction. ... cache_locale_time (extraction of LC_TIME-related info) had never been taught the lessons we previously learned about extraction of info related to LC_MONETARY and LC_NUMERIC. Specifically, commit 95a777c61 taught PGLC_localeconv() that data coming out of localeconv() was in an encoding determined by the relevant locale, but we didn't realize that there's a similar issue with strftime(). And commit a4930e7ca hardened PGLC_localeconv() against errors occurring partway through, but failed to do likewise for cache_locale_time(). So, rearrange the latter function to perform encoding conversion and not risk failure while it's got the locales set to temporary values. This time around I also changed PGLC_localeconv() to treat it as FATAL if it can't restore the previous settings of the locale values. There is no reason (except possibly OOM) for that to fail, and proceeding with the wrong locale values seems like a seriously bad idea --- especially on Windows where we have to also temporarily change LC_CTYPE. Also, protect against the possibility that we can't identify the codeset reported for LC_MONETARY or LC_NUMERIC; rather than just failing, try to validate the data without conversion. The user-visible symptom this fixes is that if LC_TIME is set to a locale name that implies an encoding different from the database encoding, non-ASCII localized day and month names would be retrieved in the wrong encoding, leading to either unexpected encoding-conversion error reports or wrong output from to_char(). The other possible failure modes are unlikely enough that we've not seen reports of them, AFAIK. The encoding conversion problems do not manifest on Windows, since we'd already created special-case code to handle that issue there. Per report from Juan José Santamaría Flecha. Back-patch to all supported versions. Juan José Santamaría Flecha and Tom Lane Discussion: https://postgr.es/m/CAC+AXB22So5aZm2vZe+MChYXec7gWfr-n-SK-iO091R0P_1Tew@mail.gmail.com	7 years ago
Peter Eisentraut	5e1963fb76	Collations with nondeterministic comparison ... This adds a flag "deterministic" to collations. If that is false, such a collation disables various optimizations that assume that strings are equal only if they are byte-wise equal. That then allows use cases such as case-insensitive or accent-insensitive comparisons or handling of strings with different Unicode normal forms. This functionality is only supported with the ICU provider. At least glibc doesn't appear to have any locales that work in a nondeterministic way, so it's not worth supporting this for the libc provider. The term "deterministic comparison" in this context is from Unicode Technical Standard #10 (https://unicode.org/reports/tr10/#Deterministic_Comparison). This patch makes changes in three areas: - CREATE COLLATION DDL changes and system catalog changes to support this new flag. - Many executor nodes and auxiliary code are extended to track collations. Previously, this code would just throw away collation information, because the eventually-called user-defined functions didn't use it since they only cared about equality, which didn't need collation information. - String data type functions that do equality comparisons and hashing are changed to take the (non-)deterministic flag into account. For comparison, this just means skipping various shortcuts and tie breakers that use byte-wise comparison. For hashing, we first need to convert the input string to a canonical "sort key" using the ICU analogue of strxfrm(). Reviewed-by: Daniel Verite <daniel@manitou-mail.org> Reviewed-by: Peter Geoghegan <pg@bowt.ie> Discussion: https://www.postgresql.org/message-id/flat/1ccc668f-4cbc-0bef-af67-450b47cdfee7@2ndquadrant.com	7 years ago
Peter Eisentraut	727921f466	Hide cascade messages in collate tests ... These are not relevant to the tests and would just uselessly bloat patches.	7 years ago
Peter Eisentraut	eccfef81e1	ICU support ... Add a column collprovider to pg_collation that determines which library provides the collation data. The existing choices are default and libc, and this adds an icu choice, which uses the ICU4C library. The pg_locale_t type is changed to a union that contains the provider-specific locale handles. Users of locale information are changed to look into that struct for the appropriate handle to use. Also add a collversion column that records the version of the collation when it is created, and check at run time whether it is still the same. This detects potentially incompatible library upgrades that can corrupt indexes and other structures. This is currently only supported by ICU-provided collations. initdb initializes the default collation set as before from the `locale -a` output but also adds all available ICU locales with a "-x-icu" appended. Currently, ICU-provided collations can only be explicitly named collations. The global database locales are still always libc-provided. ICU support is enabled by configure --with-icu. Reviewed-by: Thomas Munro <thomas.munro@enterprisedb.com> Reviewed-by: Andreas Karlsson <andreas@proxel.se>	9 years ago
Peter Eisentraut	6d16ecc646	Add CREATE COLLATION IF NOT EXISTS clause ... The core of the functionality was already implemented when pg_import_system_collations was added. This just exposes it as an option in the SQL command.	9 years ago
Heikki Linnakangas	4429f6a9e3	Support range data types. ... Selectivity estimation functions are missing for some range type operators, which is a TODO. Jeff Davis	14 years ago
Peter Eisentraut	6307fff358	Fix typo	15 years ago
Tom Lane	1abd146ddd	Adjust comments about collate.linux.utf8 regression test. ... This test should now work in any database with UTF8 encoding, regardless of the database's default locale. The former restriction was really "doesn't work if default locale is C", and that was because of not handling mbstowcs/wcstombs correctly.	15 years ago
Tom Lane	c947325856	Support a COLLATE clause in plpgsql variable declarations. ... This allows the usual rules for assigning a collation to a local variable to be overridden. Per discussion, it seems appropriate to support this rather than forcing all local variables to have the argument-derived collation.	15 years ago
Tom Lane	1e16a8107d	Teach regular expression operators to honor collations. ... This involves getting the character classification and case-folding functions in the regex library to use the collations infrastructure. Most of this work had been done already in connection with the upper/lower and LIKE logic, so it was a simple matter of transposition. While at it, split out these functions into a separate source file regc_pg_locale.c, so that they can be correctly labeled with the Postgres project's license rather than the Scriptics license. These functions are 100% Postgres-written code whereas what remains in regc_locale.c is still mostly not ours, so lumping them both under the same copyright notice was getting more and more misleading.	15 years ago
Tom Lane	f89e4dfa75	Remove collate.linux.utf8.sql's assumptions about ".utf8" in locale names. ... Tweak the test so that it does not depend on the platform using ".utf8" as the extension signifying that a locale uses UTF8 encoding. For the most part this just requires using the abbreviated collation names "en_US" etc, though I had to work a bit harder on the collation creation tests. This opens the door to using the test on platforms that spell locales differently, for example ".utf-8" or ".UTF-8". Also, the test is now somewhat useful with server encodings other than UTF8; though depending on which encoding is selected, different subsets of it will fail for lack of character set support.	15 years ago
Tom Lane	a19002d4e5	Adjust collation determination rules as per discussion. ... Remove crude hack that tried to propagate collation through a function-returning-record, ie, from the function's arguments to individual fields selected from its result record. That is just plain inconsistent, because the function result is composite and cannot have a collation; and there's no hope of making this kind of action-at-a-distance work consistently. Adjust regression test cases that expected this to happen. Meanwhile, the behavior of casting to a domain with a declared collation stays the same as it was, since that seemed to be the consensus.	15 years ago
Tom Lane	a4425e3200	Fix collation handling in plpgsql functions. ... Make plpgsql treat the input collation as a polymorphism variable, so that we cache separate plans for each input collation that's used in a particular session, as per recent discussion. Propagate the input collation to all collatable input parameters. I chose to also propagate the input collation to all declared variables of collatable types, which is a bit more debatable but seems to be necessary for non-astonishing behavior. (Copying a parameter into a separate local variable shouldn't result in a change of behavior, for example.) There is enough infrastructure here to support declaring a collation for each local variable to override that default, but I thought we should wait to see what the field demand is before adding such a feature. In passing, remove exec_get_rec_fieldtype(), which wasn't used anywhere. Documentation patch to follow.	15 years ago
Tom Lane	27dc7e240b	Fix handling of collation in SQL-language functions. ... Ensure that parameter symbols receive collation from the function's resolved input collation, and fix inlining to behave properly. BTW, this commit lays about 90% of the infrastructure needed to support use of argument names in SQL functions. Parsing of parameters is now done via the parser-hook infrastructure ... we'd just need to supply a column-ref hook ...	15 years ago
Tom Lane	3bba9ce945	Clean up handling of COLLATE clauses in index column definitions. ... Ensure that COLLATE at the top level of an index expression is treated the same as a grammatically separate COLLATE. Fix bogus reverse-parsing logic in pg_get_indexdef.	15 years ago
Tom Lane	37d6d07dda	Throw error for indeterminate collation of an ORDER/GROUP/DISTINCT target. ... This restores a parse error that was thrown (though only in the ORDER BY case) by the original collation patch. I had removed it in my recent revisions because it was thrown at a place where collations now haven't been computed yet; but I thought of another way to handle it. Throwing the error at parse time, rather than leaving it to be done at runtime, is good because a syntax error pointer is helpful for localizing the problem. We can reasonably assume that the comparison function for a collatable datatype will complain if it doesn't have a collation to use. Now the planner might choose to implement GROUP or DISTINCT via hashing, in which case no runtime error would actually occur, but it seems better to throw error consistently rather than let the error depend on what the planner chooses to do. Another possible objection is that the user might specify a nondefault sort operator that doesn't care about collation ... but that's surely an uncommon usage, and it wouldn't hurt him to throw in a COLLATE clause anyway. This change also makes the ORDER BY/GROUP BY/DISTINCT case more consistent with the UNION/INTERSECT/EXCEPT case, which was already coded to throw this error even though the same objections could be raised there.	15 years ago
Tom Lane	b310b6e31c	Revise collation derivation method and expression-tree representation. ... All expression nodes now have an explicit output-collation field, unless they are known to only return a noncollatable data type (such as boolean or record). Also, nodes that can invoke collation-aware functions store a separate field that is the collation value to pass to the function. This avoids confusion that arises when a function has collatable inputs and noncollatable output type, or vice versa. Also, replace the parser's on-the-fly collation assignment method with a post-pass over the completed expression tree. This allows us to use a more complex (and hopefully more nearly spec-compliant) assignment rule without paying for it in extra storage in every expression node. Fix assorted bugs in the planner's handling of collations by making collation one of the defining properties of an EquivalenceClass and by converting CollateExprs into discardable RelabelType nodes during expression preprocessing.	15 years ago
Peter Eisentraut	3d9f7ec1ff	Add test case for collation mismatch in recursive query ... This isn't very important by itself, but was left on my list of things without test coverage for the collation feature.	15 years ago
Peter Eisentraut	b9cff97fdf	Don't allow CREATE TABLE AS to create a column with invalid collation ... It is possible that an expression ends up with a collatable type but without a collation. CREATE TABLE AS could then create a table based on that. But such a column cannot be dumped with valid SQL syntax, so we disallow creating such a column. per test report from Noah Misch	15 years ago
Peter Eisentraut	4442e1975d	When creating a collation, check that the locales can be loaded ... This is the same check that would happen later when the collation is used, but it's friendlier to check the collation already when it is created.	15 years ago
Peter Eisentraut	b313bca0af	DDL support for collations ... - collowner field - CREATE COLLATION - ALTER COLLATION - DROP COLLATION - COMMENT ON COLLATION - integration with extensions - pg_dump support for the above - dependency management - psql tab completion - psql \dO command	15 years ago
Peter Eisentraut	414c5a2ea6	Per-column collation support ... This adds collation support for columns and domains, a COLLATE clause to override it per expression, and B-tree index support. Peter Eisentraut reviewed by Pavel Stehule, Itagaki Takahiro, Robert Haas, Noah Misch	15 years ago