postgres

History

Tom Lane 1cff1b95ab Represent Lists as expansible arrays, not chains of cons-cells. Originally, Postgres Lists were a more or less exact reimplementation of Lisp lists, which consist of chains of separately-allocated cons cells, each having a value and a next-cell link. We'd hacked that once before (commit `d0b4399d8`) to add a separate List header, but the data was still in cons cells. That makes some operations -- notably list_nth() -- O(N), and it's bulky because of the next-cell pointers and per-cell palloc overhead, and it's very cache-unfriendly if the cons cells end up scattered around rather than being adjacent. In this rewrite, we still have List headers, but the data is in a resizable array of values, with no next-cell links. Now we need at most two palloc's per List, and often only one, since we can allocate some values in the same palloc call as the List header. (Of course, extending an existing List may require repalloc's to enlarge the array. But this involves just O(log N) allocations not O(N).) Of course this is not without downsides. The key difficulty is that addition or deletion of a list entry may now cause other entries to move, which it did not before. For example, that breaks foreach() and sister macros, which historically used a pointer to the current cons-cell as loop state. We can repair those macros transparently by making their actual loop state be an integer list index; the exposed "ListCell " pointer is no longer state carried across loop iterations, but is just a derived value. (In practice, modern compilers can optimize things back to having just one loop state value, at least for simple cases with inline loop bodies.) In principle, this is a semantics change for cases where the loop body inserts or deletes list entries ahead of the current loop index; but I found no such cases in the Postgres code. The change is not at all transparent for code that doesn't use foreach() but chases lists "by hand" using lnext(). The largest share of such code in the backend is in loops that were maintaining "prev" and "next" variables in addition to the current-cell pointer, in order to delete list cells efficiently using list_delete_cell(). However, we no longer need a previous-cell pointer to delete a list cell efficiently. Keeping a next-cell pointer doesn't work, as explained above, but we can improve matters by changing such code to use a regular foreach() loop and then using the new macro foreach_delete_current() to delete the current cell. (This macro knows how to update the associated foreach loop's state so that no cells will be missed in the traversal.) There remains a nontrivial risk of code assuming that a ListCell pointer will remain good over an operation that could now move the list contents. To help catch such errors, list.c can be compiled with a new define symbol DEBUG_LIST_MEMORY_USAGE that forcibly moves list contents whenever that could possibly happen. This makes list operations significantly more expensive so it's not normally turned on (though it is on by default if USE_VALGRIND is on). There are two notable API differences from the previous code: * lnext() now requires the List's header pointer in addition to the current cell's address. * list_delete_cell() no longer requires a previous-cell argument. These changes are somewhat unfortunate, but on the other hand code using either function needs inspection to see if it is assuming anything it shouldn't, so it's not all bad. Programmers should be aware of these significant performance changes: * list_nth() and related functions are now O(1); so there's no major access-speed difference between a list and an array. * Inserting or deleting a list element now takes time proportional to the distance to the end of the list, due to moving the array elements. (However, it typically doesn't require palloc or pfree, so except in long lists it's probably still faster than before.) Notably, lcons() used to be about the same cost as lappend(), but that's no longer true if the list is long. Code that uses lcons() and list_delete_first() to maintain a stack might usefully be rewritten to push and pop at the end of the list rather than the beginning. * There are now list_insert_nth...() and list_delete_nth...() functions that add or remove a list cell identified by index. These have the data-movement penalty explained above, but there's no search penalty. * list_concat() and variants now copy the second list's data into storage belonging to the first list, so there is no longer any sharing of cells between the input lists. The second argument is now declared "const List *" to reflect that it isn't changed. This patch just does the minimum needed to get the new implementation in place and fix bugs exposed by the regression tests. As suggested by the foregoing, there's a fair amount of followup work remaining to do. Also, the ENABLE_LIST_COMPAT macros are finally removed in this commit. Code using those should have been gone a dozen years ago. Patch by me; thanks to David Rowley, Jesper Pedersen, and others for review. Discussion: https://postgr.es/m/11587.1550975080@sss.pgh.pa.us		6 years ago
..
adminpack	Update copyright for 2019	7 years ago
amcheck	Follow the rule that regression-test-created roles are named "regress_xxx".	6 years ago
auth_delay	Update copyright for 2019	7 years ago
auto_explain	Fix contrib/auto_explain to not cause problems in parallel workers.	6 years ago
bloom	Fix many typos and inconsistencies	6 years ago
btree_gin	Provide separate header file for built-in float types	7 years ago
btree_gist	Phase 2 pgindent run for v12.	6 years ago
citext	Fix typos in various places	6 years ago
cube	Fix inconsistencies in the code	6 years ago
dblink	Phase 2 pgindent run for v12.	6 years ago
dict_int	Update copyright for 2019	7 years ago
dict_xsyn	Update copyright for 2019	7 years ago
earthdistance	Fix earthdistance test suite function name typo.	7 years ago
file_fdw	Represent Lists as expansible arrays, not chains of cons-cells.	6 years ago
fuzzystrmatch	Update copyright for 2019	7 years ago
hstore	Move hash_any prototype from access/hash.h to utils/hashutils.h	7 years ago
hstore_plperl	Clean up PL/Perl's handling of the _() macro.	6 years ago
hstore_plpython	Avoid Python memory leaks in hstore_plpython and jsonb_plpython.	7 years ago
intagg	Schema-qualify some references to regprocedure.	9 years ago
intarray	Fix many typos and inconsistencies	6 years ago
isn	Update copyright for 2019	7 years ago
jsonb_plperl	Clean up PL/Perl's handling of the _() macro.	6 years ago
jsonb_plpython	Fix more typos and inconsistencies in the tree	6 years ago
lo	lo: Add test suite	8 years ago
ltree	Phase 2 pgindent run for v12.	6 years ago
ltree_plpython	Fix out-of-tree build for transform modules.	7 years ago
oid2name	Replace @postgresql.org with @lists.postgresql.org for mailinglists	7 years ago
pageinspect	Phase 2 pgindent run for v12.	6 years ago
passwordcheck	Update copyright for 2019	7 years ago
pg_buffercache	Remove WITH OIDS support, change oid catalog column visibility.	7 years ago
pg_freespacemap	Replace heapam.h includes with {table, relation}.h where applicable.	7 years ago
pg_prewarm	Fix more typos and inconsistencies in the tree	6 years ago
pg_standby	Replace @postgresql.org with @lists.postgresql.org for mailinglists	7 years ago
pg_stat_statements	Teach pg_stat_statements not to ignore FOR UPDATE clauses	6 years ago
pg_trgm	Represent Lists as expansible arrays, not chains of cons-cells.	6 years ago
pg_visibility	Phase 2 pgindent run for v12.	6 years ago
pgcrypto	Add gen_random_uuid function	6 years ago
pgrowlocks	Only allow heap in a number of contrib modules.	7 years ago
pgstattuple	Phase 2 pgindent run for v12.	6 years ago
postgres_fdw	Represent Lists as expansible arrays, not chains of cons-cells.	6 years ago
seg	Change floating-point output format for improved performance.	7 years ago
sepgsql	Represent Lists as expansible arrays, not chains of cons-cells.	6 years ago
spi	Fix more strcmp() calls using boolean-like comparisons for result checks	7 years ago
sslinfo	Phase 3 of pgindent updates.	8 years ago
start-scripts	Remove contrib/start-scripts/osx/.	8 years ago
tablefunc	Phase 2 pgindent run for v12.	6 years ago
tcn	Update copyright for 2019	7 years ago
test_decoding	Use appendStringInfoString and appendPQExpBufferStr where possible	6 years ago
tsm_system_rows	Phase 2 pgindent run for v12.	6 years ago
tsm_system_time	Phase 2 pgindent run for v12.	6 years ago
unaccent	Add combining characters to unaccent.rules.	7 years ago
uuid-ossp	Update copyright for 2019	7 years ago
vacuumlo	Fix copy-pasto in freeing memory on error in vacuumlo.	6 years ago
xml2	Phase 2 pgindent run for v12.	6 years ago
Makefile	Transforms for jsonb to PL/Perl	8 years ago
README	…
contrib-global.mk	…

README

The PostgreSQL contrib tree
---------------------------

This subtree contains porting tools, analysis utilities, and plug-in
features that are not part of the core PostgreSQL system, mainly
because they address a limited audience or are too experimental to be
part of the main source tree.  This does not preclude their
usefulness.

User documentation for each module appears in the main SGML
documentation.

When building from the source distribution, these modules are not
built automatically, unless you build the "world" target.  You can
also build and install them all by running "make all" and "make
install" in this directory; or to build and install just one selected
module, do the same in that module's subdirectory.

Some directories supply new user-defined functions, operators, or
types.  To make use of one of these modules, after you have installed
the code you need to register the new SQL objects in the database
system by executing a CREATE EXTENSION command.  In a fresh database,
you can simply do

    CREATE EXTENSION module_name;

See the PostgreSQL documentation for more information about this
procedure.