mirror of https://github.com/postgres/postgres
Bruce Momjian
26 years ago
parent
7559677551
commit
957e6a6921
@ -0,0 +1,2 @@ |
|||||||
|
These files are in standard Unix mailbox format, and are detail |
||||||
|
information related to the TODO list. |
||||||
@ -0,0 +1,107 @@ |
|||||||
|
From owner-pgsql-hackers@hub.org Fri May 14 16:00:46 1999 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id QAA02173 |
||||||
|
for <maillist@candle.pha.pa.us>; Fri, 14 May 1999 16:00:44 -0400 (EDT) |
||||||
|
Received: from hub.org (hub.org [209.167.229.1]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id QAA02824 for <maillist@candle.pha.pa.us>; Fri, 14 May 1999 16:00:45 -0400 (EDT) |
||||||
|
Received: from hub.org (hub.org [209.167.229.1]) |
||||||
|
by hub.org (8.9.3/8.9.3) with ESMTP id PAA47798; |
||||||
|
Fri, 14 May 1999 15:57:54 -0400 (EDT) |
||||||
|
(envelope-from owner-pgsql-hackers@hub.org) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Fri, 14 May 1999 15:54:30 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) |
||||||
|
by hub.org (8.9.3/8.9.3) id PAA47191 |
||||||
|
for pgsql-hackers-outgoing; Fri, 14 May 1999 15:54:28 -0400 (EDT) |
||||||
|
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||||
|
Received: from thelab.hub.org (nat194.147.mpoweredpc.net [142.177.194.147]) |
||||||
|
by hub.org (8.9.3/8.9.3) with ESMTP id PAA46457 |
||||||
|
for <pgsql-hackers@postgresql.org>; Fri, 14 May 1999 15:49:35 -0400 (EDT) |
||||||
|
(envelope-from scrappy@hub.org) |
||||||
|
Received: from localhost (scrappy@localhost) |
||||||
|
by thelab.hub.org (8.9.3/8.9.1) with ESMTP id QAA16128; |
||||||
|
Fri, 14 May 1999 16:49:44 -0300 (ADT) |
||||||
|
(envelope-from scrappy@hub.org) |
||||||
|
X-Authentication-Warning: thelab.hub.org: scrappy owned process doing -bs |
||||||
|
Date: Fri, 14 May 1999 16:49:44 -0300 (ADT) |
||||||
|
From: The Hermit Hacker <scrappy@hub.org> |
||||||
|
To: pgsql-hackers@postgreSQL.org |
||||||
|
cc: Jack Howarth <howarth@nitro.med.uc.edu> |
||||||
|
Subject: [HACKERS] postgresql bug report (fwd) |
||||||
|
Message-ID: <Pine.BSF.4.05.9905141649150.47191-100000@thelab.hub.org> |
||||||
|
MIME-Version: 1.0 |
||||||
|
Content-Type: TEXT/PLAIN; charset=US-ASCII |
||||||
|
Sender: owner-pgsql-hackers@postgreSQL.org |
||||||
|
Precedence: bulk |
||||||
|
Status: ROr |
||||||
|
|
||||||
|
|
||||||
|
Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy |
||||||
|
Systems Administrator @ hub.org |
||||||
|
primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org |
||||||
|
|
||||||
|
---------- Forwarded message ---------- |
||||||
|
Date: Fri, 14 May 1999 14:50:58 -0400 |
||||||
|
From: Jack Howarth <howarth@nitro.med.uc.edu> |
||||||
|
To: scrappy@hub.org |
||||||
|
Subject: postgresql bug report |
||||||
|
|
||||||
|
Marc, |
||||||
|
In porting the RedHat 6.0 srpm set for a linuxppc release we |
||||||
|
believe a bug has been identified in |
||||||
|
the postgresql source for 6.5-0.beta1. Our development tools are as |
||||||
|
follows... |
||||||
|
|
||||||
|
glibc 2.1.1 pre 2 |
||||||
|
linux 2.2.6 |
||||||
|
egcs 1.1.2 |
||||||
|
the latest binutils snapshot |
||||||
|
|
||||||
|
The bug that we see is that when egcs compiles postgresql at -O1 or |
||||||
|
higher (-O0 is fine), |
||||||
|
postgresql creates incorrectly formed databases such that when the user |
||||||
|
does a destroydb |
||||||
|
the database can not be destroyed. Franz Sirl has identified the problem |
||||||
|
as follows... |
||||||
|
|
||||||
|
it seems that this problem is a type casting/promotion bug in the |
||||||
|
source. The |
||||||
|
routine _bt_checkkeys() in backend/access/nbtree/nbtutils.c calls |
||||||
|
int2eq() in |
||||||
|
backend/utils/adt/int.c via a function pointer |
||||||
|
*fmgr_faddr(&key[0].sk_func). As |
||||||
|
the type information for int2eq is lost via the function pointer, |
||||||
|
the compiler |
||||||
|
passes 2 ints, but int2eq expects 2 (preformatted in a 32bit reg) |
||||||
|
int16's. |
||||||
|
This particular bug goes away, if I for example change int2eq to: |
||||||
|
|
||||||
|
bool |
||||||
|
int2eq(int32 arg1, int32 arg2) |
||||||
|
{ |
||||||
|
return (int16)arg1 == (int16)arg2; |
||||||
|
} |
||||||
|
|
||||||
|
This moves away the type casting/promotion "work" from caller to the |
||||||
|
callee and |
||||||
|
is probably the right thing to do for functions used via function |
||||||
|
pointers. |
||||||
|
|
||||||
|
...because of the large number of changes required to do this, Franz |
||||||
|
thought we should |
||||||
|
pass this on to the postgresql maintainers for correction. Please feel |
||||||
|
free to contact |
||||||
|
Franz Sirl (Franz.Sirl-kernel@lauterbach.com) if you have any questions |
||||||
|
on this bug |
||||||
|
report. |
||||||
|
|
||||||
|
-- |
||||||
|
------------------------------------------------------------------------------ |
||||||
|
Jack W. Howarth, Ph.D. 231 Bethesda Avenue |
||||||
|
NMR Facility Director Cincinnati, Ohio 45267-0524 |
||||||
|
Dept. of Molecular Genetics phone: (513) 558-4420 |
||||||
|
Univ. of Cincinnati College of Medicine fax: (513) 558-8474 |
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,94 @@ |
|||||||
|
From owner-pgsql-hackers@hub.org Wed Nov 25 19:01:02 1998 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id TAA16399 |
||||||
|
for <maillist@candle.pha.pa.us>; Wed, 25 Nov 1998 19:01:01 -0500 (EST) |
||||||
|
Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id SAA05250 for <maillist@candle.pha.pa.us>; Wed, 25 Nov 1998 18:53:12 -0500 (EST) |
||||||
|
Received: from localhost (majordom@localhost) |
||||||
|
by hub.org (8.9.1/8.9.1) with SMTP id SAA17798; |
||||||
|
Wed, 25 Nov 1998 18:49:38 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@hub.org) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 25 Nov 1998 18:49:07 +0000 (EST) |
||||||
|
Received: (from majordom@localhost) |
||||||
|
by hub.org (8.9.1/8.9.1) id SAA17697 |
||||||
|
for pgsql-hackers-outgoing; Wed, 25 Nov 1998 18:49:06 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||||
|
Received: from mail.enterprise.net (root@mail.enterprise.net [194.72.192.18]) |
||||||
|
by hub.org (8.9.1/8.9.1) with ESMTP id SAA17650; |
||||||
|
Wed, 25 Nov 1998 18:48:55 -0500 (EST) |
||||||
|
(envelope-from olly@lfix.co.uk) |
||||||
|
Received: from linda.lfix.co.uk (root@max01-040.enterprise.net [194.72.197.40]) |
||||||
|
by mail.enterprise.net (8.8.5/8.8.5) with ESMTP id XAA20539; |
||||||
|
Wed, 25 Nov 1998 23:48:52 GMT |
||||||
|
Received: from linda.lfix.co.uk (olly@localhost [127.0.0.1]) |
||||||
|
by linda.lfix.co.uk (8.9.1a/8.9.1/Debian/GNU) with ESMTP id XAA12089; |
||||||
|
Wed, 25 Nov 1998 23:48:52 GMT |
||||||
|
Message-Id: <199811252348.XAA12089@linda.lfix.co.uk> |
||||||
|
X-Mailer: exmh version 2.0.2 2/24/98 (debian) |
||||||
|
X-URL: http://www.lfix.co.uk/oliver |
||||||
|
X-face: "xUFVDj+ZJtL_IbURmI}!~xAyPC"Mrk=MkAm&tPQnNq(FWxv49R}\>0oI8VM?O2VY+N7@F- |
||||||
|
KMLl*!h}B)u@TW|B}6<X<J|}QsVlTi:RA:O7Abc(@D2Y/"J\S,b1!<&<B/J}b.Ii9@B]H6V!+#sE0Q |
||||||
|
_+=`K$5TI|4I0-=Cp%pt~L#QYydO'iBXR~\tT?uftep9n9AF`@SzTwsw6uqJ}pL,h(cZi}T#PB"#!k |
||||||
|
p^e=Z.K~fuw$l?]lUV)?R]U}l;f*~Ol)#fpKR)Yt}XOr6BI\_Jjr0!@GMnpCTnTym4f;c{;Ms=0{`D |
||||||
|
Lq9MO6{wj%s-*N"G,g |
||||||
|
To: bugs@postgreSQL.org, hackers@postgreSQL.org |
||||||
|
Subject: [HACKERS] Failures with arrays |
||||||
|
Mime-Version: 1.0 |
||||||
|
Content-Type: text/plain; charset=us-ascii |
||||||
|
Date: Wed, 25 Nov 1998 23:48:51 +0000 |
||||||
|
From: "Oliver Elphick" <olly@lfix.co.uk> |
||||||
|
Sender: owner-pgsql-hackers@postgreSQL.org |
||||||
|
Precedence: bulk |
||||||
|
Status: ROr |
||||||
|
|
||||||
|
This was reported as a bug with the Debian package of 6.3.2; the same |
||||||
|
behaviour is still present in 6.4. |
||||||
|
|
||||||
|
bray=> create table foo ( t text[]); |
||||||
|
CREATE |
||||||
|
bray=> insert into foo values ( '{"a"}'); |
||||||
|
INSERT 201354 1 |
||||||
|
bray=> insert into foo values ( '{"a","b"}'); |
||||||
|
INSERT 201355 1 |
||||||
|
bray=> insert into foo values ( '{"a","b","c"}'); |
||||||
|
INSERT 201356 1 |
||||||
|
bray=> select * from foo; |
||||||
|
t |
||||||
|
------------- |
||||||
|
{"a"} |
||||||
|
{"a","b"} |
||||||
|
{"a","b","c"} |
||||||
|
(3 rows) |
||||||
|
|
||||||
|
bray=> select t[1] from foo; |
||||||
|
ERROR: type name lookup of t failed |
||||||
|
bray=> select * from foo; |
||||||
|
t |
||||||
|
------------- |
||||||
|
{"a"} |
||||||
|
{"a","b"} |
||||||
|
{"a","b","c"} |
||||||
|
(3 rows) |
||||||
|
|
||||||
|
bray=> select foo.t[1] from foo; |
||||||
|
t |
||||||
|
- |
||||||
|
a |
||||||
|
a |
||||||
|
a |
||||||
|
(3 rows) |
||||||
|
|
||||||
|
bray=> select count(foo.t[1]) from foo; |
||||||
|
pqReadData() -- backend closed the channel unexpectedly. |
||||||
|
|
||||||
|
-- |
||||||
|
Oliver Elphick Oliver.Elphick@lfix.co.uk |
||||||
|
Isle of Wight http://www.lfix.co.uk/oliver |
||||||
|
PGP key from public servers; key ID 32B8FAA1 |
||||||
|
======================================== |
||||||
|
"Let us therefore come boldly unto the throne of grace, |
||||||
|
that we may obtain mercy, and find grace to help in |
||||||
|
time of need." Hebrews 4:16 |
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,351 @@ |
|||||||
|
From tgl@sss.pgh.pa.us Sun Aug 30 11:25:23 1998 |
||||||
|
Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [206.210.65.6]) |
||||||
|
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id LAA12607 |
||||||
|
for <maillist@candle.pha.pa.us>; Sun, 30 Aug 1998 11:25:20 -0400 (EDT) |
||||||
|
Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) |
||||||
|
by sss.sss.pgh.pa.us (8.9.1/8.9.1) with ESMTP id LAA15788; |
||||||
|
Sun, 30 Aug 1998 11:23:38 -0400 (EDT) |
||||||
|
To: Bruce Momjian <maillist@candle.pha.pa.us> |
||||||
|
cc: dz@cs.unitn.it (Massimo Dal Zotto), hackers@postgreSQL.org |
||||||
|
Subject: Re: [HACKERS] flock patch breaks things here |
||||||
|
In-reply-to: Your message of Sun, 30 Aug 1998 08:19:52 -0400 (EDT) |
||||||
|
<199808301219.IAA08821@candle.pha.pa.us> |
||||||
|
Date: Sun, 30 Aug 1998 11:23:38 -0400 |
||||||
|
Message-ID: <15786.904490618@sss.pgh.pa.us> |
||||||
|
From: Tom Lane <tgl@sss.pgh.pa.us> |
||||||
|
Status: RO |
||||||
|
|
||||||
|
Bruce Momjian <maillist@candle.pha.pa.us> writes: |
||||||
|
> Can't we just have configure check for flock(). Another idea is to |
||||||
|
> create a 'pid' file in the pgsql/data/base directory, and do a kill -0 |
||||||
|
> to see if it is stil running before removing the lock. |
||||||
|
|
||||||
|
The latter approach is what I was going to suggest. Writing a pid file |
||||||
|
would be a fine idea anyway --- for one thing, it makes it a lot easier |
||||||
|
to write a "kill the postmaster" script. Given that the postmaster |
||||||
|
should write a pid file, a new postmaster should look for an existing |
||||||
|
pid file, and try to do a kill(pid, 0) on the number contained therein. |
||||||
|
If this doesn't return an error, then you figure there is already a |
||||||
|
postmaster running, complain, and exit. Otherwise you figure you is it, |
||||||
|
(re)write the pid file and away you go. Then pqcomm.c can just |
||||||
|
unconditionally delete any old file that's in the way of making the |
||||||
|
pipe. |
||||||
|
|
||||||
|
The pidfile checking and creation probably ought to go in postmaster.c, |
||||||
|
not down inside pqcomm.c. I never liked the fact that a critical |
||||||
|
interlock function was being done by a low-level library that one might |
||||||
|
not even want to invoke (if all your clients are using TCP, opening up |
||||||
|
the Unix-domain socket is a waste of time, no?). |
||||||
|
|
||||||
|
BTW, there is another problem with relying on flock on the socket file |
||||||
|
for this purpose: it opens up a hole for a denial-of-service attack. |
||||||
|
Anyone who can write the file can flock it. (We already had a problem |
||||||
|
with DOS via creating a dummy file at /tmp/.s.PGSQL.5432, but it would |
||||||
|
be harder to spot the culprit with an flock-based interference.) |
||||||
|
|
||||||
|
regards, tom lane |
||||||
|
|
||||||
|
From owner-pgsql-hackers@hub.org Sun Aug 30 12:27:41 1998 |
||||||
|
Received: from hub.org (hub.org [209.47.148.200]) |
||||||
|
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id MAA12976 |
||||||
|
for <maillist@candle.pha.pa.us>; Sun, 30 Aug 1998 12:27:37 -0400 (EDT) |
||||||
|
Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.7.5) with SMTP id MAA09234; Sun, 30 Aug 1998 12:24:51 -0400 (EDT) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Sun, 30 Aug 1998 12:23:26 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) by hub.org (8.8.8/8.7.5) id MAA09167 for pgsql-hackers-outgoing; Sun, 30 Aug 1998 12:23:25 -0400 (EDT) |
||||||
|
Received: from mambo.cs.unitn.it (mambo.cs.unitn.it [193.205.199.204]) by hub.org (8.8.8/8.7.5) with SMTP id MAA09150 for <hackers@postgreSQL.org>; Sun, 30 Aug 1998 12:23:08 -0400 (EDT) |
||||||
|
Received: from boogie.cs.unitn.it (dz@boogie [193.205.199.79]) by mambo.cs.unitn.it (8.6.12/8.6.12) with ESMTP id SAA29572; Sun, 30 Aug 1998 18:21:42 +0200 |
||||||
|
Received: (from dz@localhost) by boogie.cs.unitn.it (8.8.5/8.6.9) id SAA05993; Sun, 30 Aug 1998 18:21:41 +0200 |
||||||
|
From: Massimo Dal Zotto <dz@cs.unitn.it> |
||||||
|
Message-Id: <199808301621.SAA05993@boogie.cs.unitn.it> |
||||||
|
Subject: Re: [HACKERS] flock patch breaks things here |
||||||
|
To: hackers@postgreSQL.org (PostgreSQL Hackers) |
||||||
|
Date: Sun, 30 Aug 1998 18:21:41 +0200 (MET DST) |
||||||
|
Cc: tgl@sss.pgh.pa.us (Tom Lane) |
||||||
|
In-Reply-To: <15786.904490618@sss.pgh.pa.us> from "Tom Lane" at Aug 30, 98 11:23:38 am |
||||||
|
X-Mailer: ELM [version 2.4 PL24 ME4] |
||||||
|
MIME-Version: 1.0 |
||||||
|
Content-Type: text/plain; charset=iso-8859-1 |
||||||
|
Content-Transfer-Encoding: 8bit |
||||||
|
Sender: owner-pgsql-hackers@hub.org |
||||||
|
Precedence: bulk |
||||||
|
Status: ROr |
||||||
|
|
||||||
|
> |
||||||
|
> Bruce Momjian <maillist@candle.pha.pa.us> writes: |
||||||
|
> > Can't we just have configure check for flock(). Another idea is to |
||||||
|
> > create a 'pid' file in the pgsql/data/base directory, and do a kill -0 |
||||||
|
> > to see if it is stil running before removing the lock. |
||||||
|
> |
||||||
|
> The latter approach is what I was going to suggest. Writing a pid file |
||||||
|
> would be a fine idea anyway --- for one thing, it makes it a lot easier |
||||||
|
> to write a "kill the postmaster" script. Given that the postmaster |
||||||
|
> should write a pid file, a new postmaster should look for an existing |
||||||
|
> pid file, and try to do a kill(pid, 0) on the number contained therein. |
||||||
|
> If this doesn't return an error, then you figure there is already a |
||||||
|
> postmaster running, complain, and exit. Otherwise you figure you is it, |
||||||
|
> (re)write the pid file and away you go. Then pqcomm.c can just |
||||||
|
> unconditionally delete any old file that's in the way of making the |
||||||
|
> pipe. |
||||||
|
> |
||||||
|
> The pidfile checking and creation probably ought to go in postmaster.c, |
||||||
|
> not down inside pqcomm.c. I never liked the fact that a critical |
||||||
|
> interlock function was being done by a low-level library that one might |
||||||
|
> not even want to invoke (if all your clients are using TCP, opening up |
||||||
|
> the Unix-domain socket is a waste of time, no?). |
||||||
|
> |
||||||
|
> BTW, there is another problem with relying on flock on the socket file |
||||||
|
> for this purpose: it opens up a hole for a denial-of-service attack. |
||||||
|
> Anyone who can write the file can flock it. (We already had a problem |
||||||
|
> with DOS via creating a dummy file at /tmp/.s.PGSQL.5432, but it would |
||||||
|
> be harder to spot the culprit with an flock-based interference.) |
||||||
|
|
||||||
|
This came to my mind, but I didn't think this would have happened so |
||||||
|
quickly. In my opinion the socket and the pidfile should be created in a |
||||||
|
directory owned by postgres, for example /tmp/.Pgsql-unix, like does X. |
||||||
|
|
||||||
|
-- |
||||||
|
Massimo Dal Zotto |
||||||
|
|
||||||
|
+----------------------------------------------------------------------+ |
||||||
|
| Massimo Dal Zotto email: dz@cs.unitn.it | |
||||||
|
| Via Marconi, 141 phone: ++39-461-534251 | |
||||||
|
| 38057 Pergine Valsugana (TN) www: http://www.cs.unitn.it/~dz/ | |
||||||
|
| Italy pgp: finger dz@tango.cs.unitn.it | |
||||||
|
+----------------------------------------------------------------------+ |
||||||
|
|
||||||
|
|
||||||
|
From owner-pgsql-hackers@hub.org Sun Aug 30 13:01:10 1998 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id NAA13785 |
||||||
|
for <maillist@candle.pha.pa.us>; Sun, 30 Aug 1998 13:01:09 -0400 (EDT) |
||||||
|
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id MAA29386 for <maillist@candle.pha.pa.us>; Sun, 30 Aug 1998 12:58:24 -0400 (EDT) |
||||||
|
Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.7.5) with SMTP id MAA11406; Sun, 30 Aug 1998 12:54:48 -0400 (EDT) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Sun, 30 Aug 1998 12:52:22 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) by hub.org (8.8.8/8.7.5) id MAA11310 for pgsql-hackers-outgoing; Sun, 30 Aug 1998 12:52:20 -0400 (EDT) |
||||||
|
Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [206.210.65.6]) by hub.org (8.8.8/8.7.5) with ESMTP id MAA11296 for <hackers@postgreSQL.org>; Sun, 30 Aug 1998 12:52:13 -0400 (EDT) |
||||||
|
Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) |
||||||
|
by sss.sss.pgh.pa.us (8.9.1/8.9.1) with ESMTP id MAA16094; |
||||||
|
Sun, 30 Aug 1998 12:50:55 -0400 (EDT) |
||||||
|
To: Massimo Dal Zotto <dz@cs.unitn.it> |
||||||
|
cc: hackers@postgreSQL.org (PostgreSQL Hackers) |
||||||
|
Subject: Re: [HACKERS] flock patch breaks things here |
||||||
|
In-reply-to: Your message of Sun, 30 Aug 1998 18:21:41 +0200 (MET DST) |
||||||
|
<199808301621.SAA05993@boogie.cs.unitn.it> |
||||||
|
Date: Sun, 30 Aug 1998 12:50:55 -0400 |
||||||
|
Message-ID: <16092.904495855@sss.pgh.pa.us> |
||||||
|
From: Tom Lane <tgl@sss.pgh.pa.us> |
||||||
|
Sender: owner-pgsql-hackers@hub.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
Massimo Dal Zotto <dz@cs.unitn.it> writes: |
||||||
|
> In my opinion the socket and the pidfile should be created in a |
||||||
|
> directory owned by postgres, for example /tmp/.Pgsql-unix, like does X. |
||||||
|
|
||||||
|
The pidfile belongs at the top level of the database directory (eg, |
||||||
|
/usr/local/pgsql/data/postmaster.pid), because what it actually |
||||||
|
represents is that there is a postmaster running *for that database |
||||||
|
group*. |
||||||
|
|
||||||
|
If you want to support multiple database sets on one machine (which I |
||||||
|
do), then the interlock has to be per database directory. Putting the |
||||||
|
pidfile into a common directory would mean we'd have to invent some |
||||||
|
kind of pidfile naming convention to keep multiple postmasters from |
||||||
|
tromping on each other. This is unnecessarily complex. |
||||||
|
|
||||||
|
I agree with you that putting the socket file into a less easily munged |
||||||
|
directory than /tmp would be a good idea for security. But that's a |
||||||
|
separate issue. On machines that understand stickybits for directories, |
||||||
|
the security hole is not really very big. |
||||||
|
|
||||||
|
At this point, the fact that /tmp/.s.PGSQL.port# is the socket path is |
||||||
|
effectively a version-independent aspect of the FE/BE protocol, and so |
||||||
|
we can't change it without breaking old applications. I'm not sure that |
||||||
|
that's worth the security improvement. |
||||||
|
|
||||||
|
What I'd like to see someday is a postmaster command line switch to tell |
||||||
|
it to use *only* TCP connections and not create a Unix socket at all. |
||||||
|
That hasn't been possible so far, because we were relying on the socket |
||||||
|
file to provide a safety interlock against starting multiple |
||||||
|
postmasters. But an interlock using a pidfile would be much better. |
||||||
|
(Look around; *every* other Unix daemon I know of that wants to ensure |
||||||
|
that there's only one of it uses a pidfile interlock. Not file locking. |
||||||
|
There's a reason why that's the well-trodden path.) |
||||||
|
|
||||||
|
regards, tom lane |
||||||
|
|
||||||
|
|
||||||
|
From owner-pgsql-hackers@hub.org Sun Aug 30 15:31:13 1998 |
||||||
|
Received: from hub.org (hub.org [209.47.148.200]) |
||||||
|
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id PAA15275 |
||||||
|
for <maillist@candle.pha.pa.us>; Sun, 30 Aug 1998 15:31:11 -0400 (EDT) |
||||||
|
Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.7.5) with SMTP id PAA22194; Sun, 30 Aug 1998 15:27:20 -0400 (EDT) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Sun, 30 Aug 1998 15:23:58 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) by hub.org (8.8.8/8.7.5) id PAA21800 for pgsql-hackers-outgoing; Sun, 30 Aug 1998 15:23:57 -0400 (EDT) |
||||||
|
Received: from thelab.hub.org (nat0118.mpoweredpc.net [142.177.188.118]) by hub.org (8.8.8/8.7.5) with ESMTP id PAA21696 for <hackers@postgreSQL.org>; Sun, 30 Aug 1998 15:22:51 -0400 (EDT) |
||||||
|
Received: from localhost (scrappy@localhost) |
||||||
|
by thelab.hub.org (8.9.1/8.8.8) with SMTP id QAA18542; |
||||||
|
Sun, 30 Aug 1998 16:21:29 -0300 (ADT) |
||||||
|
(envelope-from scrappy@hub.org) |
||||||
|
X-Authentication-Warning: thelab.hub.org: scrappy owned process doing -bs |
||||||
|
Date: Sun, 30 Aug 1998 16:21:28 -0300 (ADT) |
||||||
|
From: The Hermit Hacker <scrappy@hub.org> |
||||||
|
To: Tom Lane <tgl@sss.pgh.pa.us> |
||||||
|
cc: Massimo Dal Zotto <dz@cs.unitn.it>, |
||||||
|
PostgreSQL Hackers <hackers@postgreSQL.org> |
||||||
|
Subject: Re: [HACKERS] flock patch breaks things here |
||||||
|
In-Reply-To: <16092.904495855@sss.pgh.pa.us> |
||||||
|
Message-ID: <Pine.BSF.4.02.9808301618350.343-100000@thelab.hub.org> |
||||||
|
MIME-Version: 1.0 |
||||||
|
Content-Type: TEXT/PLAIN; charset=US-ASCII |
||||||
|
Sender: owner-pgsql-hackers@hub.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
On Sun, 30 Aug 1998, Tom Lane wrote: |
||||||
|
|
||||||
|
> Massimo Dal Zotto <dz@cs.unitn.it> writes: |
||||||
|
> > In my opinion the socket and the pidfile should be created in a |
||||||
|
> > directory owned by postgres, for example /tmp/.Pgsql-unix, like does X. |
||||||
|
> |
||||||
|
> The pidfile belongs at the top level of the database directory (eg, |
||||||
|
> /usr/local/pgsql/data/postmaster.pid), because what it actually |
||||||
|
> represents is that there is a postmaster running *for that database |
||||||
|
> group*. |
||||||
|
|
||||||
|
I have to agree with this one...but then it also negates the |
||||||
|
argument about the flock() DoS...*grin* |
||||||
|
|
||||||
|
BTW...I like the kill(pid,0) solution myself, primarily because it |
||||||
|
is, i think, the most portable solution. |
||||||
|
|
||||||
|
I would not consider a patch to remove the flock() solution and |
||||||
|
replace it with the kill(pid,0) solution a new feature, just an |
||||||
|
improvement of an existing one...either way, moving the pid file (or |
||||||
|
socket, for that matter) from /tmp should be listed as a security related |
||||||
|
requirement for v6.4 :) |
||||||
|
|
||||||
|
Marc G. Fournier |
||||||
|
Systems Administrator @ hub.org |
||||||
|
primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org |
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
From owner-pgsql-hackers@hub.org Sun Aug 30 22:41:10 1998 |
||||||
|
Received: from hub.org (hub.org [209.47.148.200]) |
||||||
|
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id WAA01526 |
||||||
|
for <maillist@candle.pha.pa.us>; Sun, 30 Aug 1998 22:41:08 -0400 (EDT) |
||||||
|
Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.7.5) with SMTP id WAA29298; Sun, 30 Aug 1998 22:38:18 -0400 (EDT) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Sun, 30 Aug 1998 22:35:05 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) by hub.org (8.8.8/8.7.5) id WAA29203 for pgsql-hackers-outgoing; Sun, 30 Aug 1998 22:35:03 -0400 (EDT) |
||||||
|
Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [206.210.65.6]) by hub.org (8.8.8/8.7.5) with ESMTP id WAA29017 for <hackers@postgreSQL.org>; Sun, 30 Aug 1998 22:34:55 -0400 (EDT) |
||||||
|
Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) |
||||||
|
by sss.sss.pgh.pa.us (8.9.1/8.9.1) with ESMTP id WAA20075; |
||||||
|
Sun, 30 Aug 1998 22:34:41 -0400 (EDT) |
||||||
|
To: The Hermit Hacker <scrappy@hub.org> |
||||||
|
cc: PostgreSQL Hackers <hackers@postgreSQL.org> |
||||||
|
Subject: Re: [HACKERS] flock patch breaks things here |
||||||
|
In-reply-to: Your message of Sun, 30 Aug 1998 16:21:28 -0300 (ADT) |
||||||
|
<Pine.BSF.4.02.9808301618350.343-100000@thelab.hub.org> |
||||||
|
Date: Sun, 30 Aug 1998 22:34:40 -0400 |
||||||
|
Message-ID: <20073.904530880@sss.pgh.pa.us> |
||||||
|
From: Tom Lane <tgl@sss.pgh.pa.us> |
||||||
|
Sender: owner-pgsql-hackers@hub.org |
||||||
|
Precedence: bulk |
||||||
|
Status: ROr |
||||||
|
|
||||||
|
The Hermit Hacker <scrappy@hub.org> writes: |
||||||
|
> either way, moving the pid file (or |
||||||
|
> socket, for that matter) from /tmp should be listed as a security related |
||||||
|
> requirement for v6.4 :) |
||||||
|
|
||||||
|
Huh? There is no pid file being generated in /tmp (or anywhere else) |
||||||
|
at the moment. If we do add one, it should not go into /tmp for the |
||||||
|
reasons I gave before. |
||||||
|
|
||||||
|
Where the Unix-domain socket file lives is an entirely separate issue. |
||||||
|
|
||||||
|
If we move the socket out of /tmp then we have just kicked away all the |
||||||
|
work we did to preserve backwards compatibility of the FE/BE protocol |
||||||
|
with existing clients. Being able to talk to a 1.0 client isn't much |
||||||
|
good if you aren't listening where he's going to try to contact you. |
||||||
|
So I think I have to vote in favor of leaving the socket where it is. |
||||||
|
|
||||||
|
regards, tom lane |
||||||
|
|
||||||
|
|
||||||
|
From owner-pgsql-hackers@hub.org Mon Aug 31 11:31:19 1998 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id LAA21195 |
||||||
|
for <maillist@candle.pha.pa.us>; Mon, 31 Aug 1998 11:31:13 -0400 (EDT) |
||||||
|
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id LAA06827 for <maillist@candle.pha.pa.us>; Mon, 31 Aug 1998 11:17:41 -0400 (EDT) |
||||||
|
Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.7.5) with SMTP id LAA24792; Mon, 31 Aug 1998 11:12:18 -0400 (EDT) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 31 Aug 1998 11:10:31 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) by hub.org (8.8.8/8.7.5) id LAA24742 for pgsql-hackers-outgoing; Mon, 31 Aug 1998 11:10:29 -0400 (EDT) |
||||||
|
Received: from trillium.nmsu.edu (trillium.NMSU.Edu [128.123.5.15]) by hub.org (8.8.8/8.7.5) with ESMTP id LAA24725 for <hackers@postgreSQL.org>; Mon, 31 Aug 1998 11:10:22 -0400 (EDT) |
||||||
|
Received: (from brook@localhost) |
||||||
|
by trillium.nmsu.edu (8.8.8/8.8.8) id JAA03282; |
||||||
|
Mon, 31 Aug 1998 09:09:01 -0600 (MDT) |
||||||
|
Date: Mon, 31 Aug 1998 09:09:01 -0600 (MDT) |
||||||
|
Message-Id: <199808311509.JAA03282@trillium.nmsu.edu> |
||||||
|
From: Brook Milligan <brook@trillium.NMSU.Edu> |
||||||
|
To: tgl@sss.pgh.pa.us |
||||||
|
CC: dg@informix.com, hackers@postgreSQL.org |
||||||
|
In-reply-to: <23042.904573041@sss.pgh.pa.us> (message from Tom Lane on Mon, 31 |
||||||
|
Aug 1998 10:17:21 -0400) |
||||||
|
Subject: Re: [HACKERS] flock patch breaks things here |
||||||
|
References: <23042.904573041@sss.pgh.pa.us> |
||||||
|
Sender: owner-pgsql-hackers@hub.org |
||||||
|
Precedence: bulk |
||||||
|
Status: ROr |
||||||
|
|
||||||
|
I just came up with an idea that might help alleviate the /tmp security |
||||||
|
exposure without creating a backwards-compatibility problem. It works |
||||||
|
like this: |
||||||
|
|
||||||
|
1. During installation, create a subdirectory of /tmp to hold Postgres' |
||||||
|
socket files and associated pid lockfiles. This subdirectory should be |
||||||
|
owned by the Postgres superuser and have permissions 755 |
||||||
|
(world-readable, writable only by Postgres superuser). Maybe call it |
||||||
|
/tmp/.pgsql --- the name should start with a dot to keep it out of the |
||||||
|
way. (Bruce points out that some systems clear /tmp during reboot, so |
||||||
|
it might be that a postmaster will have to be prepared to recreate this |
||||||
|
directory at startup --- anyone know if subdirectories of /tmp are |
||||||
|
zapped too? My system doesn't do that...) |
||||||
|
|
||||||
|
... |
||||||
|
|
||||||
|
I notice that on my system, the X11 socket files in /tmp/.X11-unix are |
||||||
|
actually symlinks to socket files in /usr/spool/sockets/X11. I dunno if |
||||||
|
it's worth our trouble to get into putting our sockets under /usr/spool |
||||||
|
or /var/spool or whatever --- seems like another configuration choice to |
||||||
|
mess up. It'd be nice if the socket directory lived somewhere where the |
||||||
|
parent dirs weren't world-writable, but this would mean one more thing |
||||||
|
that you have to have root permissions for in order to install pgsql. |
||||||
|
|
||||||
|
It seems like we need a directory for locks (= pid files) and one for |
||||||
|
sockets (perhaps the same one). I strongly suggest that the location |
||||||
|
for these be configurable. By default, it might make sense to put |
||||||
|
them in ~pgsql/locks and ~pgsql/sockets. It is easy (i.e., I'll be |
||||||
|
glad to do it) to modify configure.in to take options like |
||||||
|
|
||||||
|
--lock-dir=/var/spool/lock |
||||||
|
--socket-dir=/var/spool/sockets |
||||||
|
|
||||||
|
that set cc defines and have the code respond accordingly. This way, |
||||||
|
those who don't care (or don't have root access) can use the defaults, |
||||||
|
whereas those with root access who like to keep locks and sockets in a |
||||||
|
common place can do so easily. Either way, multiple postmasters (all |
||||||
|
compiled with the same options of course) can check the appropriate |
||||||
|
locks in the well-known places. Finally, drop the link into /tmp for |
||||||
|
the old socket and document that it will be disappearing at some |
||||||
|
point, and all is fine. |
||||||
|
|
||||||
|
If someone wants to give me some guidance on what preprocessor |
||||||
|
variables should be set in response to the above options (or something |
||||||
|
like them), I'll do the configure stuff. |
||||||
|
|
||||||
|
Cheers, |
||||||
|
Brook |
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,69 @@ |
|||||||
|
From owner-pgsql-general@hub.org Fri Dec 18 06:31:23 1998 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id GAA05554 |
||||||
|
for <maillist@candle.pha.pa.us>; Fri, 18 Dec 1998 06:31:21 -0500 (EST) |
||||||
|
Received: from hub.org (majordom@hub.org [209.47.145.100]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id EAA21127 for <maillist@candle.pha.pa.us>; Fri, 18 Dec 1998 04:46:38 -0500 (EST) |
||||||
|
Received: from localhost (majordom@localhost) |
||||||
|
by hub.org (8.9.1/8.9.1) with SMTP id EAA01409; |
||||||
|
Fri, 18 Dec 1998 04:44:19 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-general@hub.org) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Fri, 18 Dec 1998 04:43:22 +0000 (EST) |
||||||
|
Received: (from majordom@localhost) |
||||||
|
by hub.org (8.9.1/8.9.1) id EAA01093 |
||||||
|
for pgsql-general-outgoing; Fri, 18 Dec 1998 04:43:18 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-general@postgreSQL.org) |
||||||
|
Received: from dune.krs.ru (dune.krs.ru [195.161.16.38]) |
||||||
|
by hub.org (8.9.1/8.9.1) with ESMTP id EAA01067 |
||||||
|
for <pgsql-general@postgreSQL.org>; Fri, 18 Dec 1998 04:43:09 -0500 (EST) |
||||||
|
(envelope-from vadim@krs.ru) |
||||||
|
Received: from krs.ru (localhost.krs.ru [127.0.0.1]) |
||||||
|
by dune.krs.ru (8.8.8/8.8.7) with ESMTP id QAA16201; |
||||||
|
Fri, 18 Dec 1998 16:41:44 +0700 (KRS) |
||||||
|
(envelope-from vadim@krs.ru) |
||||||
|
Message-ID: <367A2354.E998763@krs.ru> |
||||||
|
Date: Fri, 18 Dec 1998 16:41:40 +0700 |
||||||
|
From: Vadim Mikheev <vadim@krs.ru> |
||||||
|
Organization: OJSC Rostelecom (Krasnoyarsk) |
||||||
|
X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 2.2.6-RELEASE i386) |
||||||
|
X-Accept-Language: ru, en |
||||||
|
MIME-Version: 1.0 |
||||||
|
To: Anton de Wet <adw@obsidian.co.za> |
||||||
|
CC: pgsql-general@postgreSQL.org |
||||||
|
Subject: Re: [GENERAL] Why PostgreSQL is better than other commerial softwares? |
||||||
|
References: <Pine.LNX.4.04.9812181046030.9458-100000@ra.obsidian.co.za> |
||||||
|
Content-Type: text/plain; charset=us-ascii |
||||||
|
Content-Transfer-Encoding: 7bit |
||||||
|
Sender: owner-pgsql-general@postgreSQL.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
Anton de Wet wrote: |
||||||
|
> |
||||||
|
> > |
||||||
|
> > Often quick mailing list support? |
||||||
|
> |
||||||
|
> :-) |
||||||
|
> |
||||||
|
> While on the subject I finally found the solution to a problem I (and one |
||||||
|
> or two other people) posted about without answer. (So sometimes it's slow |
||||||
|
> mailing list support). |
||||||
|
> |
||||||
|
> In importing about 5 million records (which I copy in blocks of 10000) the |
||||||
|
> copy became linearly slower. After a friend RTFM and refered me, I used |
||||||
|
> the -F switch (passed by the postmaster to the backend processes) and the |
||||||
|
> time became linear and a LOT shorter. Import time for the 5000000 records |
||||||
|
> now the same (or maybe even slightly faster, I didn't accurately time |
||||||
|
> them) as importing the data into oracle on the same machine. |
||||||
|
|
||||||
|
"While on the subject..." -:) |
||||||
|
|
||||||
|
This is the problem of buffer manager, known for very long time: |
||||||
|
when copy eats all buffers, manager begins write/fsync each |
||||||
|
durty buffer to free buffer for new data. All updated relations |
||||||
|
should be fsynced _once_ @ transaction commit. You would get |
||||||
|
the same results without -F... |
||||||
|
I still have no time to implement this -:( |
||||||
|
|
||||||
|
Vadim |
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,332 @@ |
|||||||
|
From selkovjr@mcs.anl.gov Sat Jul 25 05:31:05 1998 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id FAA16564 |
||||||
|
for <maillist@candle.pha.pa.us>; Sat, 25 Jul 1998 05:31:03 -0400 (EDT) |
||||||
|
Received: from antares.mcs.anl.gov (mcs.anl.gov [140.221.9.6]) by renoir.op.net (o1/$Revision: 1.1 $) with SMTP id FAA01775 for <maillist@candle.pha.pa.us>; Sat, 25 Jul 1998 05:28:22 -0400 (EDT) |
||||||
|
Received: from mcs.anl.gov (wit.mcs.anl.gov [140.221.5.148]) by antares.mcs.anl.gov (8.6.10/8.6.10) with ESMTP |
||||||
|
id EAA28698 for <maillist@candle.pha.pa.us>; Sat, 25 Jul 1998 04:27:05 -0500 |
||||||
|
Sender: selkovjr@mcs.anl.gov |
||||||
|
Message-ID: <35B9968D.21CF60A2@mcs.anl.gov> |
||||||
|
Date: Sat, 25 Jul 1998 08:25:49 +0000 |
||||||
|
From: "Gene Selkov, Jr." <selkovjr@mcs.anl.gov> |
||||||
|
Organization: MCS, Argonne Natl. Lab |
||||||
|
X-Mailer: Mozilla 4.03 [en] (X11; I; Linux 2.0.32 i586) |
||||||
|
MIME-Version: 1.0 |
||||||
|
To: Bruce Momjian <maillist@candle.pha.pa.us> |
||||||
|
Subject: position-aware scanners |
||||||
|
References: <199807250524.BAA07296@candle.pha.pa.us> |
||||||
|
Content-Type: text/plain; charset=us-ascii |
||||||
|
Content-Transfer-Encoding: 7bit |
||||||
|
Status: RO |
||||||
|
|
||||||
|
Bruce, |
||||||
|
|
||||||
|
I attached here (trough the web links) a couple examples, totally |
||||||
|
irrelevant to postgres but good enough to discuss token locations. I |
||||||
|
might as well try to patch the backend parser, though not sure how soon. |
||||||
|
|
||||||
|
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
||||||
|
1. |
||||||
|
|
||||||
|
The first c parser I wrote, |
||||||
|
http://wit.mcs.anl.gov/~selkovjr/unit-troff.tgz, is not very |
||||||
|
sophisticated, so token locations reported by yyerr() may be slightly |
||||||
|
incorrect (+/- one position depending on the existence and type of the |
||||||
|
lookahead token. It is a filter used to typeset the units of measurement |
||||||
|
with eqn. To use it, unpack the tar file and run make. The Makefile is |
||||||
|
not too generic but I built it on various systems including linux, |
||||||
|
freebsd and sunos 4.3. The invocation can be something like this: |
||||||
|
|
||||||
|
./check 0 parse "l**3/(mmoll*min)" |
||||||
|
parse error, expecting `BASIC_UNIT' or `INTEGER' or `POSITIVE_NUMBER' or |
||||||
|
`'('' |
||||||
|
|
||||||
|
l**3/(mmoll*min) |
||||||
|
^^^^^ |
||||||
|
|
||||||
|
Now to the guts. As far as I can imagine, the only way to consistently |
||||||
|
keep track of each character read by the scanner (regardless of the |
||||||
|
length of expressions it will match) is to redefine its YY_INPUT like |
||||||
|
this: |
||||||
|
|
||||||
|
#undef YY_INPUT |
||||||
|
#define YY_INPUT(buf,result,max_size) \ |
||||||
|
{ \ |
||||||
|
int c = (int) buffer[pos++]; \ |
||||||
|
result = (c == '\0') ? YY_NULL : (buf[0] = c, 1); \ |
||||||
|
} |
||||||
|
|
||||||
|
Here, buffer is the pointer to the origin of the string being scanned |
||||||
|
and pos is a global variable, similar in usage to a file pointer (you |
||||||
|
can both read and manipulate it at will). The buffer and the pointer are |
||||||
|
initialized by the function |
||||||
|
|
||||||
|
void setString(char *s) |
||||||
|
{ |
||||||
|
buffer = s; |
||||||
|
pos = 0; |
||||||
|
} |
||||||
|
|
||||||
|
each time the new string is to be parsed. This (exportable) function is |
||||||
|
part of the interface. |
||||||
|
|
||||||
|
In this simplistic design, yyerror() is part of the scanner module and |
||||||
|
it uses the pos variable to report the location of unexpected tokens. |
||||||
|
The downside of such arrangement is that in case of error condition, you |
||||||
|
can't easily tell whether your context is current or lookahead token, it |
||||||
|
just reports the position of the last token read (be it $ (end of |
||||||
|
buffer) or something else): |
||||||
|
|
||||||
|
./check 0 convert "mol/foo" |
||||||
|
parse error, expecting `BASIC_UNIT' or `INTEGER' or `POSITIVE_NUMBER' or |
||||||
|
`'('' |
||||||
|
|
||||||
|
mol/foo |
||||||
|
^^^ |
||||||
|
|
||||||
|
(should be at the beginning of "foo") |
||||||
|
|
||||||
|
./check 0 convert "mmol//l" |
||||||
|
parse error, expecting `BASIC_UNIT' or `INTEGER' or `POSITIVE_NUMBER' or |
||||||
|
`'('' |
||||||
|
|
||||||
|
mmol//l |
||||||
|
^ |
||||||
|
|
||||||
|
(should be at the second '/') |
||||||
|
|
||||||
|
|
||||||
|
I believe this is why most simple parsers made with yacc would report |
||||||
|
parse errors being "at or near" some token, which is fair enough if the |
||||||
|
expression is not too complex. |
||||||
|
|
||||||
|
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
||||||
|
2. The second version of the same scanner, |
||||||
|
http://wit.mcs.anl.gov/~selkovjr/scanner-example.tgz, addresses this |
||||||
|
problem by recording exact locations of the tokens in each instance of |
||||||
|
the token semantic data structure. The global, |
||||||
|
|
||||||
|
UNIT_YYSTYPE unit_yylval; |
||||||
|
|
||||||
|
would be normally used to export the token semantics (including its |
||||||
|
original or modified text and location data) to the parser. |
||||||
|
Unfortunately, I cannot show you the parser part in c, because that's |
||||||
|
about when I stopped writing parsers in c. Instead, I included a small |
||||||
|
test program, test.c, that mimics the parser's expectations for the |
||||||
|
scanner data pretty well. I am assuming here that you are not interested |
||||||
|
in digging someone else's ugly guts for relatively small bit of |
||||||
|
information; let me know if I am wrong and I will send you the complete |
||||||
|
perl code (also generated with bison). |
||||||
|
|
||||||
|
To run this example, unpack the tar file and run Make. Then do |
||||||
|
|
||||||
|
gcc test.c scanner.o |
||||||
|
|
||||||
|
and run a.out |
||||||
|
|
||||||
|
Note the line |
||||||
|
|
||||||
|
yylval = unit_getyylval(); |
||||||
|
|
||||||
|
in test.c. You will not normally need it in a c parser. It is enough to |
||||||
|
define yylval as an external variable and link it to yylval in yylex() |
||||||
|
|
||||||
|
In the bison-generated parser, yylval gets pushed into a stack (pointed |
||||||
|
to by yylsp) each time a new token is read. For each syntax rule, the |
||||||
|
bison macros @1, @2, ... are just shortcuts to locations in the stack 1, |
||||||
|
2, ... levels deep. In following code fragment, @3 refers to the |
||||||
|
location info for the third term in the rule (INTEGER): |
||||||
|
|
||||||
|
(sorry about perl, but I think you can do the same things in c without |
||||||
|
significant changes to your existing parser) |
||||||
|
|
||||||
|
term: base { |
||||||
|
$$ = $1; |
||||||
|
$$->{'order'} = 1; |
||||||
|
} |
||||||
|
| base EXP INTEGER { |
||||||
|
$$ = $1; |
||||||
|
$$->{'order'} = @3->{'text'}; |
||||||
|
$$->{'scale'} = $$->{'scale'} ** $$->{'order'}; |
||||||
|
if ( $$->{'order'} == 0 ) { |
||||||
|
yyerror("Error: expecting a non-zero |
||||||
|
integer exponent"); |
||||||
|
YYERROR; |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
|
||||||
|
which translates to: |
||||||
|
|
||||||
|
($yyn == 10) && do { |
||||||
|
$yyval = $yyvsa[-1]; |
||||||
|
$yyval->{'order'} = 1; |
||||||
|
last SWITCH; |
||||||
|
}; |
||||||
|
|
||||||
|
($yyn == 11) && do { |
||||||
|
$yyval = $yyvsa[-3]; |
||||||
|
$yyval->{'order'} = $yylsa[-1]->{'text'} |
||||||
|
$yyval->{'scale'} = $yyval->{'scale'} ** $yyval->{'order'}; |
||||||
|
if ( $yyval->{'order'} == 0 ) { |
||||||
|
yyerror("Error: expecting a non-zero integer |
||||||
|
exponent"); |
||||||
|
goto yyerrlab1 ; |
||||||
|
} |
||||||
|
last SWITCH; |
||||||
|
}; |
||||||
|
|
||||||
|
In c, you will have a bit more complicated pointer arithmetic to adress |
||||||
|
the stack, but the usage of objects will be the same. Note here that it |
||||||
|
is convenient to keep all information about the token in its location |
||||||
|
info, (yylsa, yylsp, yylval, @n), while everything relating to the value |
||||||
|
of the expression, or to the parse tree, is better placed in the |
||||||
|
semantic stack (yyssa, yyssp, yysval, $n). Also note that in some cases |
||||||
|
you can do semantic checks inside rules and report useful messages |
||||||
|
before or instead of invoking yyerror(); |
||||||
|
|
||||||
|
Finally, it is useful to make the following wrapper function around |
||||||
|
external yylex() in order to maintain your own token stack. Unlike the |
||||||
|
parser's internal stack which is only as deep as the rule being reduced, |
||||||
|
this one can hold all tokens recognized during the current run, and that |
||||||
|
can be extremely helpful for error reporting and any transformations you |
||||||
|
may need. In this way, you can even scan (tokenize) the whole buffer |
||||||
|
before handing it off to the parser (who knows, you may need a token |
||||||
|
ahead of what is currently seen by the parser): |
||||||
|
|
||||||
|
|
||||||
|
sub tokenize { |
||||||
|
undef @tokenTable; |
||||||
|
my ($tok, $text, $name, $unit, $first_line, $first_column, |
||||||
|
$last_line, $last_column); |
||||||
|
|
||||||
|
while ( ($tok = &UnitLex::yylex()) > 0 ) { # this is where the |
||||||
|
c-coded yylex is called, |
||||||
|
# UnitLex is the perl |
||||||
|
extension encapsulating it |
||||||
|
( $text, $name, $unit, $first_line, $first_column, $last_line, |
||||||
|
$last_column ) = &UnitLex::getyylval; |
||||||
|
push(@tokenTable, |
||||||
|
Unit::yyltype->new ( |
||||||
|
'token' => $tok, |
||||||
|
'text' => $text, |
||||||
|
'name' => $name, |
||||||
|
'unit' => $unit, |
||||||
|
'first_line' => $first_line, |
||||||
|
'first_column' => $first_column, |
||||||
|
'last_line' => $last_line, |
||||||
|
'last_column' => $last_column, |
||||||
|
) |
||||||
|
) |
||||||
|
} |
||||||
|
|
||||||
|
} |
||||||
|
|
||||||
|
|
||||||
|
It is now a lot easier to handle various state-related problems, such as |
||||||
|
backtracking and error reporting. The yylex() function as seen by the |
||||||
|
parser might be constructed somewhat like this: |
||||||
|
|
||||||
|
sub yylex { |
||||||
|
$yylloc = $tokenTable[$tokenNo]; # $tokenNo is a global; now |
||||||
|
instead of a "file pointer", |
||||||
|
# as in the first example, we have |
||||||
|
a "token pointer" |
||||||
|
undef $yylval; |
||||||
|
|
||||||
|
|
||||||
|
# disregard this; name this block "computing semantic values" |
||||||
|
if ( $yylloc->{'token'} == UNIT) { |
||||||
|
$yylval = Unit::Operand->new( |
||||||
|
'unit' => Unit::Dict::unit($yylloc->{'unit'}), |
||||||
|
'base' => Unit::Dict::base($yylloc->{'unit'}), |
||||||
|
'scale' => Unit::Dict::scale($yylloc->{'unit'}), |
||||||
|
'scaleToBase' => Unit::Dict::scaleToBase($yylloc->{'unit'}), |
||||||
|
'loc' => $yylloc, |
||||||
|
); |
||||||
|
} |
||||||
|
elsif ( ($yylloc->{'token'} == INTEGER ) || ($yylloc->{'token'} == |
||||||
|
POSITIVE_NUMBER) ) { |
||||||
|
$yylval = Unit::Operand->new( |
||||||
|
'unit' => '1', |
||||||
|
'base' => '1', |
||||||
|
'scale' => 1, |
||||||
|
'scaleToBase' => 1, |
||||||
|
'loc' => $yylloc, |
||||||
|
); |
||||||
|
} |
||||||
|
|
||||||
|
$tokenNo++; |
||||||
|
return(%{$yylloc}->{'token'}); # This is all the parser needs to |
||||||
|
know about this token. |
||||||
|
# But we already made sure we saved |
||||||
|
everything we need to know. |
||||||
|
} |
||||||
|
|
||||||
|
|
||||||
|
Now the most interesting part, the error reporting routine: |
||||||
|
|
||||||
|
|
||||||
|
sub yyerror { |
||||||
|
my ($str) = @_; |
||||||
|
my ($message, $start, $end, $loc); |
||||||
|
|
||||||
|
$loc = $tokenTable[$tokenNo-1]; # This is the same as to say, |
||||||
|
# "obtain the location info for the |
||||||
|
current token" |
||||||
|
|
||||||
|
# You may use this routine for your own purposes or let parser use |
||||||
|
it |
||||||
|
if( $str ne 'parse error' ) { |
||||||
|
$message = "$str instead of `" . $loc->{'name'} . "' <" . |
||||||
|
$loc->{'text'} . ">, at line " . $loc->{'first_line'} . ":\n\ |
||||||
|
n"; |
||||||
|
} |
||||||
|
else { |
||||||
|
$message = "unexpected token `" . $loc->{'name'} . "' <" . |
||||||
|
$loc->{'text'} . ">, at line " . loc->{'first_line'} . ":\n |
||||||
|
\n"; |
||||||
|
} |
||||||
|
|
||||||
|
$message .= $parseBuffer . "\n"; # that's the original string that |
||||||
|
was used to set the parser buffer |
||||||
|
|
||||||
|
$message .= ( ' ' x ($loc->{'first_column'} + 1) ) . ( '^' x |
||||||
|
length($loc->{'text'}) ). "\n"; |
||||||
|
if( $str ne 'parse error' ) { |
||||||
|
print STDERR "$str instead of `", $loc->{'name'}, "' {", |
||||||
|
$loc->{'text'}, "}, at line ", $loc->{'first_line'}, ":\n\n"; |
||||||
|
} |
||||||
|
else { |
||||||
|
print STDERR "unexpected token `", $loc->{'name'}, "' {", |
||||||
|
$loc->{'text'}, "}, at line ", $loc->{'first_line'}, ":\n\n"; |
||||||
|
} |
||||||
|
|
||||||
|
print STDERR "$parseBuffer\n"; |
||||||
|
print STDERR ' ' x ($loc->{'first_column'} + 1), '^' x |
||||||
|
length($loc->{'text'}), "\n"; |
||||||
|
} |
||||||
|
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
||||||
|
|
||||||
|
Scanners used in these examples assume there is a single line of text on |
||||||
|
the input (the first_line and last_line elements of yylloc are simply |
||||||
|
ignored). If you want to be able to parse multi-line buffers, just add a |
||||||
|
lex rule for '\n' that will increment the line count and reset the pos |
||||||
|
variable to zero. |
||||||
|
|
||||||
|
|
||||||
|
Ugly as it may seem, I find this approach extremely liberating. If the |
||||||
|
grammar becomes too complicated for a LALR(1) parser, I can cascade |
||||||
|
multiple parsers. The token table can then be used to reassemble parts |
||||||
|
of original expression for subordinate parsers, preserving the location |
||||||
|
info all the way down, so that subordinate parsers can report their |
||||||
|
problems consistently. You probably don't need this, as SQL is very well |
||||||
|
thought of and has parsable grammar. But it may be of some help, for |
||||||
|
error reporting. |
||||||
|
|
||||||
|
|
||||||
|
--Gene |
||||||
|
|
||||||
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,207 @@ |
|||||||
|
From owner-pgsql-hackers@hub.org Fri Nov 13 13:24:37 1998 |
||||||
|
Received: from hub.org (majordom@hub.org [209.47.148.200]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA13457 |
||||||
|
for <maillist@candle.pha.pa.us>; Fri, 13 Nov 1998 13:24:35 -0500 (EST) |
||||||
|
Received: from localhost (majordom@localhost) |
||||||
|
by hub.org (8.9.1/8.9.1) with SMTP id NAA02464; |
||||||
|
Fri, 13 Nov 1998 13:22:52 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@hub.org) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Fri, 13 Nov 1998 13:21:14 +0000 (EST) |
||||||
|
Received: (from majordom@localhost) |
||||||
|
by hub.org (8.9.1/8.9.1) id NAA02331 |
||||||
|
for pgsql-hackers-outgoing; Fri, 13 Nov 1998 13:21:12 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||||
|
Received: from orion.SAPserv.Hamburg.dsh.de (Tpolaris2.sapham.debis.de [53.2.131.8]) |
||||||
|
by hub.org (8.9.1/8.9.1) with SMTP id NAA02316 |
||||||
|
for <pgsql-hackers@postgreSQL.org>; Fri, 13 Nov 1998 13:21:06 -0500 (EST) |
||||||
|
(envelope-from wieck@sapserv.debis.de) |
||||||
|
Received: by orion.SAPserv.Hamburg.dsh.de |
||||||
|
for pgsql-hackers@postgreSQL.org |
||||||
|
id m0zeOEf-000EBPC; Fri, 13 Nov 98 19:46 MET |
||||||
|
Message-Id: <m0zeOEf-000EBPC@orion.SAPserv.Hamburg.dsh.de> |
||||||
|
From: jwieck@debis.com (Jan Wieck) |
||||||
|
Subject: [HACKERS] shmem limits and redolog |
||||||
|
To: pgsql-hackers@postgreSQL.org (PostgreSQL HACKERS) |
||||||
|
Date: Fri, 13 Nov 1998 19:46:20 +0100 (MET) |
||||||
|
Reply-To: jwieck@debis.com (Jan Wieck) |
||||||
|
X-Mailer: ELM [version 2.4 PL25] |
||||||
|
Content-Type: text |
||||||
|
Sender: owner-pgsql-hackers@postgreSQL.org |
||||||
|
Precedence: bulk |
||||||
|
Status: ROr |
||||||
|
|
||||||
|
Hi, |
||||||
|
|
||||||
|
I'm currently hacking around on a solution for logging all |
||||||
|
database operations at query level that can recover a crashed |
||||||
|
database from the last successful backup by redoing all the |
||||||
|
commands. |
||||||
|
|
||||||
|
Well, I wanted it to be as flexible as can. So I decided to |
||||||
|
make it per database configurable. One could say which |
||||||
|
databases are logged and if a database is, if it is logged |
||||||
|
sync or async (in sync mode, every COMMIT forces an fsync of |
||||||
|
the actual logfile and controlfiles). |
||||||
|
|
||||||
|
To make async mode as fast as can, I'm using a shared memory |
||||||
|
of 32K per database (not per backend) that is used as a wrap |
||||||
|
around buffer from the backends to place their query |
||||||
|
information. So the log writer can fall a little behind if |
||||||
|
there are many backends doing different things that don't |
||||||
|
lock each other. |
||||||
|
|
||||||
|
Now I'm a little in doubt about the shared memory limits |
||||||
|
reported. Was it a good decision to use shared memory? Am I |
||||||
|
better off using socket's? |
||||||
|
|
||||||
|
The bad thing in what I have up to now (it's far from |
||||||
|
complete) is, that even if a database isn't currently logged, |
||||||
|
a redolog writer is started and creates the 32K shmem segment |
||||||
|
(plus a semaphore set with 5 semaphores). This is because I |
||||||
|
plan to create commands like |
||||||
|
|
||||||
|
ALTER DATABASE LOG MODE=ASYNC LOGDIR='/somewhere/dbname'; |
||||||
|
|
||||||
|
and the like that can be used at runtime (while more than one |
||||||
|
backend is connected to the database) to turn logging on/off, |
||||||
|
switch to/from backup mode (all other activity is stopped) |
||||||
|
etc. |
||||||
|
|
||||||
|
So every 32 databases will require another megabyte of shared |
||||||
|
memory. The logging master controls which databases have |
||||||
|
activity and kills redolog writers after some time of |
||||||
|
inactivity, and the shmem is freed then. But it can hurt if |
||||||
|
someone really has many many databases that are all used at |
||||||
|
the same time. |
||||||
|
|
||||||
|
What do the others say? |
||||||
|
|
||||||
|
|
||||||
|
Jan |
||||||
|
|
||||||
|
-- |
||||||
|
|
||||||
|
#======================================================================# |
||||||
|
# It's easier to get forgiveness for being wrong than for being right. # |
||||||
|
# Let's break this rule - forgive me. # |
||||||
|
#======================================== jwieck@debis.com (Jan Wieck) # |
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
From owner-pgsql-hackers@hub.org Wed Dec 16 15:46:41 1998 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id PAA00521 |
||||||
|
for <maillist@candle.pha.pa.us>; Wed, 16 Dec 1998 15:46:40 -0500 (EST) |
||||||
|
Received: from hub.org (majordom@hub.org [209.47.145.100]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id PAA08772 for <maillist@candle.pha.pa.us>; Wed, 16 Dec 1998 15:10:01 -0500 (EST) |
||||||
|
Received: from localhost (majordom@localhost) |
||||||
|
by hub.org (8.9.1/8.9.1) with SMTP id PAA01254; |
||||||
|
Wed, 16 Dec 1998 15:06:56 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@hub.org) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 16 Dec 1998 14:58:11 +0000 (EST) |
||||||
|
Received: (from majordom@localhost) |
||||||
|
by hub.org (8.9.1/8.9.1) id OAA00660 |
||||||
|
for pgsql-hackers-outgoing; Wed, 16 Dec 1998 14:58:10 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||||
|
Received: from orion.SAPserv.Hamburg.dsh.de (Tpolaris2.sapham.debis.de [53.2.131.8]) |
||||||
|
by hub.org (8.9.1/8.9.1) with SMTP id OAA00643 |
||||||
|
for <pgsql-hackers@postgreSQL.org>; Wed, 16 Dec 1998 14:58:05 -0500 (EST) |
||||||
|
(envelope-from wieck@sapserv.debis.de) |
||||||
|
Received: by orion.SAPserv.Hamburg.dsh.de |
||||||
|
for pgsql-hackers@postgreSQL.org |
||||||
|
id m0zqNDo-000EBTC; Wed, 16 Dec 98 21:07 MET |
||||||
|
Message-Id: <m0zqNDo-000EBTC@orion.SAPserv.Hamburg.dsh.de> |
||||||
|
From: jwieck@debis.com (Jan Wieck) |
||||||
|
Subject: Re: [HACKERS] redolog - for discussion |
||||||
|
To: vadim@krs.ru (Vadim Mikheev) |
||||||
|
Date: Wed, 16 Dec 1998 21:07:00 +0100 (MET) |
||||||
|
Cc: jwieck@debis.com, pgsql-hackers@postgreSQL.org |
||||||
|
Reply-To: jwieck@debis.com (Jan Wieck) |
||||||
|
In-Reply-To: <3677B71D.C67462B3@krs.ru> from "Vadim Mikheev" at Dec 16, 98 08:35:25 pm |
||||||
|
X-Mailer: ELM [version 2.4 PL25] |
||||||
|
Content-Type: text |
||||||
|
Sender: owner-pgsql-hackers@postgreSQL.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
Vadim wrote: |
||||||
|
|
||||||
|
> |
||||||
|
> Jan Wieck wrote: |
||||||
|
> > |
||||||
|
> > RECOVER DATABASE {ALL | UNTIL 'datetime' | RESET}; |
||||||
|
> > |
||||||
|
> ... |
||||||
|
> > |
||||||
|
> > For the others, the backend starts the recovery program |
||||||
|
> > which reads the redolog files, establishes database |
||||||
|
> > connections as required and reruns all the commands in |
||||||
|
> ^^^^^^^^^^^^^^^^^^^^^^^^^^ |
||||||
|
> > them. If a required logfile isn't found, it tells the |
||||||
|
> ^^^^^ |
||||||
|
> |
||||||
|
> I foresee problems with using _commands_ logging for |
||||||
|
> recovery/replication -:(( |
||||||
|
> |
||||||
|
> Let's consider two concurrent updates in READ COMMITTED mode: |
||||||
|
> |
||||||
|
> update test set x = 2 where y = 1; |
||||||
|
> |
||||||
|
> and |
||||||
|
> |
||||||
|
> update test set x = 3 where y = 1; |
||||||
|
> |
||||||
|
> The result of both committed transaction will be x = 2 |
||||||
|
> if the 1st transaction updated row _after_ 2nd transaction |
||||||
|
> and x = 3 if the 2nd transaction gets row after 1st one. |
||||||
|
> Order of updates is not defined by order in which commands |
||||||
|
> begun and so order in which commands should be rerun |
||||||
|
> will be unknown... |
||||||
|
|
||||||
|
Yepp, the order in which commands begun is absolutely not of |
||||||
|
interest. Locking could already delay the execution of one |
||||||
|
command until another one started later has finished and |
||||||
|
released the lock. It's a classic race condition. |
||||||
|
|
||||||
|
Thus, my plan was to log the queries just before the call to |
||||||
|
CommitTransactionCommand() in tcop. This has the advantage, |
||||||
|
that queries which bail out with errors don't get into the |
||||||
|
log at all and must not get rerun. And I can set a static |
||||||
|
flag to false before starting the command, which is set to |
||||||
|
true in the buffer manager when a buffer is written (marked |
||||||
|
dirty), so filtering out queries that do no updates at all is |
||||||
|
easy. |
||||||
|
|
||||||
|
Unfortunately query level logging get's hit by the current |
||||||
|
implementation of sequence numbers. If a query that get's |
||||||
|
aborted somewhere in the middle (maybe by a trigger) called |
||||||
|
nextval() for rows processed earlier, the sequence number |
||||||
|
isn't advanced at recovery time, because the query is |
||||||
|
suppressed at all. And sequences aren't locked, so for |
||||||
|
concurrently running queries getting numbers from the same |
||||||
|
sequence, the results aren't reproduceable. If some |
||||||
|
application selects a value resulting from a sequence and |
||||||
|
uses that later in another query, how could the redolog know |
||||||
|
that this has changed? It's a Const in the query logged, and |
||||||
|
all that corrupts the whole thing. |
||||||
|
|
||||||
|
All that is painful and I don't see another solution yet than |
||||||
|
to hook into nextval(), log out the numbers generated in |
||||||
|
normal operation and getting back the same numbers in redo |
||||||
|
mode. |
||||||
|
|
||||||
|
The whole thing gets more and more complicated :-( |
||||||
|
|
||||||
|
|
||||||
|
Jan |
||||||
|
|
||||||
|
-- |
||||||
|
|
||||||
|
#======================================================================# |
||||||
|
# It's easier to get forgiveness for being wrong than for being right. # |
||||||
|
# Let's break this rule - forgive me. # |
||||||
|
#======================================== jwieck@debis.com (Jan Wieck) # |
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,119 @@ |
|||||||
|
From owner-pgsql-general@hub.org Fri Oct 9 18:22:09 1998 |
||||||
|
Received: from hub.org (majordom@hub.org [209.47.148.200]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id SAA04220 |
||||||
|
for <maillist@candle.pha.pa.us>; Fri, 9 Oct 1998 18:22:08 -0400 (EDT) |
||||||
|
Received: from localhost (majordom@localhost) |
||||||
|
by hub.org (8.8.8/8.8.8) with SMTP id SAA26960; |
||||||
|
Fri, 9 Oct 1998 18:18:29 -0400 (EDT) |
||||||
|
(envelope-from owner-pgsql-general@hub.org) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Fri, 09 Oct 1998 18:18:07 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) |
||||||
|
by hub.org (8.8.8/8.8.8) id SAA26917 |
||||||
|
for pgsql-general-outgoing; Fri, 9 Oct 1998 18:18:04 -0400 (EDT) |
||||||
|
(envelope-from owner-pgsql-general@postgreSQL.org) |
||||||
|
X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-general@postgreSQL.org using -f |
||||||
|
Received: from gecko.statsol.com (gecko.statsol.com [198.11.51.133]) |
||||||
|
by hub.org (8.8.8/8.8.8) with ESMTP id SAA26904 |
||||||
|
for <pgsql-general@postgresql.org>; Fri, 9 Oct 1998 18:17:46 -0400 (EDT) |
||||||
|
(envelope-from statsol@statsol.com) |
||||||
|
Received: from gecko (gecko [198.11.51.133]) |
||||||
|
by gecko.statsol.com (8.9.0/8.9.0) with SMTP id SAA00557 |
||||||
|
for <pgsql-general@postgresql.org>; Fri, 9 Oct 1998 18:18:00 -0400 (EDT) |
||||||
|
Date: Fri, 9 Oct 1998 18:18:00 -0400 (EDT) |
||||||
|
From: Steve Doliov <statsol@statsol.com> |
||||||
|
X-Sender: statsol@gecko |
||||||
|
To: pgsql-general@postgreSQL.org |
||||||
|
Subject: Re: [GENERAL] Making NULLs visible. |
||||||
|
Message-ID: <Pine.GSO.3.96.981009181716.545B-100000@gecko> |
||||||
|
MIME-Version: 1.0 |
||||||
|
Content-Type: TEXT/PLAIN; charset=US-ASCII |
||||||
|
Sender: owner-pgsql-general@postgreSQL.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
On Fri, 9 Oct 1998, Bruce Momjian wrote: |
||||||
|
|
||||||
|
> [Charset iso-8859-1 unsupported, filtering to ASCII...] |
||||||
|
> > > Yes, \ always outputs as \\, excepts someone changed it last week, and I |
||||||
|
> > > am requesting a reversal. Do you like the \N if it is unique? |
||||||
|
> > |
||||||
|
> > Well, it's certainly clear, but could be confused with \n (newline). Can we |
||||||
|
> > have \0 instead? |
||||||
|
> |
||||||
|
> Yes, but it is uppercase. \0 looks like an octal number to me, and I |
||||||
|
> think we even output octals sometimes, don't we? |
||||||
|
> |
||||||
|
|
||||||
|
my first suggestion may have been hare-brained, but why not just make the |
||||||
|
specifics of the output user-configurable. So if the user chooses \0, so |
||||||
|
be it, if the user chooses \N so be it, if the user likes NULL so be it. |
||||||
|
but the option would only have one value per database at any given point |
||||||
|
in time. so database x could use \N on tuesday and NULL on wednesday, but |
||||||
|
database x could never have two references to the characters(s) used to |
||||||
|
represent a null value. |
||||||
|
|
||||||
|
steve |
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
From owner-pgsql-general@hub.org Sun Oct 11 17:31:08 1998 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id RAA20043 |
||||||
|
for <maillist@candle.pha.pa.us>; Sun, 11 Oct 1998 17:31:02 -0400 (EDT) |
||||||
|
Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id RAA03069 for <maillist@candle.pha.pa.us>; Sun, 11 Oct 1998 17:10:34 -0400 (EDT) |
||||||
|
Received: from localhost (majordom@localhost) |
||||||
|
by hub.org (8.8.8/8.8.8) with SMTP id QAA10856; |
||||||
|
Sun, 11 Oct 1998 16:57:34 -0400 (EDT) |
||||||
|
(envelope-from owner-pgsql-general@hub.org) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Sun, 11 Oct 1998 16:53:35 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) |
||||||
|
by hub.org (8.8.8/8.8.8) id QAA10393 |
||||||
|
for pgsql-general-outgoing; Sun, 11 Oct 1998 16:53:34 -0400 (EDT) |
||||||
|
(envelope-from owner-pgsql-general@postgreSQL.org) |
||||||
|
X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-general@postgreSQL.org using -f |
||||||
|
Received: from mail1.panix.com (mail1.panix.com [166.84.0.212]) |
||||||
|
by hub.org (8.8.8/8.8.8) with ESMTP id QAA10378 |
||||||
|
for <pgsql-general@postgreSQL.org>; Sun, 11 Oct 1998 16:53:28 -0400 (EDT) |
||||||
|
(envelope-from tomg@admin.nrnet.org) |
||||||
|
Received: from mailhost.nrnet.org (root@mailhost.nrnet.org [166.84.192.39]) |
||||||
|
by mail1.panix.com (8.8.8/8.8.8/PanixM1.3) with ESMTP id QAA16311 |
||||||
|
for <pgsql-general@postgreSQL.org>; Sun, 11 Oct 1998 16:53:24 -0400 (EDT) |
||||||
|
Received: from admin.nrnet.org (uucp@localhost) |
||||||
|
by mailhost.nrnet.org (8.8.7/8.8.4) with UUCP |
||||||
|
id QAA16345 for pgsql-general@postgreSQL.org; Sun, 11 Oct 1998 16:28:47 -0400 |
||||||
|
Received: from localhost (tomg@localhost) |
||||||
|
by admin.nrnet.org (8.8.7/8.8.7) with SMTP id QAA11569 |
||||||
|
for <pgsql-general@postgreSQL.org>; Sun, 11 Oct 1998 16:28:41 -0400 |
||||||
|
Date: Sun, 11 Oct 1998 16:28:41 -0400 (EDT) |
||||||
|
From: Thomas Good <tomg@admin.nrnet.org> |
||||||
|
To: pgsql-general@postgreSQL.org |
||||||
|
Subject: Re: [GENERAL] Making NULLs visible. |
||||||
|
In-Reply-To: <Pine.GSO.3.96.981009181716.545B-100000@gecko> |
||||||
|
Message-ID: <Pine.LNX.3.96.981011161908.11556A-100000@admin.nrnet.org> |
||||||
|
MIME-Version: 1.0 |
||||||
|
Content-Type: TEXT/PLAIN; charset=US-ASCII |
||||||
|
Sender: owner-pgsql-general@postgreSQL.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
Watching all this go by...as a guy who has to move alot of data |
||||||
|
from legacy dbs to postgres, I've gotten used to \N being a null. |
||||||
|
|
||||||
|
My vote, if I were allowed to cast one, would be to have one null |
||||||
|
and that would be the COPY command null. I have no difficulty |
||||||
|
distinguishing a null from a newline... |
||||||
|
|
||||||
|
At the pgsql command prompt I would find seeing \N rather reassuring. |
||||||
|
I've seen alot of these little guys. |
||||||
|
|
||||||
|
---------- Sisters of Charity Medical Center ---------- |
||||||
|
Department of Psychiatry |
||||||
|
---- |
||||||
|
Thomas Good <tomg@q8.nrnet.org> |
||||||
|
Coordinator, North Richmond C.M.H.C. Information Systems |
||||||
|
75 Vanderbilt Ave, Quarters 8 Phone: 718-354-5528 |
||||||
|
Staten Island, NY 10304 Fax: 718-354-5056 |
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,987 @@ |
|||||||
|
From owner-pgsql-hackers@hub.org Mon Mar 22 18:43:41 1999 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id SAA23978 |
||||||
|
for <maillist@candle.pha.pa.us>; Mon, 22 Mar 1999 18:43:39 -0500 (EST) |
||||||
|
Received: from hub.org (majordom@hub.org [209.47.145.100]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id SAA06472 for <maillist@candle.pha.pa.us>; Mon, 22 Mar 1999 18:36:44 -0500 (EST) |
||||||
|
Received: from localhost (majordom@localhost) |
||||||
|
by hub.org (8.9.2/8.9.1) with SMTP id SAA92604; |
||||||
|
Mon, 22 Mar 1999 18:34:23 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@hub.org) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 22 Mar 1999 18:33:50 +0000 (EST) |
||||||
|
Received: (from majordom@localhost) |
||||||
|
by hub.org (8.9.2/8.9.1) id SAA92469 |
||||||
|
for pgsql-hackers-outgoing; Mon, 22 Mar 1999 18:33:47 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||||
|
Received: from po8.andrew.cmu.edu (PO8.ANDREW.CMU.EDU [128.2.10.108]) |
||||||
|
by hub.org (8.9.2/8.9.1) with ESMTP id SAA92456 |
||||||
|
for <pgsql-hackers@postgresql.org>; Mon, 22 Mar 1999 18:33:41 -0500 (EST) |
||||||
|
(envelope-from er1p+@andrew.cmu.edu) |
||||||
|
Received: (from postman@localhost) by po8.andrew.cmu.edu (8.8.5/8.8.2) id SAA12894 for pgsql-hackers@postgresql.org; Mon, 22 Mar 1999 18:33:38 -0500 (EST) |
||||||
|
Received: via switchmail; Mon, 22 Mar 1999 18:33:38 -0500 (EST) |
||||||
|
Received: from cloudy.me.cmu.edu via qmail |
||||||
|
ID </afs/andrew.cmu.edu/service/mailqs/q007/QF.Aqxh7Lu00gNtQ0TZE5>; |
||||||
|
Mon, 22 Mar 1999 18:27:20 -0500 (EST) |
||||||
|
Received: from cloudy.me.cmu.edu via qmail |
||||||
|
ID </afs/andrew.cmu.edu/usr2/er1p/.Outgoing/QF.Uqxh7JS00gNtMmTJFk>; |
||||||
|
Mon, 22 Mar 1999 18:27:17 -0500 (EST) |
||||||
|
Received: from mms.4.60.Jun.27.1996.03.05.56.sun4.41.EzMail.2.0.CUILIB.3.45.SNAP.NOT.LINKED.cloudy.me.cmu.edu.sun4m.412 |
||||||
|
via MS.5.6.cloudy.me.cmu.edu.sun4_41; |
||||||
|
Mon, 22 Mar 1999 18:27:15 -0500 (EST) |
||||||
|
Message-ID: <sqxh7H_00gNtAmTJ5Q@andrew.cmu.edu> |
||||||
|
Date: Mon, 22 Mar 1999 18:27:15 -0500 (EST) |
||||||
|
From: Erik Riedel <riedel+@CMU.EDU> |
||||||
|
To: pgsql-hackers@postgreSQL.org |
||||||
|
Subject: [HACKERS] optimizer and type question |
||||||
|
Sender: owner-pgsql-hackers@postgreSQL.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
|
||||||
|
[last week aggregation, this week, the optimizer] |
||||||
|
|
||||||
|
I have a somewhat general optimizer question/problem that I would like |
||||||
|
to get some input on - i.e. I'd like to know what is "supposed" to |
||||||
|
work here and what I should be expecting. Sadly, I think the patch |
||||||
|
for this is more involved than my last message. |
||||||
|
|
||||||
|
Using my favorite table these days: |
||||||
|
|
||||||
|
Table = lineitem |
||||||
|
+------------------------+----------------------------------+-------+ |
||||||
|
| Field | Type | Length| |
||||||
|
+------------------------+----------------------------------+-------+ |
||||||
|
| l_orderkey | int4 not null | 4 | |
||||||
|
| l_partkey | int4 not null | 4 | |
||||||
|
| l_suppkey | int4 not null | 4 | |
||||||
|
| l_linenumber | int4 not null | 4 | |
||||||
|
| l_quantity | float4 not null | 4 | |
||||||
|
| l_extendedprice | float4 not null | 4 | |
||||||
|
| l_discount | float4 not null | 4 | |
||||||
|
| l_tax | float4 not null | 4 | |
||||||
|
| l_returnflag | char() not null | 1 | |
||||||
|
| l_linestatus | char() not null | 1 | |
||||||
|
| l_shipdate | date | 4 | |
||||||
|
| l_commitdate | date | 4 | |
||||||
|
| l_receiptdate | date | 4 | |
||||||
|
| l_shipinstruct | char() not null | 25 | |
||||||
|
| l_shipmode | char() not null | 10 | |
||||||
|
| l_comment | char() not null | 44 | |
||||||
|
+------------------------+----------------------------------+-------+ |
||||||
|
Index: lineitem_index_ |
||||||
|
|
||||||
|
and the query: |
||||||
|
|
||||||
|
-- |
||||||
|
-- Query 1 |
||||||
|
-- |
||||||
|
explain select l_returnflag, l_linestatus, sum(l_quantity) as sum_qty, |
||||||
|
sum(l_extendedprice) as sum_base_price, |
||||||
|
sum(l_extendedprice*(1-l_discount)) as sum_disc_price, |
||||||
|
sum(l_extendedprice*(1-l_discount)*(1+l_tax)) as sum_charge, |
||||||
|
avg(l_quantity) as avg_qty, avg(l_extendedprice) as avg_price, |
||||||
|
avg(l_discount) as avg_disc, count(*) as count_order |
||||||
|
from lineitem |
||||||
|
where l_shipdate <= '1998-09-02'::date |
||||||
|
group by l_returnflag, l_linestatus |
||||||
|
order by l_returnflag, l_linestatus; |
||||||
|
|
||||||
|
|
||||||
|
note that I have eliminated the date calculation in my query of last |
||||||
|
week and manually replaced it with a constant (since this wasn't |
||||||
|
happening automatically - but let's not worry about that for now). |
||||||
|
And this is only an explain, we care about the optimizer. So we get: |
||||||
|
|
||||||
|
Sort (cost=34467.88 size=0 width=0) |
||||||
|
-> Aggregate (cost=34467.88 size=0 width=0) |
||||||
|
-> Group (cost=34467.88 size=0 width=0) |
||||||
|
-> Sort (cost=34467.88 size=0 width=0) |
||||||
|
-> Seq Scan on lineitem (cost=34467.88 size=200191 width=44) |
||||||
|
|
||||||
|
so let's think about the selectivity that is being chosen for the |
||||||
|
seq scan (the where l_shipdate <= '1998-09-02'). |
||||||
|
|
||||||
|
Turns out the optimizer is choosing "33%", even though the real answer |
||||||
|
is somewhere in 90+% (that's how the query is designed). So, why does |
||||||
|
it do that? |
||||||
|
|
||||||
|
Turns out that selectivity in this case is determined via |
||||||
|
plancat::restriction_selectivity() which calls into functionOID = 103 |
||||||
|
(intltsel) for operatorOID = 1096 (date "<=") on relation OID = 18663 |
||||||
|
(my lineitem). |
||||||
|
|
||||||
|
This all follows because of the description of 1096 (date "<=") in |
||||||
|
pg_operator. Looking at local1_template1.bki.source near line 1754 |
||||||
|
shows: |
||||||
|
|
||||||
|
insert OID = 1096 ( "<=" PGUID 0 <...> date_le intltsel intltjoinsel ) |
||||||
|
|
||||||
|
where we see that indeed, it thinks "intltsel" is the right function |
||||||
|
to use for "oprrest" in the case of dates. |
||||||
|
|
||||||
|
Question 1 - is intltsel the right thing for selectivity on dates? |
||||||
|
|
||||||
|
Hope someone is still with me. |
||||||
|
|
||||||
|
So now we're running selfuncs::intltsel() where we make a further call |
||||||
|
to selfuncs::gethilokey(). The job of gethilokey is to determine the |
||||||
|
min and max values of a particular attribute in the table, which will |
||||||
|
then be used with the constant in my where clause to estimate the |
||||||
|
selectivity. It is going to search the pg_statistic relation with |
||||||
|
three key values: |
||||||
|
|
||||||
|
Anum_pg_statistic_starelid 18663 (lineitem) |
||||||
|
Anum_pg_statistic_staattnum 11 (l_shipdate) |
||||||
|
Anum_pg_statistic_staop 1096 (date "<=") |
||||||
|
|
||||||
|
this finds no tuples in pg_statistic. Why is that? The only nearby |
||||||
|
tuple in pg_statistic is: |
||||||
|
|
||||||
|
starelid|staattnum|staop|stalokey |stahikey |
||||||
|
--------+---------+-----+----------------+---------------- |
||||||
|
18663| 11| 0|01-02-1992 |12-01-1998 |
||||||
|
|
||||||
|
and the reason the query doesn't match anything? Because 1096 != 0. |
||||||
|
But why is it 0 in pg_statistic? Statistics are determined near line |
||||||
|
1844 in vacuum.c (assuming a 'vacuum analyze' run at some point) |
||||||
|
|
||||||
|
i = 0; |
||||||
|
values[i++] = (Datum) relid; /* 1 */ |
||||||
|
values[i++] = (Datum) attp->attnum; /* 2 */ |
||||||
|
====> values[i++] = (Datum) InvalidOid; /* 3 */ |
||||||
|
fmgr_info(stats->outfunc, &out_function); |
||||||
|
out_string = <...min...> |
||||||
|
values[i++] = (Datum) fmgr(F_TEXTIN, out_string); |
||||||
|
pfree(out_string); |
||||||
|
out_string = <...max...> |
||||||
|
values[i++] = (Datum) fmgr(F_TEXTIN, out_string); |
||||||
|
pfree(out_string); |
||||||
|
stup = heap_formtuple(sd->rd_att, values, nulls); |
||||||
|
|
||||||
|
the "offending" line is setting the staop to InvalidOid (i.e. 0). |
||||||
|
|
||||||
|
Question 2 - is this right? Is the intent for 0 to serve as a |
||||||
|
"wildcard", or should it be inserting an entry for each operation |
||||||
|
individually? |
||||||
|
|
||||||
|
In the case of "wildcard" then gethilokey() should allow a match for |
||||||
|
|
||||||
|
Anum_pg_statistic_staop 0 |
||||||
|
|
||||||
|
instead of requiring the more restrictive 1096. In the current code, |
||||||
|
what happens next is gethilokey() returns "not found" and intltsel() |
||||||
|
returns the default 1/3 which I see in the resultant query plan (size |
||||||
|
= 200191 is 1/3 of the number of lineitem tuples). |
||||||
|
|
||||||
|
Question 3 - is there any inherent reason it couldn't get this right? |
||||||
|
The statistic is in the table 1992 to 1998, so the '1998-09-02' date |
||||||
|
should be 90-some% selectivity, a much better guess than 33%. |
||||||
|
|
||||||
|
Doesn't make a difference for this particular query, of course, |
||||||
|
because the seq scan must proceed anyhow, but it could easily affect |
||||||
|
other queries where selectivities matter (and it affects the |
||||||
|
modifications I am trying to test in the optimizer to be "smarter" |
||||||
|
about selectivities - my overall context is to understand/improve the |
||||||
|
behavior that the underlying storage system sees from queries like this). |
||||||
|
|
||||||
|
OK, so let's say we treat 0 as a "wildcard" and stop checking for |
||||||
|
1096. Not we let gethilokey() return the two dates from the statistic |
||||||
|
table. The immediate next thing that intltsel() does, near lines 122 |
||||||
|
in selfuncs.c is call atol() on the strings from gethilokey(). And |
||||||
|
guess what it comes up with? |
||||||
|
|
||||||
|
low = 1 |
||||||
|
high = 12 |
||||||
|
|
||||||
|
because it calls atol() on '01-02-1992' and '12-01-1998'. This |
||||||
|
clearly isn't right, it should get some large integer that includes |
||||||
|
the year and day in the result. Then it should compare reasonably |
||||||
|
with my constant from the where clause and give a decent selectivity |
||||||
|
value. This leads to a re-visit of Question 1. |
||||||
|
|
||||||
|
Question 4 - should date "<=" use a dateltsel() function instead of |
||||||
|
intltsel() as oprrest? |
||||||
|
|
||||||
|
If anyone is still with me, could you tell me if this makes sense, or |
||||||
|
if there is some other location where the appropriate type conversion |
||||||
|
could take place so that intltsel() gets something reasonable when it |
||||||
|
does the atol() calls? |
||||||
|
|
||||||
|
Could someone also give me a sense for how far out-of-whack the whole |
||||||
|
current selectivity-handling structure is? It seems that most of the |
||||||
|
operators in pg_operator actually use intltsel() and would have |
||||||
|
type-specific problems like that described. Or is the problem in the |
||||||
|
way attribute values are stored in pg_statistic by vacuum analyze? Or |
||||||
|
is there another layer where type conversion belongs? |
||||||
|
|
||||||
|
Phew. Enough typing, hope someone can follow this and address at |
||||||
|
least some of the questions. |
||||||
|
|
||||||
|
Thanks. |
||||||
|
|
||||||
|
Erik Riedel |
||||||
|
Carnegie Mellon University |
||||||
|
www.cs.cmu.edu/~riedel |
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
From owner-pgsql-hackers@hub.org Mon Mar 22 20:31:11 1999 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id UAA00802 |
||||||
|
for <maillist@candle.pha.pa.us>; Mon, 22 Mar 1999 20:31:09 -0500 (EST) |
||||||
|
Received: from hub.org (majordom@hub.org [209.47.145.100]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id UAA13231 for <maillist@candle.pha.pa.us>; Mon, 22 Mar 1999 20:15:20 -0500 (EST) |
||||||
|
Received: from localhost (majordom@localhost) |
||||||
|
by hub.org (8.9.2/8.9.1) with SMTP id UAA01981; |
||||||
|
Mon, 22 Mar 1999 20:14:04 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@hub.org) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 22 Mar 1999 20:13:32 +0000 (EST) |
||||||
|
Received: (from majordom@localhost) |
||||||
|
by hub.org (8.9.2/8.9.1) id UAA01835 |
||||||
|
for pgsql-hackers-outgoing; Mon, 22 Mar 1999 20:13:28 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||||
|
Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [206.210.65.6]) |
||||||
|
by hub.org (8.9.2/8.9.1) with ESMTP id UAA01822 |
||||||
|
for <pgsql-hackers@postgreSQL.org>; Mon, 22 Mar 1999 20:13:21 -0500 (EST) |
||||||
|
(envelope-from tgl@sss.pgh.pa.us) |
||||||
|
Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) |
||||||
|
by sss.sss.pgh.pa.us (8.9.1/8.9.1) with ESMTP id UAA23294; |
||||||
|
Mon, 22 Mar 1999 20:12:43 -0500 (EST) |
||||||
|
To: Erik Riedel <riedel+@CMU.EDU> |
||||||
|
cc: pgsql-hackers@postgreSQL.org |
||||||
|
Subject: Re: [HACKERS] optimizer and type question |
||||||
|
In-reply-to: Your message of Mon, 22 Mar 1999 18:27:15 -0500 (EST) |
||||||
|
<sqxh7H_00gNtAmTJ5Q@andrew.cmu.edu> |
||||||
|
Date: Mon, 22 Mar 1999 20:12:43 -0500 |
||||||
|
Message-ID: <23292.922151563@sss.pgh.pa.us> |
||||||
|
From: Tom Lane <tgl@sss.pgh.pa.us> |
||||||
|
Sender: owner-pgsql-hackers@postgreSQL.org |
||||||
|
Precedence: bulk |
||||||
|
Status: ROr |
||||||
|
|
||||||
|
Erik Riedel <riedel+@CMU.EDU> writes: |
||||||
|
> [ optimizer doesn't find relevant pg_statistic entry ] |
||||||
|
|
||||||
|
It's clearly a bug that the selectivity code is not finding this tuple. |
||||||
|
If your analysis is correct, then selectivity estimation has *never* |
||||||
|
worked properly, or at least not in recent memory :-(. Yipes. |
||||||
|
Bruce and I found a bunch of other problems in the optimizer recently, |
||||||
|
so it doesn't faze me to assume that this is broken too. |
||||||
|
|
||||||
|
> the "offending" line is setting the staop to InvalidOid (i.e. 0). |
||||||
|
> Question 2 - is this right? Is the intent for 0 to serve as a |
||||||
|
> "wildcard", |
||||||
|
|
||||||
|
My thought is that what the staop column ought to be is the OID of the |
||||||
|
comparison function that was used to determine the sort order of the |
||||||
|
column. Without a sort op the lowest and highest keys in the column are |
||||||
|
not well defined, so it makes no sense to assert "these are the lowest |
||||||
|
and highest values" without providing the sort op that determined that. |
||||||
|
(For sufficiently complex data types one could reasonably have multiple |
||||||
|
ordering operators. A crude example is sorting on "circumference" and |
||||||
|
"area" for polygons.) But typically the sort op will be the "<" |
||||||
|
operator for the column data type. |
||||||
|
|
||||||
|
So, the vacuum code is definitely broken --- it's not storing the sort |
||||||
|
op that it used. The code in gethilokey might be broken too, depending |
||||||
|
on how it is producing the operator it's trying to match against the |
||||||
|
tuple. For example, if the actual operator in the query is any of |
||||||
|
< <= > >= on int4, then int4lt ought to be used to probe the pg_statistic |
||||||
|
table. I'm not sure if we have adequate info in pg_operator or pg_type |
||||||
|
to let the optimizer code determine the right thing to probe with :-( |
||||||
|
|
||||||
|
> The immediate next thing that intltsel() does, near lines 122 |
||||||
|
> in selfuncs.c is call atol() on the strings from gethilokey(). And |
||||||
|
> guess what it comes up with? |
||||||
|
> low = 1 |
||||||
|
> high = 12 |
||||||
|
> because it calls atol() on '01-02-1992' and '12-01-1998'. This |
||||||
|
> clearly isn't right, it should get some large integer that includes |
||||||
|
> the year and day in the result. Then it should compare reasonably |
||||||
|
> with my constant from the where clause and give a decent selectivity |
||||||
|
> value. This leads to a re-visit of Question 1. |
||||||
|
> Question 4 - should date "<=" use a dateltsel() function instead of |
||||||
|
> intltsel() as oprrest? |
||||||
|
|
||||||
|
This is clearly busted as well. I'm not sure that creating dateltsel() |
||||||
|
is the right fix, however, because if you go down that path then every |
||||||
|
single datatype needs its own selectivity function; that's more than we |
||||||
|
need. |
||||||
|
|
||||||
|
What we really want here is to be able to map datatype values into |
||||||
|
some sort of numeric range so that we can compute what fraction of the |
||||||
|
low-key-to-high-key range is on each side of the probe value (the |
||||||
|
constant taken from the query). This general concept will apply to |
||||||
|
many scalar types, so what we want is a type-specific mapping function |
||||||
|
and a less-specific fraction-computing-function. Offhand I'd say that |
||||||
|
we want intltsel() and floatltsel(), plus conversion routines that can |
||||||
|
produce either int4 or float8 from a data type as seems appropriate. |
||||||
|
Anything that couldn't map to one or the other would have to supply its |
||||||
|
own selectivity function. |
||||||
|
|
||||||
|
> Or is the problem in the |
||||||
|
> way attribute values are stored in pg_statistic by vacuum analyze? |
||||||
|
|
||||||
|
Looks like it converts the low and high values to text and stores them |
||||||
|
that way. Ugly as can be :-( but I'm not sure there is a good |
||||||
|
alternative. We have no "wild card" column type AFAIK, which is what |
||||||
|
these columns of pg_statistic would have to be to allow storage of |
||||||
|
unconverted min and max values. |
||||||
|
|
||||||
|
I think you've found a can of worms here. Congratulations ;-) |
||||||
|
|
||||||
|
regards, tom lane |
||||||
|
|
||||||
|
|
||||||
|
From owner-pgsql-hackers@hub.org Mon Mar 22 23:31:00 1999 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id XAA03384 |
||||||
|
for <maillist@candle.pha.pa.us>; Mon, 22 Mar 1999 23:30:58 -0500 (EST) |
||||||
|
Received: from hub.org (majordom@hub.org [209.47.145.100]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id XAA25586 for <maillist@candle.pha.pa.us>; Mon, 22 Mar 1999 23:18:25 -0500 (EST) |
||||||
|
Received: from localhost (majordom@localhost) |
||||||
|
by hub.org (8.9.2/8.9.1) with SMTP id XAA17955; |
||||||
|
Mon, 22 Mar 1999 23:17:24 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@hub.org) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 22 Mar 1999 23:16:49 +0000 (EST) |
||||||
|
Received: (from majordom@localhost) |
||||||
|
by hub.org (8.9.2/8.9.1) id XAA17764 |
||||||
|
for pgsql-hackers-outgoing; Mon, 22 Mar 1999 23:16:46 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||||
|
Received: from po8.andrew.cmu.edu (PO8.ANDREW.CMU.EDU [128.2.10.108]) |
||||||
|
by hub.org (8.9.2/8.9.1) with ESMTP id XAA17745 |
||||||
|
for <pgsql-hackers@postgreSQL.org>; Mon, 22 Mar 1999 23:16:39 -0500 (EST) |
||||||
|
(envelope-from er1p+@andrew.cmu.edu) |
||||||
|
Received: (from postman@localhost) by po8.andrew.cmu.edu (8.8.5/8.8.2) id XAA04273; Mon, 22 Mar 1999 23:16:37 -0500 (EST) |
||||||
|
Received: via switchmail; Mon, 22 Mar 1999 23:16:37 -0500 (EST) |
||||||
|
Received: from hazy.adsl.net.cmu.edu via qmail |
||||||
|
ID </afs/andrew.cmu.edu/service/mailqs/q000/QF.kqxlJ:S00anI00p040>; |
||||||
|
Mon, 22 Mar 1999 23:15:09 -0500 (EST) |
||||||
|
Received: from hazy.adsl.net.cmu.edu via qmail |
||||||
|
ID </afs/andrew.cmu.edu/usr2/er1p/.Outgoing/QF.MqxlJ3q00anI01hKE0>; |
||||||
|
Mon, 22 Mar 1999 23:15:00 -0500 (EST) |
||||||
|
Received: from mms.4.60.Jun.27.1996.03.02.53.sun4.51.EzMail.2.0.CUILIB.3.45.SNAP.NOT.LINKED.hazy.adsl.net.cmu.edu.sun4m.54 |
||||||
|
via MS.5.6.hazy.adsl.net.cmu.edu.sun4_51; |
||||||
|
Mon, 22 Mar 1999 23:14:55 -0500 (EST) |
||||||
|
Message-ID: <4qxlJ0200anI01hK40@andrew.cmu.edu> |
||||||
|
Date: Mon, 22 Mar 1999 23:14:55 -0500 (EST) |
||||||
|
From: Erik Riedel <riedel+@CMU.EDU> |
||||||
|
To: Tom Lane <tgl@sss.pgh.pa.us> |
||||||
|
Subject: Re: [HACKERS] optimizer and type question |
||||||
|
Cc: pgsql-hackers@postgreSQL.org |
||||||
|
In-Reply-To: <23292.922151563@sss.pgh.pa.us> |
||||||
|
References: <23292.922151563@sss.pgh.pa.us> |
||||||
|
Sender: owner-pgsql-hackers@postgreSQL.org |
||||||
|
Precedence: bulk |
||||||
|
Status: ROr |
||||||
|
|
||||||
|
|
||||||
|
OK, building on your high-level explanation, I am attaching a patch that |
||||||
|
attempts to do something "better" than the current code. Note that I |
||||||
|
have only tested this with the date type and my particular query. I |
||||||
|
haven't run it through the regression, so consider it "proof of concept" |
||||||
|
at best. Although hopefully it will serve my purposes. |
||||||
|
|
||||||
|
> My thought is that what the staop column ought to be is the OID of the |
||||||
|
> comparison function that was used to determine the sort order of the |
||||||
|
> column. Without a sort op the lowest and highest keys in the column are |
||||||
|
> not well defined, so it makes no sense to assert "these are the lowest |
||||||
|
> and highest values" without providing the sort op that determined that. |
||||||
|
> |
||||||
|
> (For sufficiently complex data types one could reasonably have multiple |
||||||
|
> ordering operators. A crude example is sorting on "circumference" and |
||||||
|
> "area" for polygons.) But typically the sort op will be the "<" |
||||||
|
> operator for the column data type. |
||||||
|
> |
||||||
|
I changed vacuum.c to do exactly that. oid of the lt sort op. |
||||||
|
|
||||||
|
> So, the vacuum code is definitely broken --- it's not storing the sort |
||||||
|
> op that it used. The code in gethilokey might be broken too, depending |
||||||
|
> on how it is producing the operator it's trying to match against the |
||||||
|
> tuple. For example, if the actual operator in the query is any of |
||||||
|
> < <= > >= on int4, then int4lt ought to be used to probe the pg_statistic |
||||||
|
> table. I'm not sure if we have adequate info in pg_operator or pg_type |
||||||
|
> to let the optimizer code determine the right thing to probe with :-( |
||||||
|
> |
||||||
|
This indeed seems like a bigger problem. I thought about somehow using |
||||||
|
type-matching from the sort op and the actual operator in the query - if |
||||||
|
both the left and right type match, then consider them the same for |
||||||
|
purposes of this probe. That seemed complicated, so I punted in my |
||||||
|
example - it just does the search with relid and attnum and assumes that |
||||||
|
only returns one tuple. This works in my case (maybe in all cases, |
||||||
|
because of the way vacuum is currently written - ?). |
||||||
|
|
||||||
|
> What we really want here is to be able to map datatype values into |
||||||
|
> some sort of numeric range so that we can compute what fraction of the |
||||||
|
> low-key-to-high-key range is on each side of the probe value (the |
||||||
|
> constant taken from the query). This general concept will apply to |
||||||
|
> many scalar types, so what we want is a type-specific mapping function |
||||||
|
> and a less-specific fraction-computing-function. Offhand I'd say that |
||||||
|
> we want intltsel() and floatltsel(), plus conversion routines that can |
||||||
|
> produce either int4 or float8 from a data type as seems appropriate. |
||||||
|
> Anything that couldn't map to one or the other would have to supply its |
||||||
|
> own selectivity function. |
||||||
|
> |
||||||
|
This is what my example then does. Uses the stored sort op to get the |
||||||
|
type and then uses typinput to convert from the string to an int4. |
||||||
|
|
||||||
|
Then puts the int4 back into string format because that's what everyone |
||||||
|
was expecting. |
||||||
|
|
||||||
|
It seems to work for my particular query. I now get: |
||||||
|
|
||||||
|
(selfuncs) gethilokey() obj 18663 attr 11 opid 1096 (ignored) |
||||||
|
(selfuncs) gethilokey() found op 1087 in pg_proc |
||||||
|
(selfuncs) gethilokey() found type 1082 in pg_type |
||||||
|
(selfuncs) gethilokey() going to use 1084 to convert type 1082 |
||||||
|
(selfuncs) gethilokey() have low -2921 high -396 |
||||||
|
(selfuncs) intltsel() high -396 low -2921 val -486 |
||||||
|
(plancat) restriction_selectivity() for func 103 op 1096 rel 18663 attr |
||||||
|
11 const -486 flag 3 returns 0.964356 |
||||||
|
NOTICE: QUERY PLAN: |
||||||
|
|
||||||
|
Sort (cost=34467.88 size=0 width=0) |
||||||
|
-> Aggregate (cost=34467.88 size=0 width=0) |
||||||
|
-> Group (cost=34467.88 size=0 width=0) |
||||||
|
-> Sort (cost=34467.88 size=0 width=0) |
||||||
|
-> Seq Scan on lineitem (cost=34467.88 size=579166 width=44) |
||||||
|
|
||||||
|
including my printfs, which exist in the patch as well. |
||||||
|
|
||||||
|
Selectivity is now the expected 96% and the size estimate for the seq |
||||||
|
scan is much closer to correct. |
||||||
|
|
||||||
|
Again, not tested with anything besides date, so caveat not-tested. |
||||||
|
|
||||||
|
Hope this helps. |
||||||
|
|
||||||
|
Erik |
||||||
|
|
||||||
|
----------------------[optimizer_fix.sh]------------------------ |
||||||
|
|
||||||
|
#! /bin/sh |
||||||
|
# This is a shell archive, meaning: |
||||||
|
# 1. Remove everything above the #! /bin/sh line. |
||||||
|
# 2. Save the resulting text in a file. |
||||||
|
# 3. Execute the file with /bin/sh (not csh) to create: |
||||||
|
# selfuncs.c.diff |
||||||
|
# vacuum.c.diff |
||||||
|
# This archive created: Mon Mar 22 22:58:14 1999 |
||||||
|
export PATH; PATH=/bin:/usr/bin:$PATH |
||||||
|
if test -f 'selfuncs.c.diff' |
||||||
|
then |
||||||
|
echo shar: "will not over-write existing file 'selfuncs.c.diff'" |
||||||
|
else |
||||||
|
cat << \SHAR_EOF > 'selfuncs.c.diff' |
||||||
|
*** |
||||||
|
/afs/ece.cmu.edu/project/lcs/lcs-004/er1p/postgres/611/src/backend/utils/adt |
||||||
|
/selfuncs.c Thu Mar 11 23:59:35 1999 |
||||||
|
--- |
||||||
|
/afs/ece.cmu.edu/project/lcs/lcs-004/er1p/postgres/615/src/backend/utils/adt |
||||||
|
/selfuncs.c Mon Mar 22 22:57:25 1999 |
||||||
|
*************** |
||||||
|
*** 32,37 **** |
||||||
|
--- 32,40 ---- |
||||||
|
#include "utils/lsyscache.h" /* for get_oprrest() */ |
||||||
|
#include "catalog/pg_statistic.h" |
||||||
|
|
||||||
|
+ #include "catalog/pg_proc.h" /* for Form_pg_proc */ |
||||||
|
+ #include "catalog/pg_type.h" /* for Form_pg_type */ |
||||||
|
+ |
||||||
|
/* N is not a valid var/constant or relation id */ |
||||||
|
#define NONVALUE(N) ((N) == -1) |
||||||
|
|
||||||
|
*************** |
||||||
|
*** 103,110 **** |
||||||
|
bottom; |
||||||
|
|
||||||
|
result = (float64) palloc(sizeof(float64data)); |
||||||
|
! if (NONVALUE(attno) || NONVALUE(relid)) |
||||||
|
*result = 1.0 / 3; |
||||||
|
else |
||||||
|
{ |
||||||
|
/* XXX val = atol(value); */ |
||||||
|
--- 106,114 ---- |
||||||
|
bottom; |
||||||
|
|
||||||
|
result = (float64) palloc(sizeof(float64data)); |
||||||
|
! if (NONVALUE(attno) || NONVALUE(relid)) { |
||||||
|
*result = 1.0 / 3; |
||||||
|
+ } |
||||||
|
else |
||||||
|
{ |
||||||
|
/* XXX val = atol(value); */ |
||||||
|
*************** |
||||||
|
*** 117,130 **** |
||||||
|
} |
||||||
|
high = atol(highchar); |
||||||
|
low = atol(lowchar); |
||||||
|
if ((flag & SEL_RIGHT && val < low) || |
||||||
|
(!(flag & SEL_RIGHT) && val > high)) |
||||||
|
{ |
||||||
|
float32data nvals; |
||||||
|
|
||||||
|
nvals = getattdisbursion(relid, (int) attno); |
||||||
|
! if (nvals == 0) |
||||||
|
*result = 1.0 / 3.0; |
||||||
|
else |
||||||
|
{ |
||||||
|
*result = 3.0 * (float64data) nvals; |
||||||
|
--- 121,136 ---- |
||||||
|
} |
||||||
|
high = atol(highchar); |
||||||
|
low = atol(lowchar); |
||||||
|
+ printf("(selfuncs) intltsel() high %d low %d val %d\n",high,low,val); |
||||||
|
if ((flag & SEL_RIGHT && val < low) || |
||||||
|
(!(flag & SEL_RIGHT) && val > high)) |
||||||
|
{ |
||||||
|
float32data nvals; |
||||||
|
|
||||||
|
nvals = getattdisbursion(relid, (int) attno); |
||||||
|
! if (nvals == 0) { |
||||||
|
*result = 1.0 / 3.0; |
||||||
|
+ } |
||||||
|
else |
||||||
|
{ |
||||||
|
*result = 3.0 * (float64data) nvals; |
||||||
|
*************** |
||||||
|
*** 336,341 **** |
||||||
|
--- 342,353 ---- |
||||||
|
{ |
||||||
|
Relation rel; |
||||||
|
HeapScanDesc scan; |
||||||
|
+ /* this assumes there is only one row in the statistics table for any |
||||||
|
particular */ |
||||||
|
+ /* relid, attnum pair - could be more complicated if staop is also |
||||||
|
used. */ |
||||||
|
+ /* at the moment, if there are multiple rows, this code ends up |
||||||
|
picking the */ |
||||||
|
+ /* "first" one |
||||||
|
- er1p */ |
||||||
|
+ /* the actual "ignoring" is done in the call to heap_beginscan() |
||||||
|
below, where */ |
||||||
|
+ /* we only mention 2 of the 3 keys in this array |
||||||
|
- er1p */ |
||||||
|
static ScanKeyData key[3] = { |
||||||
|
{0, Anum_pg_statistic_starelid, F_OIDEQ, {0, 0, F_OIDEQ}}, |
||||||
|
{0, Anum_pg_statistic_staattnum, F_INT2EQ, {0, 0, F_INT2EQ}}, |
||||||
|
*************** |
||||||
|
*** 344,355 **** |
||||||
|
bool isnull; |
||||||
|
HeapTuple tuple; |
||||||
|
|
||||||
|
rel = heap_openr(StatisticRelationName); |
||||||
|
|
||||||
|
key[0].sk_argument = ObjectIdGetDatum(relid); |
||||||
|
key[1].sk_argument = Int16GetDatum((int16) attnum); |
||||||
|
key[2].sk_argument = ObjectIdGetDatum(opid); |
||||||
|
! scan = heap_beginscan(rel, 0, SnapshotNow, 3, key); |
||||||
|
tuple = heap_getnext(scan, 0); |
||||||
|
if (!HeapTupleIsValid(tuple)) |
||||||
|
{ |
||||||
|
--- 356,377 ---- |
||||||
|
bool isnull; |
||||||
|
HeapTuple tuple; |
||||||
|
|
||||||
|
+ HeapTuple tup; |
||||||
|
+ Form_pg_proc proc; |
||||||
|
+ Form_pg_type typ; |
||||||
|
+ Oid which_op; |
||||||
|
+ Oid which_type; |
||||||
|
+ int32 low_value; |
||||||
|
+ int32 high_value; |
||||||
|
+ |
||||||
|
rel = heap_openr(StatisticRelationName); |
||||||
|
|
||||||
|
key[0].sk_argument = ObjectIdGetDatum(relid); |
||||||
|
key[1].sk_argument = Int16GetDatum((int16) attnum); |
||||||
|
key[2].sk_argument = ObjectIdGetDatum(opid); |
||||||
|
! printf("(selfuncs) gethilokey() obj %d attr %d opid %d (ignored)\n", |
||||||
|
! key[0].sk_argument,key[1].sk_argument,key[2].sk_argument); |
||||||
|
! scan = heap_beginscan(rel, 0, SnapshotNow, 2, key); |
||||||
|
tuple = heap_getnext(scan, 0); |
||||||
|
if (!HeapTupleIsValid(tuple)) |
||||||
|
{ |
||||||
|
*************** |
||||||
|
*** 376,383 **** |
||||||
|
--- 398,461 ---- |
||||||
|
&isnull)); |
||||||
|
if (isnull) |
||||||
|
elog(DEBUG, "gethilokey: low key is null"); |
||||||
|
+ |
||||||
|
heap_endscan(scan); |
||||||
|
heap_close(rel); |
||||||
|
+ |
||||||
|
+ /* now we deal with type conversion issues |
||||||
|
*/ |
||||||
|
+ /* when intltsel() calls this routine (who knows what other callers |
||||||
|
might do) */ |
||||||
|
+ /* it assumes that it can call atol() on the strings and then use |
||||||
|
integer */ |
||||||
|
+ /* comparison from there. what we are going to do here, then, is try |
||||||
|
to use */ |
||||||
|
+ /* the type information from Anum_pg_statistic_staop to convert the |
||||||
|
high */ |
||||||
|
+ /* and low values |
||||||
|
- er1p */ |
||||||
|
+ |
||||||
|
+ /* WARNING: this code has only been tested with the date type and has |
||||||
|
NOT */ |
||||||
|
+ /* been regression tested. consider it "sample" code of what might |
||||||
|
be the */ |
||||||
|
+ /* right kind of thing to do |
||||||
|
- er1p */ |
||||||
|
+ |
||||||
|
+ /* get the 'op' from pg_statistic and look it up in pg_proc */ |
||||||
|
+ which_op = heap_getattr(tuple, |
||||||
|
+ Anum_pg_statistic_staop, |
||||||
|
+ RelationGetDescr(rel), |
||||||
|
+ &isnull); |
||||||
|
+ if (InvalidOid == which_op) { |
||||||
|
+ /* ignore all this stuff, try conversion only if we have a valid staop */ |
||||||
|
+ /* note that there is an accompanying change to 'vacuum analyze' that */ |
||||||
|
+ /* gets this set to something useful. */ |
||||||
|
+ } else { |
||||||
|
+ /* staop looks valid, so let's see what we can do about conversion */ |
||||||
|
+ tup = SearchSysCacheTuple(PROOID, ObjectIdGetDatum(which_op), 0, 0, 0); |
||||||
|
+ if (!HeapTupleIsValid(tup)) { |
||||||
|
+ elog(ERROR, "selfuncs: unable to find op in pg_proc %d", which_op); |
||||||
|
+ } |
||||||
|
+ printf("(selfuncs) gethilokey() found op %d in pg_proc\n",which_op); |
||||||
|
+ |
||||||
|
+ /* use that to determine the type of stahikey and stalokey via pg_type */ |
||||||
|
+ proc = (Form_pg_proc) GETSTRUCT(tup); |
||||||
|
+ which_type = proc->proargtypes[0]; /* XXX - use left and right |
||||||
|
separately? */ |
||||||
|
+ tup = SearchSysCacheTuple(TYPOID, ObjectIdGetDatum(which_type), 0, 0, 0); |
||||||
|
+ if (!HeapTupleIsValid(tup)) { |
||||||
|
+ elog(ERROR, "selfuncs: unable to find type in pg_type %d", which_type); |
||||||
|
+ } |
||||||
|
+ printf("(selfuncs) gethilokey() found type %d in pg_type\n",which_type); |
||||||
|
+ |
||||||
|
+ /* and use that type to get the conversion function to int4 */ |
||||||
|
+ typ = (Form_pg_type) GETSTRUCT(tup); |
||||||
|
+ printf("(selfuncs) gethilokey() going to use %d to convert type |
||||||
|
%d\n",typ->typinput,which_type); |
||||||
|
+ |
||||||
|
+ /* and convert the low and high strings */ |
||||||
|
+ low_value = (int32) fmgr(typ->typinput, *low, -1); |
||||||
|
+ high_value = (int32) fmgr(typ->typinput, *high, -1); |
||||||
|
+ printf("(selfuncs) gethilokey() have low %d high |
||||||
|
%d\n",low_value,high_value); |
||||||
|
+ |
||||||
|
+ /* now we have int4's, which we put back into strings because |
||||||
|
that's what out */ |
||||||
|
+ /* callers (intltsel() at least) expect |
||||||
|
- er1p */ |
||||||
|
+ pfree(*low); pfree(*high); /* let's not leak the old strings */ |
||||||
|
+ *low = int4out(low_value); |
||||||
|
+ *high = int4out(high_value); |
||||||
|
+ |
||||||
|
+ /* XXX - this probably leaks the two tups we got from |
||||||
|
SearchSysCacheTuple() - er1p */ |
||||||
|
+ } |
||||||
|
} |
||||||
|
|
||||||
|
float64 |
||||||
|
SHAR_EOF |
||||||
|
fi |
||||||
|
if test -f 'vacuum.c.diff' |
||||||
|
then |
||||||
|
echo shar: "will not over-write existing file 'vacuum.c.diff'" |
||||||
|
else |
||||||
|
cat << \SHAR_EOF > 'vacuum.c.diff' |
||||||
|
*** |
||||||
|
/afs/ece.cmu.edu/project/lcs/lcs-004/er1p/postgres/611/src/backend/commands/ |
||||||
|
vacuum.c Thu Mar 11 23:59:09 1999 |
||||||
|
--- |
||||||
|
/afs/ece.cmu.edu/project/lcs/lcs-004/er1p/postgres/615/src/backend/commands/ |
||||||
|
vacuum.c Mon Mar 22 21:23:15 1999 |
||||||
|
*************** |
||||||
|
*** 1842,1848 **** |
||||||
|
i = 0; |
||||||
|
values[i++] = (Datum) relid; /* 1 */ |
||||||
|
values[i++] = (Datum) attp->attnum; /* 2 */ |
||||||
|
! values[i++] = (Datum) InvalidOid; /* 3 */ |
||||||
|
fmgr_info(stats->outfunc, &out_function); |
||||||
|
out_string = (*fmgr_faddr(&out_function)) (stats->min, |
||||||
|
stats->attr->atttypid); |
||||||
|
values[i++] = (Datum) fmgr(F_TEXTIN, out_string); |
||||||
|
--- 1842,1848 ---- |
||||||
|
i = 0; |
||||||
|
values[i++] = (Datum) relid; /* 1 */ |
||||||
|
values[i++] = (Datum) attp->attnum; /* 2 */ |
||||||
|
! values[i++] = (Datum) stats->f_cmplt.fn_oid; /* 3 */ /* get the |
||||||
|
'<' oid, instead of 'invalid' - er1p */ |
||||||
|
fmgr_info(stats->outfunc, &out_function); |
||||||
|
out_string = (*fmgr_faddr(&out_function)) (stats->min, |
||||||
|
stats->attr->atttypid); |
||||||
|
values[i++] = (Datum) fmgr(F_TEXTIN, out_string); |
||||||
|
SHAR_EOF |
||||||
|
fi |
||||||
|
exit 0 |
||||||
|
# End of shell archive |
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
From owner-pgsql-hackers@hub.org Tue Mar 23 12:31:05 1999 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id MAA17491 |
||||||
|
for <maillist@candle.pha.pa.us>; Tue, 23 Mar 1999 12:31:04 -0500 (EST) |
||||||
|
Received: from hub.org (majordom@hub.org [209.47.145.100]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id MAA08839 for <maillist@candle.pha.pa.us>; Tue, 23 Mar 1999 12:08:14 -0500 (EST) |
||||||
|
Received: from localhost (majordom@localhost) |
||||||
|
by hub.org (8.9.2/8.9.1) with SMTP id MAA93649; |
||||||
|
Tue, 23 Mar 1999 12:04:57 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@hub.org) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Tue, 23 Mar 1999 12:03:00 +0000 (EST) |
||||||
|
Received: (from majordom@localhost) |
||||||
|
by hub.org (8.9.2/8.9.1) id MAA93355 |
||||||
|
for pgsql-hackers-outgoing; Tue, 23 Mar 1999 12:02:55 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||||
|
Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [206.210.65.6]) |
||||||
|
by hub.org (8.9.2/8.9.1) with ESMTP id MAA93336 |
||||||
|
for <pgsql-hackers@postgreSQL.org>; Tue, 23 Mar 1999 12:02:43 -0500 (EST) |
||||||
|
(envelope-from tgl@sss.pgh.pa.us) |
||||||
|
Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) |
||||||
|
by sss.sss.pgh.pa.us (8.9.1/8.9.1) with ESMTP id MAA24455; |
||||||
|
Tue, 23 Mar 1999 12:01:57 -0500 (EST) |
||||||
|
To: Erik Riedel <riedel+@CMU.EDU> |
||||||
|
cc: pgsql-hackers@postgreSQL.org |
||||||
|
Subject: Re: [HACKERS] optimizer and type question |
||||||
|
In-reply-to: Your message of Mon, 22 Mar 1999 23:14:55 -0500 (EST) |
||||||
|
<4qxlJ0200anI01hK40@andrew.cmu.edu> |
||||||
|
Date: Tue, 23 Mar 1999 12:01:57 -0500 |
||||||
|
Message-ID: <24453.922208517@sss.pgh.pa.us> |
||||||
|
From: Tom Lane <tgl@sss.pgh.pa.us> |
||||||
|
Sender: owner-pgsql-hackers@postgreSQL.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
Erik Riedel <riedel+@CMU.EDU> writes: |
||||||
|
> OK, building on your high-level explanation, I am attaching a patch that |
||||||
|
> attempts to do something "better" than the current code. Note that I |
||||||
|
> have only tested this with the date type and my particular query. |
||||||
|
|
||||||
|
Glad to see you working on this. I don't like the details of your |
||||||
|
patch too much though ;-). Here are some suggestions for making it |
||||||
|
better. |
||||||
|
|
||||||
|
1. I think just removing staop from the lookup in gethilokey is OK for |
||||||
|
now, though I'm dubious about Bruce's thought that we could delete that |
||||||
|
field entirely. As you observe, vacuum will not currently put more |
||||||
|
than one tuple for a column into pg_statistic, so we can just do the |
||||||
|
lookup with relid and attno and leave it at that. But I think we ought |
||||||
|
to leave the field there, with the idea that vacuum might someday |
||||||
|
compute more than one statistic for a data column. Fixing vacuum to |
||||||
|
put its sort op into the field is a good idea in the meantime. |
||||||
|
|
||||||
|
2. The type conversion you're doing in gethilokey is a mess; I think |
||||||
|
what you ought to make it do is simply the inbound conversion of the |
||||||
|
string from pg_statistic into the internal representation for the |
||||||
|
column's datatype, and return that value as a Datum. It also needs |
||||||
|
a cleaner success/failure return convention --- this business with |
||||||
|
"n" return is ridiculously type-specific. Also, the best and easiest |
||||||
|
way to find the type to convert to is to look up the column type in |
||||||
|
the info for the given relid, not search pg_proc with the staop value. |
||||||
|
(I'm not sure that will even work, since there are pg_proc entries |
||||||
|
with wildcard argument types.) |
||||||
|
|
||||||
|
3. The atol() calls currently found in intltsel are a type-specific |
||||||
|
cheat on what is conceptually a two-step process: |
||||||
|
* Convert the string stored in pg_statistic back to the internal |
||||||
|
form for the column data type. |
||||||
|
* Generate a numeric representation of the data value that can be |
||||||
|
used as an estimate of the range of values in the table. |
||||||
|
The second step is trivial for integers, which may obscure the fact |
||||||
|
that there are two steps involved, but nonetheless there are. If |
||||||
|
you think about applying selectivity logic to strings, say, it |
||||||
|
becomes clear that the second step is a necessary component of the |
||||||
|
process. Furthermore, the second step must also be applied to the |
||||||
|
probe value that's being passed into the selectivity operator. |
||||||
|
(The probe value is already in internal form, of course; but it is |
||||||
|
not necessarily in a useful numeric form.) |
||||||
|
|
||||||
|
We can do the first of these steps by applying the appropriate "XXXin" |
||||||
|
conversion function for the column data type, as you have done. The |
||||||
|
interesting question is how to do the second one. A really clean |
||||||
|
solution would require adding a column to pg_type that points to a |
||||||
|
function that will do the appropriate conversion. I'd be inclined to |
||||||
|
make all of these functions return "double" (float8) and just have one |
||||||
|
top-level selectivity routine for all data types that can use |
||||||
|
range-based selectivity logic. |
||||||
|
|
||||||
|
We could probably hack something together that would not use an explicit |
||||||
|
conversion function for each data type, but instead would rely on |
||||||
|
type-specific assumptions inside the selectivity routines. We'd need many |
||||||
|
more selectivity routines though (at least one for each of int, float4, |
||||||
|
float8, and text data types) so I'm not sure we'd really save any work |
||||||
|
compared to doing it right. |
||||||
|
|
||||||
|
BTW, now that I look at this issue it's real clear that the selectivity |
||||||
|
entries in pg_operator are horribly broken. The intltsel/intgtsel |
||||||
|
selectivity routines are currently applied to 32 distinct data types: |
||||||
|
|
||||||
|
regression=> select distinct typname,oprleft from pg_operator, pg_type |
||||||
|
regression-> where pg_type.oid = oprleft |
||||||
|
regression-> and oprrest in (103,104); |
||||||
|
typname |oprleft |
||||||
|
---------+------- |
||||||
|
_aclitem | 1034 |
||||||
|
abstime | 702 |
||||||
|
bool | 16 |
||||||
|
box | 603 |
||||||
|
bpchar | 1042 |
||||||
|
char | 18 |
||||||
|
cidr | 650 |
||||||
|
circle | 718 |
||||||
|
date | 1082 |
||||||
|
datetime | 1184 |
||||||
|
float4 | 700 |
||||||
|
float8 | 701 |
||||||
|
inet | 869 |
||||||
|
int2 | 21 |
||||||
|
int4 | 23 |
||||||
|
int8 | 20 |
||||||
|
line | 628 |
||||||
|
lseg | 601 |
||||||
|
macaddr | 829 |
||||||
|
money | 790 |
||||||
|
name | 19 |
||||||
|
numeric | 1700 |
||||||
|
oid | 26 |
||||||
|
oid8 | 30 |
||||||
|
path | 602 |
||||||
|
point | 600 |
||||||
|
polygon | 604 |
||||||
|
text | 25 |
||||||
|
time | 1083 |
||||||
|
timespan | 1186 |
||||||
|
timestamp| 1296 |
||||||
|
varchar | 1043 |
||||||
|
(32 rows) |
||||||
|
|
||||||
|
many of which are very obviously not compatible with integer for *any* |
||||||
|
purpose. It looks to me like a lot of data types were added to |
||||||
|
pg_operator just by copy-and-paste, without paying attention to whether |
||||||
|
the selectivity routines were actually correct for the data type. |
||||||
|
|
||||||
|
As the code stands today, the bogus entries don't matter because |
||||||
|
gethilokey always fails, so we always get 1/3 as the selectivity |
||||||
|
estimate for any comparison operator (except = and != of course). |
||||||
|
I had actually noticed that fact and assumed that it was supposed |
||||||
|
to work that way :-(. But, clearly, there is code in here that |
||||||
|
is *trying* to be smarter. |
||||||
|
|
||||||
|
As soon as we fix gethilokey so that it can succeed, we will start |
||||||
|
getting essentially-random selectivity estimates for those data types |
||||||
|
that aren't actually binary-compatible with integer. That will not do; |
||||||
|
we have to do something about the issue. |
||||||
|
|
||||||
|
regards, tom lane |
||||||
|
|
||||||
|
|
||||||
|
From tgl@sss.pgh.pa.us Tue Mar 23 12:31:02 1999 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id MAA17484 |
||||||
|
for <maillist@candle.pha.pa.us>; Tue, 23 Mar 1999 12:31:01 -0500 (EST) |
||||||
|
Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [206.210.65.6]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id MAA09042 for <maillist@candle.pha.pa.us>; Tue, 23 Mar 1999 12:10:55 -0500 (EST) |
||||||
|
Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) |
||||||
|
by sss.sss.pgh.pa.us (8.9.1/8.9.1) with ESMTP id MAA24474; |
||||||
|
Tue, 23 Mar 1999 12:09:52 -0500 (EST) |
||||||
|
To: Bruce Momjian <maillist@candle.pha.pa.us> |
||||||
|
cc: riedel+@CMU.EDU, pgsql-hackers@postgreSQL.org |
||||||
|
Subject: Re: [HACKERS] optimizer and type question |
||||||
|
In-reply-to: Your message of Mon, 22 Mar 1999 21:25:45 -0500 (EST) |
||||||
|
<199903230225.VAA01641@candle.pha.pa.us> |
||||||
|
Date: Tue, 23 Mar 1999 12:09:52 -0500 |
||||||
|
Message-ID: <24471.922208992@sss.pgh.pa.us> |
||||||
|
From: Tom Lane <tgl@sss.pgh.pa.us> |
||||||
|
Status: RO |
||||||
|
|
||||||
|
Bruce Momjian <maillist@candle.pha.pa.us> writes: |
||||||
|
> What we really need is some way to determine how far the requested value |
||||||
|
> is from the min/max values. With int, we just do (val-min)/(max-min). |
||||||
|
> That works, but how do we do that for types that don't support division. |
||||||
|
> Strings come to mind in this case. |
||||||
|
|
||||||
|
What I'm envisioning is that we still apply the (val-min)/(max-min) |
||||||
|
logic, but apply it to numeric values that are produced in a |
||||||
|
type-dependent way. |
||||||
|
|
||||||
|
For ints and floats the conversion is trivial, of course. |
||||||
|
|
||||||
|
For strings, the first thing that comes to mind is to return 0 for a |
||||||
|
null string and the value of the first byte for a non-null string. |
||||||
|
This would give you one-part-in-256 selectivity which is plenty good |
||||||
|
enough for what the selectivity code needs to do. (Actually, it's |
||||||
|
only that good if the strings' first bytes are pretty well spread out. |
||||||
|
If you have a table containing English words, for example, you might |
||||||
|
only get about one part in 26 this way, since the first bytes will |
||||||
|
probably only run from A to Z. Might be better to use the first two |
||||||
|
characters of the string to compute the selectivity representation.) |
||||||
|
|
||||||
|
In general, you can apply this logic as long as you can come up with |
||||||
|
some numerical approximation to the data type's sorting order. It |
||||||
|
doesn't have to be exact. |
||||||
|
|
||||||
|
regards, tom lane |
||||||
|
|
||||||
|
From owner-pgsql-hackers@hub.org Tue Mar 23 12:31:03 1999 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id MAA17488 |
||||||
|
for <maillist@candle.pha.pa.us>; Tue, 23 Mar 1999 12:31:02 -0500 (EST) |
||||||
|
Received: from hub.org (majordom@hub.org [209.47.145.100]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id MAA09987 for <maillist@candle.pha.pa.us>; Tue, 23 Mar 1999 12:21:34 -0500 (EST) |
||||||
|
Received: from localhost (majordom@localhost) |
||||||
|
by hub.org (8.9.2/8.9.1) with SMTP id MAA95155; |
||||||
|
Tue, 23 Mar 1999 12:18:33 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@hub.org) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Tue, 23 Mar 1999 12:17:00 +0000 (EST) |
||||||
|
Received: (from majordom@localhost) |
||||||
|
by hub.org (8.9.2/8.9.1) id MAA94857 |
||||||
|
for pgsql-hackers-outgoing; Tue, 23 Mar 1999 12:16:56 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||||
|
Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [206.210.65.6]) |
||||||
|
by hub.org (8.9.2/8.9.1) with ESMTP id MAA94469 |
||||||
|
for <pgsql-hackers@postgreSQL.org>; Tue, 23 Mar 1999 12:11:33 -0500 (EST) |
||||||
|
(envelope-from tgl@sss.pgh.pa.us) |
||||||
|
Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) |
||||||
|
by sss.sss.pgh.pa.us (8.9.1/8.9.1) with ESMTP id MAA24474; |
||||||
|
Tue, 23 Mar 1999 12:09:52 -0500 (EST) |
||||||
|
To: Bruce Momjian <maillist@candle.pha.pa.us> |
||||||
|
cc: riedel+@CMU.EDU, pgsql-hackers@postgreSQL.org |
||||||
|
Subject: Re: [HACKERS] optimizer and type question |
||||||
|
In-reply-to: Your message of Mon, 22 Mar 1999 21:25:45 -0500 (EST) |
||||||
|
<199903230225.VAA01641@candle.pha.pa.us> |
||||||
|
Date: Tue, 23 Mar 1999 12:09:52 -0500 |
||||||
|
Message-ID: <24471.922208992@sss.pgh.pa.us> |
||||||
|
From: Tom Lane <tgl@sss.pgh.pa.us> |
||||||
|
Sender: owner-pgsql-hackers@postgreSQL.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
Bruce Momjian <maillist@candle.pha.pa.us> writes: |
||||||
|
> What we really need is some way to determine how far the requested value |
||||||
|
> is from the min/max values. With int, we just do (val-min)/(max-min). |
||||||
|
> That works, but how do we do that for types that don't support division. |
||||||
|
> Strings come to mind in this case. |
||||||
|
|
||||||
|
What I'm envisioning is that we still apply the (val-min)/(max-min) |
||||||
|
logic, but apply it to numeric values that are produced in a |
||||||
|
type-dependent way. |
||||||
|
|
||||||
|
For ints and floats the conversion is trivial, of course. |
||||||
|
|
||||||
|
For strings, the first thing that comes to mind is to return 0 for a |
||||||
|
null string and the value of the first byte for a non-null string. |
||||||
|
This would give you one-part-in-256 selectivity which is plenty good |
||||||
|
enough for what the selectivity code needs to do. (Actually, it's |
||||||
|
only that good if the strings' first bytes are pretty well spread out. |
||||||
|
If you have a table containing English words, for example, you might |
||||||
|
only get about one part in 26 this way, since the first bytes will |
||||||
|
probably only run from A to Z. Might be better to use the first two |
||||||
|
characters of the string to compute the selectivity representation.) |
||||||
|
|
||||||
|
In general, you can apply this logic as long as you can come up with |
||||||
|
some numerical approximation to the data type's sorting order. It |
||||||
|
doesn't have to be exact. |
||||||
|
|
||||||
|
regards, tom lane |
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,313 @@ |
|||||||
|
From lockhart@alumni.caltech.edu Thu Jan 7 13:31:08 1999 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA07771 |
||||||
|
for <maillist@candle.pha.pa.us>; Thu, 7 Jan 1999 13:31:06 -0500 (EST) |
||||||
|
Received: from golem.jpl.nasa.gov (IDENT:root@hectic-2.jpl.nasa.gov [128.149.68.204]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id NAA14597 for <maillist@candle.pha.pa.us>; Thu, 7 Jan 1999 13:27:37 -0500 (EST) |
||||||
|
Received: from alumni.caltech.edu (localhost [127.0.0.1]) |
||||||
|
by golem.jpl.nasa.gov (8.8.5/8.8.5) with ESMTP id SAA13416; |
||||||
|
Thu, 7 Jan 1999 18:26:56 GMT |
||||||
|
Sender: tgl@mythos.jpl.nasa.gov |
||||||
|
Message-ID: <3694FC70.FAD67BC3@alumni.caltech.edu> |
||||||
|
Date: Thu, 07 Jan 1999 18:26:56 +0000 |
||||||
|
From: "Thomas G. Lockhart" <lockhart@alumni.caltech.edu> |
||||||
|
Organization: Caltech/JPL |
||||||
|
X-Mailer: Mozilla 4.07 [en] (X11; I; Linux 2.0.30 i686) |
||||||
|
MIME-Version: 1.0 |
||||||
|
To: Bruce Momjian <maillist@candle.pha.pa.us> |
||||||
|
CC: Postgres Hackers List <hackers@postgresql.org> |
||||||
|
Subject: Outer Joins (and need CASE help) |
||||||
|
References: <199901071747.MAA07054@candle.pha.pa.us> |
||||||
|
Content-Type: text/plain; charset=us-ascii |
||||||
|
Content-Transfer-Encoding: 7bit |
||||||
|
Status: RO |
||||||
|
|
||||||
|
> Thomas, do you need help on outer joins? |
||||||
|
|
||||||
|
Yes. I'm going slowly partly because I get distracted with other |
||||||
|
Postgres stuff like docs, and partly because I don't understand all of |
||||||
|
the pieces I'm working with. |
||||||
|
|
||||||
|
I've identified the place in the MergeJoin code where the null filling |
||||||
|
for outer joins needs to happen, and have the "merge walk" code done. |
||||||
|
But I don't have the supporting code which actually would know how to |
||||||
|
null-fill a result tuple from the left or right. I thought you might be |
||||||
|
interested in that? |
||||||
|
|
||||||
|
I've done some work in the parser, and can now do things like: |
||||||
|
|
||||||
|
postgres=> select * from t1 join t2 using (i); |
||||||
|
NOTICE: JOIN not yet implemented |
||||||
|
i|j|i|k |
||||||
|
-+-+-+- |
||||||
|
1|2|1|3 |
||||||
|
(1 row) |
||||||
|
|
||||||
|
But this is just an inner join, and the result isn't quite right since |
||||||
|
the second "i" column should probably be omitted. At the moment I |
||||||
|
transform it from the syntax above into existing parse nodes, and |
||||||
|
everything from there on works. |
||||||
|
|
||||||
|
I don't yet pass an explicit join node into the planner/optimizer, and |
||||||
|
that will be the hardest part I assume. Perhaps we can work on that |
||||||
|
together. |
||||||
|
|
||||||
|
So, what I'll try to do (soon, in the next few days?) is put in |
||||||
|
|
||||||
|
#ifdef ENABLE_OUTER_JOINS |
||||||
|
|
||||||
|
conditional code into the parser area (already there for the executor) |
||||||
|
and commit everything to the development tree. Does that sound OK? |
||||||
|
|
||||||
|
Oh, and if anyone is looking for something to do, I've got a couple of |
||||||
|
CASE statements in the case.sql regression test which are commented out |
||||||
|
because they crash the backend. They involve references to multiple |
||||||
|
tables within a single result column, and in other contexts that |
||||||
|
construct works. It would be great if someone had time to track it |
||||||
|
down... |
||||||
|
|
||||||
|
- Tom |
||||||
|
|
||||||
|
From lockhart@alumni.caltech.edu Mon Feb 22 02:01:13 1999 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id CAA22073 |
||||||
|
for <maillist@candle.pha.pa.us>; Mon, 22 Feb 1999 02:01:12 -0500 (EST) |
||||||
|
Received: from golem.jpl.nasa.gov (IDENT:root@hectic-2.jpl.nasa.gov [128.149.68.204]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id BAA26054 for <maillist@candle.pha.pa.us>; Mon, 22 Feb 1999 01:57:00 -0500 (EST) |
||||||
|
Received: from alumni.caltech.edu (localhost [127.0.0.1]) |
||||||
|
by golem.jpl.nasa.gov (8.8.5/8.8.5) with ESMTP id GAA04715; |
||||||
|
Mon, 22 Feb 1999 06:56:36 GMT |
||||||
|
Sender: tgl@mythos.jpl.nasa.gov |
||||||
|
Message-ID: <36D0FFA4.32ADB75C@alumni.caltech.edu> |
||||||
|
Date: Mon, 22 Feb 1999 06:56:36 +0000 |
||||||
|
From: "Thomas G. Lockhart" <lockhart@alumni.caltech.edu> |
||||||
|
Organization: Caltech/JPL |
||||||
|
X-Mailer: Mozilla 4.07 [en] (X11; I; Linux 2.0.36 i686) |
||||||
|
MIME-Version: 1.0 |
||||||
|
To: Bruce Momjian <maillist@candle.pha.pa.us> |
||||||
|
CC: hackers@postgreSQL.org |
||||||
|
Subject: Re: start on outer join |
||||||
|
References: <199902220304.WAA10066@candle.pha.pa.us> |
||||||
|
Content-Type: text/plain; charset=us-ascii |
||||||
|
Content-Transfer-Encoding: 7bit |
||||||
|
Status: ROr |
||||||
|
|
||||||
|
Bruce Momjian wrote: |
||||||
|
> |
||||||
|
> > Will apply ... some other changes laying a bit of |
||||||
|
> > groundwork for outer joins so you can start on the planner/optimizer |
||||||
|
> > parts :) |
||||||
|
> Those will be a synch now that I understand the optimizer. In fact, I |
||||||
|
> think it all will happen in the executor. |
||||||
|
|
||||||
|
I've modified executor/nodeMergeJoin.c to walk a left/right/both outer |
||||||
|
join, but didn't fill in the part which actually creates the result |
||||||
|
tuple (which will be the current left- or right-side tuple plus nulls |
||||||
|
for filler). I hope this is up your alley :) |
||||||
|
|
||||||
|
So far, I'm not certain what to pass to the planner. The syntax leads me |
||||||
|
to pass a select structure from gram.y with a "JoinExpr" structure in |
||||||
|
the "fromClause" list. I need to expand that with a combination of |
||||||
|
column names and qualifications, but at the time I see the JoinExpr I |
||||||
|
don't have access to the top query structure itself. So I may just keep |
||||||
|
a modestly transformed JoinExpr to expand later or to pass to the |
||||||
|
planner. |
||||||
|
|
||||||
|
btw, the EXCEPT/INTERSECT stuff from Stefan has some ugliness in gram.y |
||||||
|
which needs to be fixed (the shift/reduce conflict is not acceptable for |
||||||
|
our release version) and some of that code clearly needs to move to |
||||||
|
analyze.c or some other module. |
||||||
|
|
||||||
|
- Tom |
||||||
|
|
||||||
|
From maillist Wed Feb 24 05:27:08 1999 |
||||||
|
Received: (from maillist@localhost) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) id FAA09648; |
||||||
|
Wed, 24 Feb 1999 05:27:08 -0500 (EST) |
||||||
|
From: Bruce Momjian <maillist> |
||||||
|
Message-Id: <199902241027.FAA09648@candle.pha.pa.us> |
||||||
|
Subject: Re: [HACKERS] OUTER joins |
||||||
|
In-Reply-To: <199902240953.EAA08561@candle.pha.pa.us> from Bruce Momjian at "Feb 24, 1999 4:53:21 am" |
||||||
|
To: maillist@candle.pha.pa.us (Bruce Momjian) |
||||||
|
Date: Wed, 24 Feb 1999 05:27:07 -0500 (EST) |
||||||
|
Cc: lockhart@alumni.caltech.edu, hackers@postgreSQL.org |
||||||
|
X-Mailer: ELM [version 2.4ME+ PL47 (25)] |
||||||
|
MIME-Version: 1.0 |
||||||
|
Content-Type: text/plain; charset=US-ASCII |
||||||
|
Content-Transfer-Encoding: 7bit |
||||||
|
Status: RO |
||||||
|
|
||||||
|
> |
||||||
|
> How do you propose doing outer joins in non-mergejoin situations? |
||||||
|
> Mergejoins can only be used currently in equal joins. |
||||||
|
|
||||||
|
Is your solution going to be to make sure the OUTER table is always a |
||||||
|
MergeJoin, or on the outside of a join loop? That could work. |
||||||
|
|
||||||
|
That could get tricky if the table is joined to _two_ other tables. |
||||||
|
With the cleaned-up optimizer, we can disable non-merge joins in certain |
||||||
|
circumstances, and prevent OUTER tables from being inner in the others. |
||||||
|
Is that the plan? |
||||||
|
|
||||||
|
-- |
||||||
|
Bruce Momjian | http://www.op.net/~candle |
||||||
|
maillist@candle.pha.pa.us | (610) 853-3000 |
||||||
|
+ If your life is a hard drive, | 830 Blythe Avenue |
||||||
|
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026 |
||||||
|
|
||||||
|
From lockhart@alumni.caltech.edu Mon Mar 1 13:01:08 1999 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA21672 |
||||||
|
for <maillist@candle.pha.pa.us>; Mon, 1 Mar 1999 13:01:06 -0500 (EST) |
||||||
|
Received: from golem.jpl.nasa.gov (IDENT:root@hectic-2.jpl.nasa.gov [128.149.68.204]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id MAA12756 for <maillist@candle.pha.pa.us>; Mon, 1 Mar 1999 12:14:16 -0500 (EST) |
||||||
|
Received: from alumni.caltech.edu (localhost [127.0.0.1]) |
||||||
|
by golem.jpl.nasa.gov (8.8.5/8.8.5) with ESMTP id RAA09406; |
||||||
|
Mon, 1 Mar 1999 17:10:49 GMT |
||||||
|
Sender: tgl@mythos.jpl.nasa.gov |
||||||
|
Message-ID: <36DACA19.E6DBE7D8@alumni.caltech.edu> |
||||||
|
Date: Mon, 01 Mar 1999 17:10:49 +0000 |
||||||
|
From: "Thomas G. Lockhart" <lockhart@alumni.caltech.edu> |
||||||
|
Organization: Caltech/JPL |
||||||
|
X-Mailer: Mozilla 4.07 [en] (X11; I; Linux 2.0.36 i686) |
||||||
|
MIME-Version: 1.0 |
||||||
|
To: Bruce Momjian <maillist@candle.pha.pa.us> |
||||||
|
CC: PostgreSQL-development <hackers@postgreSQL.org> |
||||||
|
Subject: Re: OUTER joins |
||||||
|
References: <199902240953.EAA08561@candle.pha.pa.us> |
||||||
|
Content-Type: text/plain; charset=us-ascii |
||||||
|
Content-Transfer-Encoding: 7bit |
||||||
|
Status: ROr |
||||||
|
|
||||||
|
(back from a short vacation...) |
||||||
|
|
||||||
|
> How do you propose doing outer joins in non-mergejoin situations? |
||||||
|
> Mergejoins can only be used currently in equal joins. |
||||||
|
|
||||||
|
Hadn't thought about it, other than figuring that implementing the |
||||||
|
equi-join first was a good start. There is a class of outer join syntax |
||||||
|
(the USING clause) which is implicitly an equi-join... |
||||||
|
|
||||||
|
- Tom |
||||||
|
|
||||||
|
From lockhart@alumni.caltech.edu Mon Mar 8 21:55:02 1999 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id VAA15978 |
||||||
|
for <maillist@candle.pha.pa.us>; Mon, 8 Mar 1999 21:54:57 -0500 (EST) |
||||||
|
Received: from golem.jpl.nasa.gov (IDENT:root@hectic-1.jpl.nasa.gov [128.149.68.203]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id VAA15837 for <maillist@candle.pha.pa.us>; Mon, 8 Mar 1999 21:48:33 -0500 (EST) |
||||||
|
Received: from alumni.caltech.edu (localhost [127.0.0.1]) |
||||||
|
by golem.jpl.nasa.gov (8.8.5/8.8.5) with ESMTP id CAA06996; |
||||||
|
Tue, 9 Mar 1999 02:46:40 GMT |
||||||
|
Sender: tgl@mythos.jpl.nasa.gov |
||||||
|
Message-ID: <36E48B90.F3E902B7@alumni.caltech.edu> |
||||||
|
Date: Tue, 09 Mar 1999 02:46:40 +0000 |
||||||
|
From: "Thomas G. Lockhart" <lockhart@alumni.caltech.edu> |
||||||
|
Organization: Caltech/JPL |
||||||
|
X-Mailer: Mozilla 4.07 [en] (X11; I; Linux 2.0.36 i686) |
||||||
|
MIME-Version: 1.0 |
||||||
|
To: Bruce Momjian <maillist@candle.pha.pa.us> |
||||||
|
CC: hackers@postgreSQL.org |
||||||
|
Subject: Re: OUTER joins |
||||||
|
References: <199903070325.WAA10357@candle.pha.pa.us> |
||||||
|
Content-Type: text/plain; charset=us-ascii |
||||||
|
Content-Transfer-Encoding: 7bit |
||||||
|
Status: ROr |
||||||
|
|
||||||
|
> > Hadn't thought about it, other than figuring that implementing the |
||||||
|
> > equi-join first was a good start. There is a class of outer join |
||||||
|
> > syntax (the USING clause) which is implicitly an equi-join... |
||||||
|
> Not that easy. You don't automatically get a mergejoin from an |
||||||
|
> equijoin. I will have to force outer's to be either mergejoins, or |
||||||
|
> inners of non-merge joins. Can you add code to non-merge joins in the |
||||||
|
> executor to throw out a null row if it does not find an inner match |
||||||
|
> for the outer row, and I will handle the optimizer so it doesn't throw |
||||||
|
> a non-conforming plan to the executor. |
||||||
|
|
||||||
|
So far I don't have enough info in the parser to get the |
||||||
|
planner/optimizer going. Should we work from the front to the back, or |
||||||
|
should I go ahead and look at the non-merge joins? It's painfully |
||||||
|
obvious that I don't know anything about the middle parts of this to |
||||||
|
proceed without lots more research. |
||||||
|
|
||||||
|
- Tom |
||||||
|
|
||||||
|
From lockhart@alumni.caltech.edu Tue Mar 9 22:47:57 1999 |
||||||
|
Received: from golem.jpl.nasa.gov (IDENT:root@hectic-1.jpl.nasa.gov [128.149.68.203]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id WAA07869 |
||||||
|
for <maillist@candle.pha.pa.us>; Tue, 9 Mar 1999 22:47:54 -0500 (EST) |
||||||
|
Received: from alumni.caltech.edu (localhost [127.0.0.1]) |
||||||
|
by golem.jpl.nasa.gov (8.8.5/8.8.5) with ESMTP id DAA14761; |
||||||
|
Wed, 10 Mar 1999 03:46:43 GMT |
||||||
|
Sender: tgl@mythos.jpl.nasa.gov |
||||||
|
Message-ID: <36E5EB23.F5CD959B@alumni.caltech.edu> |
||||||
|
Date: Wed, 10 Mar 1999 03:46:43 +0000 |
||||||
|
From: "Thomas G. Lockhart" <lockhart@alumni.caltech.edu> |
||||||
|
Organization: Caltech/JPL |
||||||
|
X-Mailer: Mozilla 4.07 [en] (X11; I; Linux 2.0.36 i686) |
||||||
|
MIME-Version: 1.0 |
||||||
|
To: Bruce Momjian <maillist@candle.pha.pa.us>, tgl@mythos.jpl.nasa.gov |
||||||
|
Subject: Re: SQL outer |
||||||
|
References: <199903100112.UAA05772@candle.pha.pa.us> |
||||||
|
Content-Type: text/plain; charset=us-ascii |
||||||
|
Content-Transfer-Encoding: 7bit |
||||||
|
Status: RO |
||||||
|
|
||||||
|
> select * |
||||||
|
> from outer tab1, tab2, tab3 |
||||||
|
> where tab1.col1 = tab2.col1 and |
||||||
|
> tab1.col1 = tab3.col1 |
||||||
|
|
||||||
|
select * |
||||||
|
from t1 left join t2 using (c1) |
||||||
|
join t3 on (c1 = t3.c1) |
||||||
|
|
||||||
|
Result: |
||||||
|
t1.c1 t1.c2 t2.c2 t3.c1 |
||||||
|
2 12 NULL 32 |
||||||
|
|
||||||
|
t1: |
||||||
|
c1 c2 |
||||||
|
1 11 |
||||||
|
2 12 |
||||||
|
3 13 |
||||||
|
4 14 |
||||||
|
|
||||||
|
t2: |
||||||
|
c1 c2 |
||||||
|
1 21 |
||||||
|
3 23 |
||||||
|
|
||||||
|
t3: |
||||||
|
c1 c2 |
||||||
|
2 32 |
||||||
|
|
||||||
|
From lockhart@alumni.caltech.edu Wed Mar 10 10:48:54 1999 |
||||||
|
Received: from golem.jpl.nasa.gov (IDENT:root@hectic-1.jpl.nasa.gov [128.149.68.203]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA16741 |
||||||
|
for <maillist@candle.pha.pa.us>; Wed, 10 Mar 1999 10:48:51 -0500 (EST) |
||||||
|
Received: from alumni.caltech.edu (localhost [127.0.0.1]) |
||||||
|
by golem.jpl.nasa.gov (8.8.5/8.8.5) with ESMTP id PAA17723; |
||||||
|
Wed, 10 Mar 1999 15:48:31 GMT |
||||||
|
Sender: tgl@mythos.jpl.nasa.gov |
||||||
|
Message-ID: <36E6944F.1F93B08@alumni.caltech.edu> |
||||||
|
Date: Wed, 10 Mar 1999 15:48:31 +0000 |
||||||
|
From: "Thomas G. Lockhart" <lockhart@alumni.caltech.edu> |
||||||
|
Organization: Caltech/JPL |
||||||
|
X-Mailer: Mozilla 4.07 [en] (X11; I; Linux 2.0.36 i686) |
||||||
|
MIME-Version: 1.0 |
||||||
|
To: Bruce Momjian <maillist@candle.pha.pa.us> |
||||||
|
CC: Thomas Lockhart <lockhart@alumni.caltech.edu> |
||||||
|
Subject: Re: SQL outer |
||||||
|
References: <199903100112.UAA05772@candle.pha.pa.us> <36E5EB23.F5CD959B@alumni.caltech.edu> |
||||||
|
Content-Type: text/plain; charset=us-ascii |
||||||
|
Content-Transfer-Encoding: 7bit |
||||||
|
Status: ROr |
||||||
|
|
||||||
|
Just thinking... |
||||||
|
|
||||||
|
If the initial RelOptInfo groupings are derived from the WHERE clause |
||||||
|
expressions, how about marking the "outer" property in those expressions |
||||||
|
in the parser? istm that is where the parser knows about two tables in |
||||||
|
one place, and I'm generating those expressions anyway. We could add a |
||||||
|
field(s) to the expression structure, or pass along a slightly different |
||||||
|
structure... |
||||||
|
|
||||||
|
- Tom |
||||||
|
|
||||||
@ -0,0 +1,343 @@ |
|||||||
|
From owner-pgsql-hackers@hub.org Sun Jun 14 18:45:04 1998 |
||||||
|
Received: from hub.org (hub.org [209.47.148.200]) |
||||||
|
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id SAA03690 |
||||||
|
for <maillist@candle.pha.pa.us>; Sun, 14 Jun 1998 18:45:00 -0400 (EDT) |
||||||
|
Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.7.5) with SMTP id SAA28049; Sun, 14 Jun 1998 18:39:42 -0400 (EDT) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Sun, 14 Jun 1998 18:36:06 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) by hub.org (8.8.8/8.7.5) id SAA27943 for pgsql-hackers-outgoing; Sun, 14 Jun 1998 18:36:04 -0400 (EDT) |
||||||
|
Received: from angular.illustra.com (ifmxoak.illustra.com [206.175.10.34]) by hub.org (8.8.8/8.7.5) with ESMTP id SAA27925 for <pgsql-hackers@postgresql.org>; Sun, 14 Jun 1998 18:35:47 -0400 (EDT) |
||||||
|
Received: from hawk.illustra.com (hawk.illustra.com [158.58.61.70]) by angular.illustra.com (8.7.4/8.7.3) with SMTP id PAA21293 for <pgsql-hackers@postgresql.org>; Sun, 14 Jun 1998 15:35:12 -0700 (PDT) |
||||||
|
Received: by hawk.illustra.com (5.x/smail2.5/06-10-94/S) |
||||||
|
id AA07922; Sun, 14 Jun 1998 15:35:13 -0700 |
||||||
|
From: dg@illustra.com (David Gould) |
||||||
|
Message-Id: <9806142235.AA07922@hawk.illustra.com> |
||||||
|
Subject: [HACKERS] performance tests, initial results |
||||||
|
To: pgsql-hackers@postgreSQL.org |
||||||
|
Date: Sun, 14 Jun 1998 15:35:13 -0700 (PDT) |
||||||
|
Mime-Version: 1.0 |
||||||
|
Content-Type: text/plain; charset=US-ASCII |
||||||
|
Content-Transfer-Encoding: 7bit |
||||||
|
Sender: owner-pgsql-hackers@hub.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
|
||||||
|
I have been playing a little with the performance tests found in |
||||||
|
pgsql/src/tests/performance and have a few observations that might be of |
||||||
|
minor interest. |
||||||
|
|
||||||
|
The tests themselves are simple enough although the result parsing in the |
||||||
|
driver did not work on Linux. I am enclosing a patch below to fix this. I |
||||||
|
think it will also work better on the other systems. |
||||||
|
|
||||||
|
A summary of results from my testing are below. Details are at the bottom |
||||||
|
of this message. |
||||||
|
|
||||||
|
My test system is 'leslie': |
||||||
|
|
||||||
|
linux 2.0.32, gcc version 2.7.2.3 |
||||||
|
P133, HX chipset, 512K L2, 32MB mem |
||||||
|
NCR810 fast scsi, Quantum Atlas 2GB drive (7200 rpm). |
||||||
|
|
||||||
|
|
||||||
|
Results Summary (times in seconds) |
||||||
|
|
||||||
|
Single txn 8K txn Create 8K idx 8K random Simple |
||||||
|
Case Description 8K insert 8K insert Index Insert Scans Orderby |
||||||
|
=================== ========== ========= ====== ====== ========= ======= |
||||||
|
1 From Distribution |
||||||
|
P90 FreeBsd -B256 39.56 1190.98 3.69 46.65 65.49 2.27 |
||||||
|
IDE |
||||||
|
|
||||||
|
2 Running on leslie |
||||||
|
P133 Linux 2.0.32 15.48 326.75 2.99 20.69 35.81 1.68 |
||||||
|
SCSI 32M |
||||||
|
|
||||||
|
3 leslie, -o -F |
||||||
|
no forced writes 15.90 24.98 2.63 20.46 36.43 1.69 |
||||||
|
|
||||||
|
4 leslie, -o -F |
||||||
|
no ASSERTS 14.92 23.23 1.38 18.67 33.79 1.58 |
||||||
|
|
||||||
|
5 leslie, -o -F -B2048 |
||||||
|
more buffers 21.31 42.28 2.65 25.74 42.26 1.72 |
||||||
|
|
||||||
|
6 leslie, -o -F -B2048 |
||||||
|
more bufs, no ASSERT 20.52 39.79 1.40 24.77 39.51 1.55 |
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Case to Case Difference Factors (+ is faster) |
||||||
|
|
||||||
|
Single txn 8K txn Create 8K idx 8K random Simple |
||||||
|
Case Description 8K insert 8K insert Index Insert Scans Orderby |
||||||
|
=================== ========== ========= ====== ====== ========= ======= |
||||||
|
|
||||||
|
leslie vs BSD P90. 2.56 3.65 1.23 2.25 1.83 1.35 |
||||||
|
|
||||||
|
(noflush -F) vs no -F -1.03 13.08 1.14 1.01 -1.02 1.00 |
||||||
|
|
||||||
|
No Assert vs Assert 1.05 1.07 1.90 1.06 1.07 1.09 |
||||||
|
|
||||||
|
-B256 vs -B2048 1.34 1.69 1.01 1.26 1.16 1.02 |
||||||
|
|
||||||
|
|
||||||
|
Observations: |
||||||
|
|
||||||
|
- leslie (P133 linux) appears to be about 1.8 times faster than the |
||||||
|
P90 BSD system used for the test result distributed with the source, not |
||||||
|
counting the 8K txn insert case which was completely disk bound. |
||||||
|
|
||||||
|
- SCSI disks make a big (factor of 3.6) difference. During this test the |
||||||
|
disk was hammering and cpu utilization was < 10%. |
||||||
|
|
||||||
|
- Assertion checking seems to cost about 7% except for create index where |
||||||
|
it costs 90% |
||||||
|
|
||||||
|
- the -F option to avoid flushing buffers has tremendous effect if there are |
||||||
|
many very small transactions. Or, another way, flushing at the end of the |
||||||
|
transaction is a major disaster for performance. |
||||||
|
|
||||||
|
- Something is very wrong with our buffer cache implementation. Going from |
||||||
|
256 buffers to 2048 buffers costs an average of 25%. In the 8K txn case |
||||||
|
it costs about 70%. I see looking at the code and profiling that in the 8K |
||||||
|
txn case this is in BufferSync() which examines all the buffers at commit |
||||||
|
time. I don't quite understand why it is so costly for the single 8K row |
||||||
|
txn (35%) though. |
||||||
|
|
||||||
|
It would be nice to have some more tests. Maybe the Wisconsin stuff will |
||||||
|
be useful. |
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
----------------- patch to test harness. apply from pgsql ------------ |
||||||
|
*** src/test/performance/runtests.pl.orig Sun Jun 14 11:34:04 1998 |
||||||
|
|
||||||
|
Differences % |
||||||
|
|
||||||
|
|
||||||
|
----------------- patch to test harness. apply from pgsql ------------ |
||||||
|
*** src/test/performance/runtests.pl.orig Sun Jun 14 11:34:04 1998 |
||||||
|
--- src/test/performance/runtests.pl Sun Jun 14 12:07:30 1998 |
||||||
|
*************** |
||||||
|
*** 84,123 **** |
||||||
|
open (STDERR, ">$TmpFile") or die; |
||||||
|
select (STDERR); $| = 1; |
||||||
|
|
||||||
|
! for ($i = 0; $i <= $#perftests; $i++) |
||||||
|
! { |
||||||
|
$test = $perftests[$i]; |
||||||
|
($test, $XACTBLOCK) = split (/ /, $test); |
||||||
|
$runtest = $test; |
||||||
|
! if ( $test =~ /\.ntm/ ) |
||||||
|
! { |
||||||
|
! # |
||||||
|
# No timing for this queries |
||||||
|
- # |
||||||
|
close (STDERR); # close $TmpFile |
||||||
|
open (STDERR, ">/dev/null") or die; |
||||||
|
$runtest =~ s/\.ntm//; |
||||||
|
} |
||||||
|
! else |
||||||
|
! { |
||||||
|
close (STDOUT); |
||||||
|
open(STDOUT, ">&SAVEOUT"); |
||||||
|
print STDOUT "\nRunning: $perftests[$i+1] ..."; |
||||||
|
close (STDOUT); |
||||||
|
open (STDOUT, ">/dev/null") or die; |
||||||
|
select (STDERR); $| = 1; |
||||||
|
! printf "$perftests[$i+1]: "; |
||||||
|
} |
||||||
|
|
||||||
|
do "sqls/$runtest"; |
||||||
|
|
||||||
|
# Restore STDERR to $TmpFile |
||||||
|
! if ( $test =~ /\.ntm/ ) |
||||||
|
! { |
||||||
|
close (STDERR); |
||||||
|
open (STDERR, ">>$TmpFile") or die; |
||||||
|
} |
||||||
|
- |
||||||
|
select (STDERR); $| = 1; |
||||||
|
$i++; |
||||||
|
} |
||||||
|
--- 84,116 ---- |
||||||
|
open (STDERR, ">$TmpFile") or die; |
||||||
|
select (STDERR); $| = 1; |
||||||
|
|
||||||
|
! for ($i = 0; $i <= $#perftests; $i++) { |
||||||
|
$test = $perftests[$i]; |
||||||
|
($test, $XACTBLOCK) = split (/ /, $test); |
||||||
|
$runtest = $test; |
||||||
|
! if ( $test =~ /\.ntm/ ) { |
||||||
|
# No timing for this queries |
||||||
|
close (STDERR); # close $TmpFile |
||||||
|
open (STDERR, ">/dev/null") or die; |
||||||
|
$runtest =~ s/\.ntm//; |
||||||
|
} |
||||||
|
! else { |
||||||
|
close (STDOUT); |
||||||
|
open(STDOUT, ">&SAVEOUT"); |
||||||
|
print STDOUT "\nRunning: $perftests[$i+1] ..."; |
||||||
|
close (STDOUT); |
||||||
|
open (STDOUT, ">/dev/null") or die; |
||||||
|
select (STDERR); $| = 1; |
||||||
|
! print "$perftests[$i+1]: "; |
||||||
|
} |
||||||
|
|
||||||
|
do "sqls/$runtest"; |
||||||
|
|
||||||
|
# Restore STDERR to $TmpFile |
||||||
|
! if ( $test =~ /\.ntm/ ) { |
||||||
|
close (STDERR); |
||||||
|
open (STDERR, ">>$TmpFile") or die; |
||||||
|
} |
||||||
|
select (STDERR); $| = 1; |
||||||
|
$i++; |
||||||
|
} |
||||||
|
*************** |
||||||
|
*** 128,138 **** |
||||||
|
open (TMPF, "<$TmpFile") or die; |
||||||
|
open (RESF, ">$ResFile") or die; |
||||||
|
|
||||||
|
! while (<TMPF>) |
||||||
|
! { |
||||||
|
! $str = $_; |
||||||
|
! ($test, $rtime) = split (/:/, $str); |
||||||
|
! ($tmp, $rtime, $rest) = split (/[ ]+/, $rtime); |
||||||
|
! print RESF "$test: $rtime\n"; |
||||||
|
} |
||||||
|
|
||||||
|
--- 121,130 ---- |
||||||
|
open (TMPF, "<$TmpFile") or die; |
||||||
|
open (RESF, ">$ResFile") or die; |
||||||
|
|
||||||
|
! while (<TMPF>) { |
||||||
|
! if (m/^(.*: ).* ([0-9:.]+) *elapsed/) { |
||||||
|
! ($test, $rtime) = ($1, $2); |
||||||
|
! print RESF $test, $rtime, "\n"; |
||||||
|
! } |
||||||
|
} |
||||||
|
|
||||||
|
------------------------------------------------------------------------ |
||||||
|
|
||||||
|
|
||||||
|
------------------------- testcase detail -------------------------- |
||||||
|
|
||||||
|
1. from distribution |
||||||
|
DBMS: PostgreSQL 6.2b10 |
||||||
|
OS: FreeBSD 2.1.5-RELEASE |
||||||
|
HardWare: i586/90, 24M RAM, IDE |
||||||
|
StartUp: postmaster -B 256 '-o -S 2048' -S |
||||||
|
Compiler: gcc 2.6.3 |
||||||
|
Compiled: -O, without CASSERT checking, with |
||||||
|
-DTBL_FREE_CMD_MEMORY (to free memory |
||||||
|
if BEGIN/END after each query execution) |
||||||
|
DB connection startup: 0.20 |
||||||
|
8192 INSERTs INTO SIMPLE (1 xact): 39.58 |
||||||
|
8192 INSERTs INTO SIMPLE (8192 xacts): 1190.98 |
||||||
|
Create INDEX on SIMPLE: 3.69 |
||||||
|
8192 INSERTs INTO SIMPLE with INDEX (1 xact): 46.65 |
||||||
|
8192 random INDEX scans on SIMPLE (1 xact): 65.49 |
||||||
|
ORDER BY SIMPLE: 2.27 |
||||||
|
|
||||||
|
|
||||||
|
2. run on leslie with asserts |
||||||
|
DBMS: PostgreSQL 6.3.2 (plus changes to 98/06/01) |
||||||
|
OS: Linux 2.0.32 leslie |
||||||
|
HardWare: i586/133 HX 512, 32M RAM, fast SCSI, 7200rpm |
||||||
|
StartUp: postmaster -B 256 '-o -S 2048' -S |
||||||
|
Compiler: gcc 2.7.2.3 |
||||||
|
Compiled: -O, WITH CASSERT checking, with |
||||||
|
-DTBL_FREE_CMD_MEMORY (to free memory |
||||||
|
if BEGIN/END after each query execution) |
||||||
|
DB connection startup: 0.10 |
||||||
|
8192 INSERTs INTO SIMPLE (1 xact): 15.48 |
||||||
|
8192 INSERTs INTO SIMPLE (8192 xacts): 326.75 |
||||||
|
Create INDEX on SIMPLE: 2.99 |
||||||
|
8192 INSERTs INTO SIMPLE with INDEX (1 xact): 20.69 |
||||||
|
8192 random INDEX scans on SIMPLE (1 xact): 35.81 |
||||||
|
ORDER BY SIMPLE: 1.68 |
||||||
|
|
||||||
|
|
||||||
|
3. with -F to avoid forced i/o |
||||||
|
DBMS: PostgreSQL 6.3.2 (plus changes to 98/06/01) |
||||||
|
OS: Linux 2.0.32 leslie |
||||||
|
HardWare: i586/133 HX 512, 32M RAM, fast SCSI, 7200rpm |
||||||
|
StartUp: postmaster -B 256 '-o -S 2048 -F' -S |
||||||
|
Compiler: gcc 2.7.2.3 |
||||||
|
Compiled: -O, WITH CASSERT checking, with |
||||||
|
-DTBL_FREE_CMD_MEMORY (to free memory |
||||||
|
if BEGIN/END after each query execution) |
||||||
|
DB connection startup: 0.10 |
||||||
|
8192 INSERTs INTO SIMPLE (1 xact): 15.90 |
||||||
|
8192 INSERTs INTO SIMPLE (8192 xacts): 24.98 |
||||||
|
Create INDEX on SIMPLE: 2.63 |
||||||
|
8192 INSERTs INTO SIMPLE with INDEX (1 xact): 20.46 |
||||||
|
8192 random INDEX scans on SIMPLE (1 xact): 36.43 |
||||||
|
ORDER BY SIMPLE: 1.69 |
||||||
|
|
||||||
|
|
||||||
|
4. no asserts, -F to avoid forced I/O |
||||||
|
DBMS: PostgreSQL 6.3.2 (plus changes to 98/06/01) |
||||||
|
OS: Linux 2.0.32 leslie |
||||||
|
HardWare: i586/133 HX 512, 32M RAM, fast SCSI, 7200rpm |
||||||
|
StartUp: postmaster -B 256 '-o -S 2048' -S |
||||||
|
Compiler: gcc 2.7.2.3 |
||||||
|
Compiled: -O, No CASSERT checking, with |
||||||
|
-DTBL_FREE_CMD_MEMORY (to free memory |
||||||
|
if BEGIN/END after each query execution) |
||||||
|
DB connection startup: 0.10 |
||||||
|
8192 INSERTs INTO SIMPLE (1 xact): 14.92 |
||||||
|
8192 INSERTs INTO SIMPLE (8192 xacts): 23.23 |
||||||
|
Create INDEX on SIMPLE: 1.38 |
||||||
|
8192 INSERTs INTO SIMPLE with INDEX (1 xact): 18.67 |
||||||
|
8192 random INDEX scans on SIMPLE (1 xact): 33.79 |
||||||
|
ORDER BY SIMPLE: 1.58 |
||||||
|
|
||||||
|
|
||||||
|
5. with more buffers (2048 vs 256) and -F to avoid forced i/o |
||||||
|
DBMS: PostgreSQL 6.3.2 (plus changes to 98/06/01) |
||||||
|
OS: Linux 2.0.32 leslie |
||||||
|
HardWare: i586/133 HX 512, 32M RAM, fast SCSI, 7200rpm |
||||||
|
StartUp: postmaster -B 2048 '-o -S 2048 -F' -S |
||||||
|
Compiler: gcc 2.7.2.3 |
||||||
|
Compiled: -O, WITH CASSERT checking, with |
||||||
|
-DTBL_FREE_CMD_MEMORY (to free memory |
||||||
|
if BEGIN/END after each query execution) |
||||||
|
DB connection startup: 0.11 |
||||||
|
8192 INSERTs INTO SIMPLE (1 xact): 21.31 |
||||||
|
8192 INSERTs INTO SIMPLE (8192 xacts): 42.28 |
||||||
|
Create INDEX on SIMPLE: 2.65 |
||||||
|
8192 INSERTs INTO SIMPLE with INDEX (1 xact): 25.74 |
||||||
|
8192 random INDEX scans on SIMPLE (1 xact): 42.26 |
||||||
|
ORDER BY SIMPLE: 1.72 |
||||||
|
|
||||||
|
|
||||||
|
6. No Asserts, more buffers (2048 vs 256) and -F to avoid forced i/o |
||||||
|
DBMS: PostgreSQL 6.3.2 (plus changes to 98/06/01) |
||||||
|
OS: Linux 2.0.32 leslie |
||||||
|
HardWare: i586/133 HX 512, 32M RAM, fast SCSI, 7200rpm |
||||||
|
StartUp: postmaster -B 2048 '-o -S 2048 -F' -S |
||||||
|
Compiler: gcc 2.7.2.3 |
||||||
|
Compiled: -O, No CASSERT checking, with |
||||||
|
-DTBL_FREE_CMD_MEMORY (to free memory |
||||||
|
if BEGIN/END after each query execution) |
||||||
|
DB connection startup: 0.11 |
||||||
|
8192 INSERTs INTO SIMPLE (1 xact): 20.52 |
||||||
|
8192 INSERTs INTO SIMPLE (8192 xacts): 39.79 |
||||||
|
Create INDEX on SIMPLE: 1.40 |
||||||
|
8192 INSERTs INTO SIMPLE with INDEX (1 xact): 24.77 |
||||||
|
8192 random INDEX scans on SIMPLE (1 xact): 39.51 |
||||||
|
ORDER BY SIMPLE: 1.55 |
||||||
|
--------------------------------------------------------------------- |
||||||
|
|
||||||
|
-dg |
||||||
|
|
||||||
|
David Gould dg@illustra.com 510.628.3783 or 510.305.9468 |
||||||
|
Informix Software (No, really) 300 Lakeside Drive Oakland, CA 94612 |
||||||
|
"Don't worry about people stealing your ideas. If your ideas are any |
||||||
|
good, you'll have to ram them down people's throats." -- Howard Aiken |
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,102 @@ |
|||||||
|
From owner-pgsql-hackers@hub.org Mon May 11 11:31:09 1998 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id LAA03006 |
||||||
|
for <maillist@candle.pha.pa.us>; Mon, 11 May 1998 11:31:07 -0400 (EDT) |
||||||
|
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id LAA01663 for <maillist@candle.pha.pa.us>; Mon, 11 May 1998 11:24:42 -0400 (EDT) |
||||||
|
Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.7.5) with SMTP id LAA21841; Mon, 11 May 1998 11:15:25 -0400 (EDT) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 11 May 1998 11:15:12 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) by hub.org (8.8.8/8.7.5) id LAA21683 for pgsql-hackers-outgoing; Mon, 11 May 1998 11:15:09 -0400 (EDT) |
||||||
|
Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [206.210.65.6]) by hub.org (8.8.8/8.7.5) with ESMTP id LAA21451 for <hackers@postgreSQL.org>; Mon, 11 May 1998 11:15:03 -0400 (EDT) |
||||||
|
Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) |
||||||
|
by sss.sss.pgh.pa.us (8.8.5/8.8.5) with ESMTP id LAA24915; |
||||||
|
Mon, 11 May 1998 11:14:43 -0400 (EDT) |
||||||
|
To: Brett McCormick <brett@work.chicken.org> |
||||||
|
cc: hackers@postgreSQL.org |
||||||
|
Subject: Re: [HACKERS] Re: [PATCHES] Try again: S_LOCK reduced contentionh] |
||||||
|
In-reply-to: Your message of Mon, 11 May 1998 07:57:23 -0700 (PDT) |
||||||
|
<13655.4384.345723.466046@abraxas.scene.com> |
||||||
|
Date: Mon, 11 May 1998 11:14:43 -0400 |
||||||
|
Message-ID: <24913.894899683@sss.pgh.pa.us> |
||||||
|
From: Tom Lane <tgl@sss.pgh.pa.us> |
||||||
|
Sender: owner-pgsql-hackers@hub.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
Brett McCormick <brett@work.chicken.org> writes: |
||||||
|
> same way that the current network socket is passed -- through an execv |
||||||
|
> argument. hopefully, however, the non-execv()ing fork will be in 6.4. |
||||||
|
|
||||||
|
Um, you missed the point, Brett. David was hoping to transfer a client |
||||||
|
connection from the postmaster to an *already existing* backend process. |
||||||
|
Fork, with or without exec, solves the problem for a backend that's |
||||||
|
started after the postmaster has accepted the client socket. |
||||||
|
|
||||||
|
This does lead to a different line of thought, however. Pre-started |
||||||
|
backends would have access to the "master" connection socket on which |
||||||
|
the postmaster listens for client connections, right? Suppose that we |
||||||
|
fire the postmaster as postmaster, and demote it to being simply a |
||||||
|
manufacturer of new backend processes as old ones get used up. Have |
||||||
|
one of the idle backend processes be the one doing the accept() on the |
||||||
|
master socket. Once it has a client connection, it performs the |
||||||
|
authentication handshake and then starts serving the client (or just |
||||||
|
quits if authentication fails). Meanwhile the next idle backend process |
||||||
|
has executed accept() on the master socket and is waiting for the next |
||||||
|
client; and shortly the postmaster/factory/whateverwecallitnow notices |
||||||
|
that it needs to start another backend to add to the idle-backend pool. |
||||||
|
|
||||||
|
This'd probably need some interlocking among the backends. I have no |
||||||
|
idea whether it'd be safe to have all the idle backends trying to |
||||||
|
do accept() on the master socket simultaneously, but it sounds risky. |
||||||
|
Better to use a mutex so that only one gets to do it while the others |
||||||
|
sleep. |
||||||
|
|
||||||
|
regards, tom lane |
||||||
|
|
||||||
|
|
||||||
|
From owner-pgsql-hackers@hub.org Mon May 11 11:35:55 1998 |
||||||
|
Received: from hub.org (hub.org [209.47.148.200]) |
||||||
|
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id LAA03043 |
||||||
|
for <maillist@candle.pha.pa.us>; Mon, 11 May 1998 11:35:53 -0400 (EDT) |
||||||
|
Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.7.5) with SMTP id LAA23494; Mon, 11 May 1998 11:27:10 -0400 (EDT) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 11 May 1998 11:27:02 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) by hub.org (8.8.8/8.7.5) id LAA23473 for pgsql-hackers-outgoing; Mon, 11 May 1998 11:27:01 -0400 (EDT) |
||||||
|
Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [206.210.65.6]) by hub.org (8.8.8/8.7.5) with ESMTP id LAA23462 for <hackers@postgreSQL.org>; Mon, 11 May 1998 11:26:56 -0400 (EDT) |
||||||
|
Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) |
||||||
|
by sss.sss.pgh.pa.us (8.8.5/8.8.5) with ESMTP id LAA25006; |
||||||
|
Mon, 11 May 1998 11:26:44 -0400 (EDT) |
||||||
|
To: Brett McCormick <brett@work.chicken.org> |
||||||
|
cc: hackers@postgreSQL.org |
||||||
|
Subject: Re: [HACKERS] Re: [PATCHES] Try again: S_LOCK reduced contentionh] |
||||||
|
In-reply-to: Your message of Mon, 11 May 1998 07:57:23 -0700 (PDT) |
||||||
|
<13655.4384.345723.466046@abraxas.scene.com> |
||||||
|
Date: Mon, 11 May 1998 11:26:44 -0400 |
||||||
|
Message-ID: <25004.894900404@sss.pgh.pa.us> |
||||||
|
From: Tom Lane <tgl@sss.pgh.pa.us> |
||||||
|
Sender: owner-pgsql-hackers@hub.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
Meanwhile, *I* missed the point about Brett's second comment :-( |
||||||
|
|
||||||
|
Brett McCormick <brett@work.chicken.org> writes: |
||||||
|
> There will have to be some sort of arg parsing in any case, |
||||||
|
> considering that you can pass configurable arguments to the backend.. |
||||||
|
|
||||||
|
If we do the sort of change David and I were just discussing, then the |
||||||
|
pre-spawned backend would become responsible for parsing and dealing |
||||||
|
with the PGOPTIONS portion of the client's connection request message. |
||||||
|
That's just part of shifting the authentication handshake code from |
||||||
|
postmaster to backend, so it shouldn't be too hard. |
||||||
|
|
||||||
|
BUT: the whole point is to be able to initialize the backend before it |
||||||
|
is connected to a client. How much of the expensive backend startup |
||||||
|
work depends on having the client connection options available? |
||||||
|
Any work that needs to know the options will have to wait until after |
||||||
|
the client connects. If that means most of the startup work can't |
||||||
|
happen in advance anyway, then we're out of luck; a pre-started backend |
||||||
|
won't save enough time to be worth the effort. (Unless we are willing |
||||||
|
to eliminate or redefine the troublesome options...) |
||||||
|
|
||||||
|
regards, tom lane |
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,55 @@ |
|||||||
|
From owner-pgsql-hackers@hub.org Sun Aug 2 20:01:13 1998 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id UAA15937 |
||||||
|
for <maillist@candle.pha.pa.us>; Sun, 2 Aug 1998 20:01:11 -0400 (EDT) |
||||||
|
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id TAA01026 for <maillist@candle.pha.pa.us>; Sun, 2 Aug 1998 19:33:53 -0400 (EDT) |
||||||
|
Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.7.5) with SMTP id TAA19878; Sun, 2 Aug 1998 19:30:59 -0400 (EDT) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Sun, 02 Aug 1998 19:28:23 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) by hub.org (8.8.8/8.7.5) id TAA19534 for pgsql-hackers-outgoing; Sun, 2 Aug 1998 19:28:22 -0400 (EDT) |
||||||
|
Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [206.210.65.6]) by hub.org (8.8.8/8.7.5) with ESMTP id TAA19521 for <pgsql-hackers@postgreSQL.org>; Sun, 2 Aug 1998 19:28:15 -0400 (EDT) |
||||||
|
Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) |
||||||
|
by sss.sss.pgh.pa.us (8.9.1/8.9.1) with ESMTP id TAA22594 |
||||||
|
for <pgsql-hackers@postgreSQL.org>; Sun, 2 Aug 1998 19:28:13 -0400 (EDT) |
||||||
|
To: pgsql-hackers@postgreSQL.org |
||||||
|
Subject: [HACKERS] TODO item: make pg_shadow updates more robust |
||||||
|
Date: Sun, 02 Aug 1998 19:28:13 -0400 |
||||||
|
Message-ID: <22591.902100493@sss.pgh.pa.us> |
||||||
|
From: Tom Lane <tgl@sss.pgh.pa.us> |
||||||
|
Sender: owner-pgsql-hackers@hub.org |
||||||
|
Precedence: bulk |
||||||
|
Status: ROr |
||||||
|
|
||||||
|
I learned the hard way last night that the postmaster's password |
||||||
|
authentication routines don't look at the pg_shadow table. They |
||||||
|
look at a separate file named pg_pwd, which certain backend operations |
||||||
|
will update from pg_shadow. (This is not documented in any user |
||||||
|
documentation that I could find; I had to burrow into |
||||||
|
src/backend/commands/user.c to discover it.) |
||||||
|
|
||||||
|
Unfortunately, if a clueless dbadmin (like me ;-)) tries to update |
||||||
|
password data with the obvious thing, |
||||||
|
update pg_shadow set passwd = 'xxxxx' where usename = 'yyyy'; |
||||||
|
pg_pwd doesn't get fixed. |
||||||
|
|
||||||
|
A more drastic problem is that pg_dump believes it can save and |
||||||
|
restore pg_shadow data using "copy". Following an initdb and restore |
||||||
|
from a pg_dump -z script, pg_shadow will look just fine, but only |
||||||
|
the database admin will be listed in pg_pwd. This is likely to provoke |
||||||
|
some confusion, IMHO. |
||||||
|
|
||||||
|
As a short-term thing, the fact that you *must* set passwords with |
||||||
|
ALTER USER ought to be documented, preferably someplace where a |
||||||
|
dbadmin who's never heard of ALTER USER is likely to find it. |
||||||
|
|
||||||
|
As a longer-term thing, I think it would be far better if ordinary |
||||||
|
SQL operations on pg_shadow just did the right thing. Wouldn't it |
||||||
|
be possible to implement copying to pg_pwd by means of a trigger on |
||||||
|
pg_shadow updates, or something like that? |
||||||
|
|
||||||
|
(I'm afraid that pg_dump -z is pretty well broken for operations on |
||||||
|
a password-protected database, btw. Has anyone used it successfully |
||||||
|
in that situation?) |
||||||
|
|
||||||
|
regards, tom lane |
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,98 @@ |
|||||||
|
From owner-pgsql-hackers@hub.org Wed Nov 18 14:40:49 1998 |
||||||
|
Received: from hub.org (majordom@hub.org [209.47.148.200]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id OAA29743 |
||||||
|
for <maillist@candle.pha.pa.us>; Wed, 18 Nov 1998 14:40:36 -0500 (EST) |
||||||
|
Received: from localhost (majordom@localhost) |
||||||
|
by hub.org (8.9.1/8.9.1) with SMTP id OAA03716; |
||||||
|
Wed, 18 Nov 1998 14:37:04 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@hub.org) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 18 Nov 1998 14:34:39 +0000 (EST) |
||||||
|
Received: (from majordom@localhost) |
||||||
|
by hub.org (8.9.1/8.9.1) id OAA03395 |
||||||
|
for pgsql-hackers-outgoing; Wed, 18 Nov 1998 14:34:37 -0500 (EST) |
||||||
|
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||||
|
Received: from orion.SAPserv.Hamburg.dsh.de (Tpolaris2.sapham.debis.de [53.2.131.8]) |
||||||
|
by hub.org (8.9.1/8.9.1) with SMTP id OAA03381 |
||||||
|
for <pgsql-hackers@hub.org>; Wed, 18 Nov 1998 14:34:31 -0500 (EST) |
||||||
|
(envelope-from wieck@sapserv.debis.de) |
||||||
|
Received: by orion.SAPserv.Hamburg.dsh.de |
||||||
|
for pgsql-hackers@hub.org |
||||||
|
id m0zgDnj-000EBTC; Wed, 18 Nov 98 21:02 MET |
||||||
|
Message-Id: <m0zgDnj-000EBTC@orion.SAPserv.Hamburg.dsh.de> |
||||||
|
From: jwieck@debis.com (Jan Wieck) |
||||||
|
Subject: Re: [HACKERS] PREPARE |
||||||
|
To: meskes@usa.net (Michael Meskes) |
||||||
|
Date: Wed, 18 Nov 1998 21:02:06 +0100 (MET) |
||||||
|
Cc: pgsql-hackers@hub.org |
||||||
|
Reply-To: jwieck@debis.com (Jan Wieck) |
||||||
|
In-Reply-To: <19981118084843.B869@usa.net> from "Michael Meskes" at Nov 18, 98 08:48:43 am |
||||||
|
X-Mailer: ELM [version 2.4 PL25] |
||||||
|
Content-Type: text |
||||||
|
Sender: owner-pgsql-hackers@postgreSQL.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
Michael Meskes wrote: |
||||||
|
|
||||||
|
> |
||||||
|
> On Wed, Nov 18, 1998 at 03:23:30AM +0000, Thomas G. Lockhart wrote: |
||||||
|
> > > I didn't get this one completly. What input do you mean? |
||||||
|
> > |
||||||
|
> > Just the original string/query to be prepared... |
||||||
|
> |
||||||
|
> I see. But wouldn't it be more useful to preprocess the query and store the |
||||||
|
> resulting nodes instead? We don't want to parse the statement everytime a |
||||||
|
> variable binding comes in. |
||||||
|
|
||||||
|
Right. A real improvement would only be to have the prepared |
||||||
|
execution plan in the backend and just giving the parameter |
||||||
|
values. |
||||||
|
|
||||||
|
I can think of the following construct: |
||||||
|
|
||||||
|
PREPARE optimizable-statement; |
||||||
|
|
||||||
|
That one will run parser/rewrite/planner, create a new memory |
||||||
|
context with a unique identifier and saves the querytree's |
||||||
|
and plan's in it. Parameter values are identified by the |
||||||
|
usual $n notation. The command returns the identifier. |
||||||
|
|
||||||
|
EXECUTE QUERY identifier [value [, ...]]; |
||||||
|
|
||||||
|
then get's back the prepared plan and querytree by the id, |
||||||
|
creates an executor context with the given values in the |
||||||
|
parameter array and calls ExecutorRun() for them. |
||||||
|
|
||||||
|
The PREPARE needs to analyze the resulting parsetrees to get |
||||||
|
the datatypes (and maybe atttypmod's) of the parameters, so |
||||||
|
EXECUTE QUERY can convert the values into Datum's using the |
||||||
|
types input functions. And the EXECUTE has to be handled |
||||||
|
special in tcop (it's something between a regular query and |
||||||
|
an utility statement). But it's not too hard to implement. |
||||||
|
|
||||||
|
Finally a |
||||||
|
|
||||||
|
FORGET QUERY identifier; |
||||||
|
|
||||||
|
(don't remember how the others named it) will remove the |
||||||
|
prepared plan etc. simply by destroying the memory context |
||||||
|
and dropping the identifier from the id->mcontext+prepareinfo |
||||||
|
mapping. |
||||||
|
|
||||||
|
This all restricts the usage of PREPARE to optimizable |
||||||
|
statements. Is it required to be able to prepare utility |
||||||
|
statements (like CREATE TABLE or so) too? |
||||||
|
|
||||||
|
|
||||||
|
Jan |
||||||
|
|
||||||
|
-- |
||||||
|
|
||||||
|
#======================================================================# |
||||||
|
# It's easier to get forgiveness for being wrong than for being right. # |
||||||
|
# Let's break this rule - forgive me. # |
||||||
|
#======================================== jwieck@debis.com (Jan Wieck) # |
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,159 @@ |
|||||||
|
From owner-pgsql-hackers@hub.org Fri Sep 4 00:47:06 1998 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id AAA01047 |
||||||
|
for <maillist@candle.pha.pa.us>; Fri, 4 Sep 1998 00:47:05 -0400 (EDT) |
||||||
|
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id XAA02044 for <maillist@candle.pha.pa.us>; Thu, 3 Sep 1998 23:11:07 -0400 (EDT) |
||||||
|
Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.7.5) with SMTP id XAA27418; Thu, 3 Sep 1998 23:06:16 -0400 (EDT) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Thu, 03 Sep 1998 23:04:11 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) by hub.org (8.8.8/8.7.5) id XAA27185 for pgsql-hackers-outgoing; Thu, 3 Sep 1998 23:04:09 -0400 (EDT) |
||||||
|
Received: from dune.krs.ru (dune.krs.ru [195.161.16.38]) by hub.org (8.8.8/8.7.5) with ESMTP id XAA27169 for <hackers@postgreSQL.org>; Thu, 3 Sep 1998 23:03:59 -0400 (EDT) |
||||||
|
Received: from krs.ru (localhost.krs.ru [127.0.0.1]) |
||||||
|
by dune.krs.ru (8.8.8/8.8.8) with ESMTP id LAA10059; |
||||||
|
Fri, 4 Sep 1998 11:03:00 +0800 (KRSS) |
||||||
|
(envelope-from vadim@krs.ru) |
||||||
|
Message-ID: <35EF5864.E5142D35@krs.ru> |
||||||
|
Date: Fri, 04 Sep 1998 11:03:00 +0800 |
||||||
|
From: Vadim Mikheev <vadim@krs.ru> |
||||||
|
Organization: OJSC Rostelecom (Krasnoyarsk) |
||||||
|
X-Mailer: Mozilla 4.05 [en] (X11; I; FreeBSD 2.2.6-RELEASE i386) |
||||||
|
MIME-Version: 1.0 |
||||||
|
To: "D'Arcy J.M. Cain" <darcy@druid.net> |
||||||
|
CC: "Thomas G. Lockhart" <lockhart@alumni.caltech.edu>, hackers@postgreSQL.org |
||||||
|
Subject: Re: [HACKERS] Adding PRIMARY KEY info |
||||||
|
References: <m0zEaoV-00006JC@druid.net> |
||||||
|
Content-Type: text/plain; charset=us-ascii |
||||||
|
Content-Transfer-Encoding: 7bit |
||||||
|
Sender: owner-pgsql-hackers@hub.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
D'Arcy J.M. Cain wrote: |
||||||
|
> |
||||||
|
> Thus spake Vadim Mikheev |
||||||
|
> > Imho, indices should be used/created for FOREIGN keys and so pg_index |
||||||
|
> > is good place for both PRIMARY and FOREIGN keys infos. |
||||||
|
> |
||||||
|
> Are you sure? I don't know about implementing it but it seems more |
||||||
|
> like an attribute thing rather than an index thing. Certainly from a |
||||||
|
> database design viewpoint you want to refer to the fields, not the |
||||||
|
> index on them. If you put it into the index then you have to do |
||||||
|
> an extra join to get the information. |
||||||
|
> |
||||||
|
> Perhaps you have to do the extra join anyway for other purposes so it |
||||||
|
> may not matter. All I want is to be able to be able to extract the |
||||||
|
> field that the designer specified as the key. As long as I can design |
||||||
|
> a select statement that gives me that I don't much care how it is |
||||||
|
> implemented. I'll cache the information anyway so it won't have a |
||||||
|
> huge impact on my programs. |
||||||
|
|
||||||
|
First, let me note that you have to add int28 field to pg_class, |
||||||
|
not just oid field, to know what attributeS are in primary key |
||||||
|
(we support multi-attribute primary keys). |
||||||
|
This could be done... |
||||||
|
But what about foreign and unique (!) keys ? |
||||||
|
There may be _many_ foreign/unique keys defined for one table! |
||||||
|
And so foreign/unique keys info have to be stored somewhere else, |
||||||
|
not in pg_class. |
||||||
|
|
||||||
|
pg_index is good place for all _3_ key types because of: |
||||||
|
|
||||||
|
1. index should be created for each foreign key - |
||||||
|
just for performance. |
||||||
|
2. pg_index already has int28 field for key attributes. |
||||||
|
3. pg_index already has indisunique (note that foreign keys |
||||||
|
may reference unique keys, not just primary ones). |
||||||
|
|
||||||
|
- so we have just add two fields to pg_index: |
||||||
|
|
||||||
|
bool indisprimary; |
||||||
|
oid indreferenced; |
||||||
|
^^^^^^^^^^^^^^^^^^ |
||||||
|
this is for foreign keys: oid of referenced relation' |
||||||
|
primary/unique key index. |
||||||
|
|
||||||
|
I agreed that indices are just implementation... |
||||||
|
If you don't like to store key infos in pg_index then |
||||||
|
new pg_key relation have to be added... |
||||||
|
|
||||||
|
Comments ? |
||||||
|
|
||||||
|
Vadim |
||||||
|
|
||||||
|
|
||||||
|
From owner-pgsql-hackers@hub.org Sat Sep 5 02:01:13 1998 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id CAA14437 |
||||||
|
for <maillist@candle.pha.pa.us>; Sat, 5 Sep 1998 02:01:11 -0400 (EDT) |
||||||
|
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id BAA09928 for <maillist@candle.pha.pa.us>; Sat, 5 Sep 1998 01:48:32 -0400 (EDT) |
||||||
|
Received: from localhost (majordom@localhost) by hub.org (8.8.8/8.7.5) with SMTP id BAA18282; Sat, 5 Sep 1998 01:43:16 -0400 (EDT) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Sat, 05 Sep 1998 01:41:40 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) by hub.org (8.8.8/8.7.5) id BAA18241 for pgsql-hackers-outgoing; Sat, 5 Sep 1998 01:41:38 -0400 (EDT) |
||||||
|
Received: from dune.krs.ru (dune.krs.ru [195.161.16.38]) by hub.org (8.8.8/8.7.5) with ESMTP id BAA18211; Sat, 5 Sep 1998 01:41:21 -0400 (EDT) |
||||||
|
Received: from krs.ru (localhost.krs.ru [127.0.0.1]) |
||||||
|
by dune.krs.ru (8.8.8/8.8.8) with ESMTP id NAA20555; |
||||||
|
Sat, 5 Sep 1998 13:40:44 +0800 (KRSS) |
||||||
|
(envelope-from vadim@krs.ru) |
||||||
|
Message-ID: <35F0CEDB.AD721090@krs.ru> |
||||||
|
Date: Sat, 05 Sep 1998 13:40:43 +0800 |
||||||
|
From: Vadim Mikheev <vadim@krs.ru> |
||||||
|
Organization: OJSC Rostelecom (Krasnoyarsk) |
||||||
|
X-Mailer: Mozilla 4.05 [en] (X11; I; FreeBSD 2.2.6-RELEASE i386) |
||||||
|
MIME-Version: 1.0 |
||||||
|
To: "D'Arcy J.M. Cain" <darcy@druid.net> |
||||||
|
CC: hackers@postgreSQL.org, pgsql-core@postgreSQL.org |
||||||
|
Subject: Re: [HACKERS] Adding PRIMARY KEY info |
||||||
|
References: <m0zEvLK-00006FC@druid.net> |
||||||
|
Content-Type: text/plain; charset=us-ascii |
||||||
|
Content-Transfer-Encoding: 7bit |
||||||
|
Sender: owner-pgsql-hackers@hub.org |
||||||
|
Precedence: bulk |
||||||
|
Status: ROr |
||||||
|
|
||||||
|
D'Arcy J.M. Cain wrote: |
||||||
|
> |
||||||
|
> > |
||||||
|
> > pg_index is good place for all _3_ key types because of: |
||||||
|
> > |
||||||
|
> > 1. index should be created for each foreign key - |
||||||
|
> > just for performance. |
||||||
|
> > 2. pg_index already has int28 field for key attributes. |
||||||
|
> > 3. pg_index already has indisunique (note that foreign keys |
||||||
|
> > may reference unique keys, not just primary ones). |
||||||
|
> > |
||||||
|
> > - so we have just add two fields to pg_index: |
||||||
|
> > |
||||||
|
> > bool indisprimary; |
||||||
|
> > oid indreferenced; |
||||||
|
> > ^^^^^^^^^^^^^^^^^^ |
||||||
|
> > this is for foreign keys: oid of referenced relation' |
||||||
|
> > primary/unique key index. |
||||||
|
> |
||||||
|
> Sounds fine to me. Any chance of seeing this in 6.4? |
||||||
|
|
||||||
|
I could add this (and FOREIGN key implementation) before |
||||||
|
11-13 Sep... But not the ALTER TABLE ADD/DROP CONSTRAINT |
||||||
|
stuff (ok for Entry SQL). |
||||||
|
But we are in beta... |
||||||
|
|
||||||
|
Comments? |
||||||
|
|
||||||
|
> Nope, pg_index is fine by me. Now, once we have this, how do we find |
||||||
|
> the index for a particular attribute? I can't seem to figure out the |
||||||
|
> relationship between pg_attribute and pg_index. The chart in the docs |
||||||
|
> suggests that indkey is the relation but I can't see any useful info |
||||||
|
> there for joining the tables. |
||||||
|
|
||||||
|
pg_index: |
||||||
|
indrelid - oid of indexed relation |
||||||
|
indkey - up to the 8 attnums |
||||||
|
|
||||||
|
pg_attribute: |
||||||
|
attrelid - oid of relation |
||||||
|
attnum - ... |
||||||
|
|
||||||
|
Without outer join you have to query pg_attribute for each |
||||||
|
valid attnum from pg_index->indkey -:( |
||||||
|
|
||||||
|
Vadim |
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,240 @@ |
|||||||
|
From owner-pgsql-patches@hub.org Wed Oct 14 17:31:26 1998 |
||||||
|
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||||
|
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id RAA01594 |
||||||
|
for <maillist@candle.pha.pa.us>; Wed, 14 Oct 1998 17:31:24 -0400 (EDT) |
||||||
|
Received: from hub.org (majordom@hub.org [209.47.148.200]) by renoir.op.net (o1/$Revision: 1.1 $) with ESMTP id RAA01745 for <maillist@candle.pha.pa.us>; Wed, 14 Oct 1998 17:12:28 -0400 (EDT) |
||||||
|
Received: from localhost (majordom@localhost) |
||||||
|
by hub.org (8.8.8/8.8.8) with SMTP id RAA06607; |
||||||
|
Wed, 14 Oct 1998 17:10:43 -0400 (EDT) |
||||||
|
(envelope-from owner-pgsql-patches@hub.org) |
||||||
|
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 14 Oct 1998 17:10:27 +0000 (EDT) |
||||||
|
Received: (from majordom@localhost) |
||||||
|
by hub.org (8.8.8/8.8.8) id RAA06562 |
||||||
|
for pgsql-patches-outgoing; Wed, 14 Oct 1998 17:10:26 -0400 (EDT) |
||||||
|
(envelope-from owner-pgsql-patches@postgreSQL.org) |
||||||
|
X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-patches@postgreSQL.org using -f |
||||||
|
Received: from mambo.cs.unitn.it (mambo.cs.unitn.it [193.205.199.204]) |
||||||
|
by hub.org (8.8.8/8.8.8) with SMTP id RAA06494 |
||||||
|
for <pgsql-patches@postgreSQL.org>; Wed, 14 Oct 1998 17:10:01 -0400 (EDT) |
||||||
|
(envelope-from dz@cs.unitn.it) |
||||||
|
Received: from nikita.wizard.net (ts-slip31.gelso.unitn.it [193.205.200.31]) by mambo.cs.unitn.it (8.6.12/8.6.12) with ESMTP id XAA20316 for <pgsql-patches@postgreSQL.org>; Wed, 14 Oct 1998 23:09:52 +0200 |
||||||
|
Received: (from dz@localhost) by nikita.wizard.net (8.8.5/8.6.9) id WAA00489 for pgsql-patches@postgreSQL.org; Wed, 14 Oct 1998 22:56:58 +0200 |
||||||
|
From: Massimo Dal Zotto <dz@cs.unitn.it> |
||||||
|
Message-Id: <199810142056.WAA00489@nikita.wizard.net> |
||||||
|
Subject: [PATCHES] TCL_ARRAYS |
||||||
|
To: pgsql-patches@postgreSQL.org (Pgsql Patches) |
||||||
|
Date: Wed, 14 Oct 1998 22:56:58 +0200 (MET DST) |
||||||
|
X-Mailer: ELM [version 2.4 PL24 ME4] |
||||||
|
MIME-Version: 1.0 |
||||||
|
Content-Type: text/plain; charset=iso-8859-1 |
||||||
|
Content-Transfer-Encoding: 8bit |
||||||
|
Sender: owner-pgsql-patches@postgreSQL.org |
||||||
|
Precedence: bulk |
||||||
|
Status: RO |
||||||
|
|
||||||
|
Hi, |
||||||
|
|
||||||
|
I have written this patch which fixes some problems with TCL_ARRAYS. |
||||||
|
The new array code uses a temporary buffer and is disabled by default |
||||||
|
because it depends on contrib/string-io which most of you don't use. |
||||||
|
This raises once again the problem of backslashes/escapes and various |
||||||
|
ambiguities in pgsql output. I hope this will be solved in 6.5. |
||||||
|
|
||||||
|
*** src/interfaces/libpgtcl/pgtclCmds.c.orig Mon Sep 21 09:00:19 1998 |
||||||
|
--- src/interfaces/libpgtcl/pgtclCmds.c Wed Oct 14 15:32:21 1998 |
||||||
|
*************** |
||||||
|
*** 602,616 **** |
||||||
|
{ |
||||||
|
for (i = 0; i < PQnfields(result); i++) |
||||||
|
{ |
||||||
|
sprintf(nameBuffer, "%d,%.200s", tupno, PQfname(result, i)); |
||||||
|
if (Tcl_SetVar2(interp, arrVar, nameBuffer, |
||||||
|
! #ifdef TCL_ARRAYS |
||||||
|
! tcl_value(PQgetvalue(result, tupno, i)), |
||||||
|
#else |
||||||
|
PQgetvalue(result, tupno, i), |
||||||
|
- #endif |
||||||
|
TCL_LEAVE_ERR_MSG) == NULL) |
||||||
|
return TCL_ERROR; |
||||||
|
} |
||||||
|
} |
||||||
|
Tcl_AppendResult(interp, arrVar, 0); |
||||||
|
--- 602,624 ---- |
||||||
|
{ |
||||||
|
for (i = 0; i < PQnfields(result); i++) |
||||||
|
{ |
||||||
|
+ #ifdef TCL_ARRAYS |
||||||
|
+ char *buff = strdup(PQgetvalue(result, tupno, i)); |
||||||
|
sprintf(nameBuffer, "%d,%.200s", tupno, PQfname(result, i)); |
||||||
|
if (Tcl_SetVar2(interp, arrVar, nameBuffer, |
||||||
|
! tcl_value(buff), |
||||||
|
! TCL_LEAVE_ERR_MSG) == NULL) { |
||||||
|
! free(buff); |
||||||
|
! return TCL_ERROR; |
||||||
|
! } |
||||||
|
! free(buff); |
||||||
|
#else |
||||||
|
+ sprintf(nameBuffer, "%d,%.200s", tupno, PQfname(result, i)); |
||||||
|
+ if (Tcl_SetVar2(interp, arrVar, nameBuffer, |
||||||
|
PQgetvalue(result, tupno, i), |
||||||
|
TCL_LEAVE_ERR_MSG) == NULL) |
||||||
|
return TCL_ERROR; |
||||||
|
+ #endif |
||||||
|
} |
||||||
|
} |
||||||
|
Tcl_AppendResult(interp, arrVar, 0); |
||||||
|
*************** |
||||||
|
*** 636,643 **** |
||||||
|
*/ |
||||||
|
for (tupno = 0; tupno < PQntuples(result); tupno++) |
||||||
|
{ |
||||||
|
const char *field0 = PQgetvalue(result, tupno, 0); |
||||||
|
! char * workspace = malloc(strlen(field0) + strlen(appendstr) + 210); |
||||||
|
|
||||||
|
for (i = 1; i < PQnfields(result); i++) |
||||||
|
{ |
||||||
|
--- 644,674 ---- |
||||||
|
*/ |
||||||
|
for (tupno = 0; tupno < PQntuples(result); tupno++) |
||||||
|
{ |
||||||
|
+ #ifdef TCL_ARRAYS |
||||||
|
+ char *buff = strdup(PQgetvalue(result, tupno, 0)); |
||||||
|
+ const char *field0 = tcl_value(buff); |
||||||
|
+ char *workspace = malloc(strlen(field0) + 210 + strlen(appendstr)); |
||||||
|
+ |
||||||
|
+ for (i = 1; i < PQnfields(result); i++) |
||||||
|
+ { |
||||||
|
+ free(buff); |
||||||
|
+ buff = strdup(PQgetvalue(result, tupno, i)); |
||||||
|
+ sprintf(workspace, "%s,%.200s%s", field0, PQfname(result,i), |
||||||
|
+ appendstr); |
||||||
|
+ if (Tcl_SetVar2(interp, arrVar, workspace, |
||||||
|
+ tcl_value(buff), |
||||||
|
+ TCL_LEAVE_ERR_MSG) == NULL) |
||||||
|
+ { |
||||||
|
+ free(buff); |
||||||
|
+ free(workspace); |
||||||
|
+ return TCL_ERROR; |
||||||
|
+ } |
||||||
|
+ } |
||||||
|
+ free(buff); |
||||||
|
+ free(workspace); |
||||||
|
+ #else |
||||||
|
const char *field0 = PQgetvalue(result, tupno, 0); |
||||||
|
! char *workspace = malloc(strlen(field0) + 210 + strlen(appendstr)); |
||||||
|
|
||||||
|
for (i = 1; i < PQnfields(result); i++) |
||||||
|
{ |
||||||
|
*************** |
||||||
|
*** 652,657 **** |
||||||
|
--- 683,689 ---- |
||||||
|
} |
||||||
|
} |
||||||
|
free(workspace); |
||||||
|
+ #endif |
||||||
|
} |
||||||
|
Tcl_AppendResult(interp, arrVar, 0); |
||||||
|
return TCL_OK; |
||||||
|
*************** |
||||||
|
*** 669,676 **** |
||||||
|
--- 701,716 ---- |
||||||
|
Tcl_AppendResult(interp, "argument to getTuple cannot exceed number of tuples - 1", 0); |
||||||
|
return TCL_ERROR; |
||||||
|
} |
||||||
|
+ #ifdef TCL_ARRAYS |
||||||
|
+ for (i = 0; i < PQnfields(result); i++) { |
||||||
|
+ char *buff = strdup(PQgetvalue(result, tupno, i)); |
||||||
|
+ Tcl_AppendElement(interp, tcl_value(buff)); |
||||||
|
+ free(buff); |
||||||
|
+ } |
||||||
|
+ #else |
||||||
|
for (i = 0; i < PQnfields(result); i++) |
||||||
|
Tcl_AppendElement(interp, PQgetvalue(result, tupno, i)); |
||||||
|
+ #endif |
||||||
|
return TCL_OK; |
||||||
|
} |
||||||
|
else if (strcmp(opt, "-tupleArray") == 0) |
||||||
|
*************** |
||||||
|
*** 688,697 **** |
||||||
|
--- 728,748 ---- |
||||||
|
} |
||||||
|
for (i = 0; i < PQnfields(result); i++) |
||||||
|
{ |
||||||
|
+ #ifdef TCL_ARRAYS |
||||||
|
+ char *buff = strdup(PQgetvalue(result, tupno, i)); |
||||||
|
+ if (Tcl_SetVar2(interp, argv[4], PQfname(result, i), |
||||||
|
+ tcl_value(buff), |
||||||
|
+ TCL_LEAVE_ERR_MSG) == NULL) { |
||||||
|
+ free(buff); |
||||||
|
+ return TCL_ERROR; |
||||||
|
+ } |
||||||
|
+ free(buff); |
||||||
|
+ #else |
||||||
|
if (Tcl_SetVar2(interp, argv[4], PQfname(result, i), |
||||||
|
PQgetvalue(result, tupno, i), |
||||||
|
TCL_LEAVE_ERR_MSG) == NULL) |
||||||
|
return TCL_ERROR; |
||||||
|
+ #endif |
||||||
|
} |
||||||
|
return TCL_OK; |
||||||
|
} |
||||||
|
*************** |
||||||
|
*** 1303,1310 **** |
||||||
|
sprintf(buffer, "%d", tupno); |
||||||
|
Tcl_SetVar2(interp, argv[3], ".tupno", buffer, 0); |
||||||
|
|
||||||
|
for (column = 0; column < ncols; column++) |
||||||
|
! Tcl_SetVar2(interp, argv[3], info[column].cname, PQgetvalue(result, tupno, column), 0); |
||||||
|
|
||||||
|
Tcl_SetVar2(interp, argv[3], ".command", "update", 0); |
||||||
|
|
||||||
|
--- 1354,1371 ---- |
||||||
|
sprintf(buffer, "%d", tupno); |
||||||
|
Tcl_SetVar2(interp, argv[3], ".tupno", buffer, 0); |
||||||
|
|
||||||
|
+ #ifdef TCL_ARRAYS |
||||||
|
+ for (column = 0; column < ncols; column++) { |
||||||
|
+ char *buff = strdup(PQgetvalue(result, tupno, column)); |
||||||
|
+ Tcl_SetVar2(interp, argv[3], info[column].cname, |
||||||
|
+ tcl_value(buff), 0); |
||||||
|
+ free(buff); |
||||||
|
+ } |
||||||
|
+ #else |
||||||
|
for (column = 0; column < ncols; column++) |
||||||
|
! Tcl_SetVar2(interp, argv[3], info[column].cname, |
||||||
|
! PQgetvalue(result, tupno, column), 0); |
||||||
|
! #endif |
||||||
|
|
||||||
|
Tcl_SetVar2(interp, argv[3], ".command", "update", 0); |
||||||
|
|
||||||
|
*** src/include/config.h.in.orig Wed Aug 26 09:01:16 1998 |
||||||
|
--- src/include/config.h.in Wed Oct 14 22:44:00 1998 |
||||||
|
*************** |
||||||
|
*** 312,318 **** |
||||||
|
* of postgres C-like arrays, for example {{"a1" "a2"} {"b1" "b2"}} instead |
||||||
|
* of {{"a1","a2"},{"b1","b2"}}. |
||||||
|
*/ |
||||||
|
! #define TCL_ARRAYS |
||||||
|
|
||||||
|
/* |
||||||
|
* The following flag allows limiting the number of rows returned by a query. |
||||||
|
--- 312,318 ---- |
||||||
|
* of postgres C-like arrays, for example {{"a1" "a2"} {"b1" "b2"}} instead |
||||||
|
* of {{"a1","a2"},{"b1","b2"}}. |
||||||
|
*/ |
||||||
|
! /* #define TCL_ARRAYS */ |
||||||
|
|
||||||
|
/* |
||||||
|
* The following flag allows limiting the number of rows returned by a query. |
||||||
|
|
||||||
|
-- |
||||||
|
Massimo Dal Zotto |
||||||
|
|
||||||
|
+----------------------------------------------------------------------+ |
||||||
|
| Massimo Dal Zotto email: dz@cs.unitn.it | |
||||||
|
| Via Marconi, 141 phone: ++39-461-534251 | |
||||||
|
| 38057 Pergine Valsugana (TN) www: http://www.cs.unitn.it/~dz/ | |
||||||
|
| Italy pgp: finger dz@tango.cs.unitn.it | |
||||||
|
+----------------------------------------------------------------------+ |
||||||
|
|
||||||
|
|
||||||
Loading…
Reference in new issue