mirror of https://github.com/postgres/postgres
Bruce Momjian
25 years ago
parent
da0e6bfeaf
commit
99f964fcc6
@ -1,519 +0,0 @@ |
||||
From owner-pgsql-hackers@hub.org Wed Sep 22 20:31:02 1999 |
||||
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id UAA15611 |
||||
for <maillist@candle.pha.pa.us>; Wed, 22 Sep 1999 20:31:01 -0400 (EDT) |
||||
Received: from hub.org (hub.org [216.126.84.1]) by renoir.op.net (o1/$ Revision: 1.18 $) with ESMTP id UAA02926 for <maillist@candle.pha.pa.us>; Wed, 22 Sep 1999 20:21:24 -0400 (EDT) |
||||
Received: from hub.org (hub.org [216.126.84.1]) |
||||
by hub.org (8.9.3/8.9.3) with ESMTP id UAA75413; |
||||
Wed, 22 Sep 1999 20:09:35 -0400 (EDT) |
||||
(envelope-from owner-pgsql-hackers@hub.org) |
||||
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 22 Sep 1999 20:08:50 +0000 (EDT) |
||||
Received: (from majordom@localhost) |
||||
by hub.org (8.9.3/8.9.3) id UAA75058 |
||||
for pgsql-hackers-outgoing; Wed, 22 Sep 1999 20:06:58 -0400 (EDT) |
||||
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||
Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) |
||||
by hub.org (8.9.3/8.9.3) with ESMTP id UAA74982 |
||||
for <pgsql-hackers@postgreSQL.org>; Wed, 22 Sep 1999 20:06:25 -0400 (EDT) |
||||
(envelope-from tgl@sss.pgh.pa.us) |
||||
Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) |
||||
by sss.sss.pgh.pa.us (8.9.1/8.9.1) with ESMTP id UAA06411 |
||||
for <pgsql-hackers@postgreSQL.org>; Wed, 22 Sep 1999 20:05:40 -0400 (EDT) |
||||
To: pgsql-hackers@postgreSQL.org |
||||
Subject: [HACKERS] Progress report: buffer refcount bugs and SQL functions |
||||
Date: Wed, 22 Sep 1999 20:05:39 -0400 |
||||
Message-ID: <6408.938045139@sss.pgh.pa.us> |
||||
From: Tom Lane <tgl@sss.pgh.pa.us> |
||||
Sender: owner-pgsql-hackers@postgreSQL.org |
||||
Precedence: bulk |
||||
Status: RO |
||||
|
||||
I have been finding a lot of interesting stuff while looking into |
||||
the buffer reference count/leakage issue. |
||||
|
||||
It turns out that there were two specific things that were camouflaging |
||||
the existence of bugs in this area: |
||||
|
||||
1. The BufferLeakCheck routine that's run at transaction commit was |
||||
only looking for nonzero PrivateRefCount to indicate a missing unpin. |
||||
It failed to notice nonzero LastRefCount --- which meant that an |
||||
error in refcount save/restore usage could leave a buffer pinned, |
||||
and BufferLeakCheck wouldn't notice. |
||||
|
||||
2. The BufferIsValid macro, which you'd think just checks whether |
||||
it's handed a valid buffer identifier or not, actually did more: |
||||
it only returned true if the buffer ID was valid *and* the buffer |
||||
had positive PrivateRefCount. That meant that the common pattern |
||||
if (BufferIsValid(buf)) |
||||
ReleaseBuffer(buf); |
||||
wouldn't complain if it were handed a valid but already unpinned buffer. |
||||
And that behavior masks bugs that result in buffers being unpinned too |
||||
early. For example, consider a sequence like |
||||
|
||||
1. LockBuffer (buffer now has refcount 1). Store reference to |
||||
a tuple on that buffer page in a tuple table slot. |
||||
2. Copy buffer reference to a second tuple-table slot, but forget to |
||||
increment buffer's refcount. |
||||
3. Release second tuple table slot. Buffer refcount drops to 0, |
||||
so it's unpinned. |
||||
4. Release original tuple slot. Because of BufferIsValid behavior, |
||||
no assert happens here; in fact nothing at all happens. |
||||
|
||||
This is, of course, buggy code: during the interval from 3 to 4 you |
||||
still have an apparently valid tuple reference in the original slot, |
||||
which someone might try to use; but the buffer it points to is unpinned |
||||
and could be replaced at any time by another backend. |
||||
|
||||
In short, we had errors that would mask both missing-pin bugs and |
||||
missing-unpin bugs. And naturally there were a few such bugs lurking |
||||
behind them... |
||||
|
||||
3. The buffer refcount save/restore stuff, which I had suspected |
||||
was useless, is not only useless but also buggy. The reason it's |
||||
buggy is that it only works if used in a nested fashion. You could |
||||
save state A, pin some buffers, save state B, pin some more |
||||
buffers, restore state B (thereby unpinning what you pinned since |
||||
the save), and finally restore state A (unpinning the earlier stuff). |
||||
What you could not do is save state A, pin, save B, pin more, then |
||||
restore state A --- that might unpin some of A's buffers, or some |
||||
of B's buffers, or some unforeseen combination thereof. If you |
||||
restore A and then restore B, you do not necessarily return to a zero- |
||||
pins state, either. And it turns out the actual usage pattern was a |
||||
nearly random sequence of saves and restores, compounded by a failure to |
||||
do all of the restores reliably (which was masked by the oversight in |
||||
BufferLeakCheck). |
||||
|
||||
|
||||
What I have done so far is to rip out the buffer refcount save/restore |
||||
support (including LastRefCount), change BufferIsValid to a simple |
||||
validity check (so that you get an assert if you unpin something that |
||||
was pinned), change ExecStoreTuple so that it increments the refcount |
||||
when it is handed a buffer reference (for symmetry with ExecClearTuple's |
||||
decrement of the refcount), and fix about a dozen bugs exposed by these |
||||
changes. |
||||
|
||||
I am still getting Buffer Leak notices in the "misc" regression test, |
||||
specifically in the queries that invoke more than one SQL function. |
||||
What I find there is that SQL functions are not always run to |
||||
completion. Apparently, when a function can return multiple tuples, |
||||
it won't necessarily be asked to produce them all. And when it isn't, |
||||
postquel_end() isn't invoked for the function's current query, so its |
||||
tuple table isn't cleared, so we have dangling refcounts if any of the |
||||
tuples involved are in disk buffers. |
||||
|
||||
It may be that the save/restore code was a misguided attempt to fix |
||||
this problem. I can't tell. But I think what we really need to do is |
||||
find some way of ensuring that Postquel function execution contexts |
||||
always get shut down by the end of the query, so that they don't leak |
||||
resources. |
||||
|
||||
I suppose a straightforward approach would be to keep a list of open |
||||
function contexts somewhere (attached to the outer execution context, |
||||
perhaps), and clean them up at outer-plan shutdown. |
||||
|
||||
What I am wondering, though, is whether this addition is actually |
||||
necessary, or is it a bug that the functions aren't run to completion |
||||
in the first place? I don't really understand the semantics of this |
||||
"nested dot notation". I suppose it is a Berkeleyism; I can't find |
||||
anything about it in the SQL92 document. The test cases shown in the |
||||
misc regress test seem peculiar, not to say wrong. For example: |
||||
|
||||
regression=> SELECT p.hobbies.equipment.name, p.hobbies.name, p.name FROM person p; |
||||
name |name |name |
||||
-------------+-----------+----- |
||||
advil |posthacking|mike |
||||
peet's coffee|basketball |joe |
||||
hightops |basketball |sally |
||||
(3 rows) |
||||
|
||||
which doesn't appear to agree with the contents of the underlying |
||||
relations: |
||||
|
||||
regression=> SELECT * FROM hobbies_r; |
||||
name |person |
||||
-----------+------ |
||||
posthacking|mike |
||||
posthacking|jeff |
||||
basketball |joe |
||||
basketball |sally |
||||
skywalking | |
||||
(5 rows) |
||||
|
||||
regression=> SELECT * FROM equipment_r; |
||||
name |hobby |
||||
-------------+----------- |
||||
advil |posthacking |
||||
peet's coffee|posthacking |
||||
hightops |basketball |
||||
guts |skywalking |
||||
(4 rows) |
||||
|
||||
I'd have expected an output along the lines of |
||||
|
||||
advil |posthacking|mike |
||||
peet's coffee|posthacking|mike |
||||
hightops |basketball |joe |
||||
hightops |basketball |sally |
||||
|
||||
Is the regression test's expected output wrong, or am I misunderstanding |
||||
what this query is supposed to do? Is there any documentation anywhere |
||||
about how SQL functions returning multiple tuples are supposed to |
||||
behave? |
||||
|
||||
regards, tom lane |
||||
|
||||
************ |
||||
|
||||
|
||||
From owner-pgsql-hackers@hub.org Thu Sep 23 11:03:19 1999 |
||||
Received: from hub.org (hub.org [216.126.84.1]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id LAA16211 |
||||
for <maillist@candle.pha.pa.us>; Thu, 23 Sep 1999 11:03:17 -0400 (EDT) |
||||
Received: from hub.org (hub.org [216.126.84.1]) |
||||
by hub.org (8.9.3/8.9.3) with ESMTP id KAA58151; |
||||
Thu, 23 Sep 1999 10:53:46 -0400 (EDT) |
||||
(envelope-from owner-pgsql-hackers@hub.org) |
||||
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Thu, 23 Sep 1999 10:53:05 +0000 (EDT) |
||||
Received: (from majordom@localhost) |
||||
by hub.org (8.9.3/8.9.3) id KAA57948 |
||||
for pgsql-hackers-outgoing; Thu, 23 Sep 1999 10:52:23 -0400 (EDT) |
||||
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||
Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) |
||||
by hub.org (8.9.3/8.9.3) with ESMTP id KAA57841 |
||||
for <hackers@postgreSQL.org>; Thu, 23 Sep 1999 10:51:50 -0400 (EDT) |
||||
(envelope-from tgl@sss.pgh.pa.us) |
||||
Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) |
||||
by sss.sss.pgh.pa.us (8.9.1/8.9.1) with ESMTP id KAA14211; |
||||
Thu, 23 Sep 1999 10:51:10 -0400 (EDT) |
||||
To: Andreas Zeugswetter <andreas.zeugswetter@telecom.at> |
||||
cc: hackers@postgreSQL.org |
||||
Subject: Re: [HACKERS] Progress report: buffer refcount bugs and SQL functions |
||||
In-reply-to: Your message of Thu, 23 Sep 1999 10:07:24 +0200 |
||||
<37E9DFBC.5C0978F@telecom.at> |
||||
Date: Thu, 23 Sep 1999 10:51:10 -0400 |
||||
Message-ID: <14209.938098270@sss.pgh.pa.us> |
||||
From: Tom Lane <tgl@sss.pgh.pa.us> |
||||
Sender: owner-pgsql-hackers@postgreSQL.org |
||||
Precedence: bulk |
||||
Status: RO |
||||
|
||||
Andreas Zeugswetter <andreas.zeugswetter@telecom.at> writes: |
||||
> That is what I use it for. I have never used it with a |
||||
> returns setof function, but reading the comments in the regression test, |
||||
> -- mike needs advil and peet's coffee, |
||||
> -- joe and sally need hightops, and |
||||
> -- everyone else is fine. |
||||
> it looks like the results you expected are correct, and currently the |
||||
> wrong result is given. |
||||
|
||||
Yes, I have concluded the same (and partially fixed it, per my previous |
||||
message). |
||||
|
||||
> Those that don't have a hobbie should return name|NULL|NULL. A hobbie |
||||
> that does'nt need equipment name|hobbie|NULL. |
||||
|
||||
That's a good point. Currently (both with and without my uncommitted |
||||
fix) you get *no* rows out from ExecTargetList if there are any Iters |
||||
that return empty result sets. It might be more reasonable to treat an |
||||
empty result set as if it were NULL, which would give the behavior you |
||||
suggest. |
||||
|
||||
This would be an easy change to my current patch, and I'm prepared to |
||||
make it before committing what I have, if people agree that that's a |
||||
more reasonable definition. Comments? |
||||
|
||||
regards, tom lane |
||||
|
||||
************ |
||||
|
||||
|
||||
From owner-pgsql-hackers@hub.org Thu Sep 23 04:31:15 1999 |
||||
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id EAA11344 |
||||
for <maillist@candle.pha.pa.us>; Thu, 23 Sep 1999 04:31:15 -0400 (EDT) |
||||
Received: from hub.org (hub.org [216.126.84.1]) by renoir.op.net (o1/$ Revision: 1.18 $) with ESMTP id EAA05350 for <maillist@candle.pha.pa.us>; Thu, 23 Sep 1999 04:24:29 -0400 (EDT) |
||||
Received: from hub.org (hub.org [216.126.84.1]) |
||||
by hub.org (8.9.3/8.9.3) with ESMTP id EAA85679; |
||||
Thu, 23 Sep 1999 04:16:26 -0400 (EDT) |
||||
(envelope-from owner-pgsql-hackers@hub.org) |
||||
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Thu, 23 Sep 1999 04:09:52 +0000 (EDT) |
||||
Received: (from majordom@localhost) |
||||
by hub.org (8.9.3/8.9.3) id EAA84708 |
||||
for pgsql-hackers-outgoing; Thu, 23 Sep 1999 04:08:57 -0400 (EDT) |
||||
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||
Received: from gandalf.telecom.at (gandalf.telecom.at [194.118.26.84]) |
||||
by hub.org (8.9.3/8.9.3) with ESMTP id EAA84632 |
||||
for <hackers@postgresql.org>; Thu, 23 Sep 1999 04:08:03 -0400 (EDT) |
||||
(envelope-from andreas.zeugswetter@telecom.at) |
||||
Received: from telecom.at (w0188000580.f000.d0188.sd.spardat.at [172.18.65.249]) |
||||
by gandalf.telecom.at (xxx/xxx) with ESMTP id KAA195294 |
||||
for <hackers@postgresql.org>; Thu, 23 Sep 1999 10:07:27 +0200 |
||||
Message-ID: <37E9DFBC.5C0978F@telecom.at> |
||||
Date: Thu, 23 Sep 1999 10:07:24 +0200 |
||||
From: Andreas Zeugswetter <andreas.zeugswetter@telecom.at> |
||||
X-Mailer: Mozilla 4.61 [en] (Win95; I) |
||||
X-Accept-Language: en |
||||
MIME-Version: 1.0 |
||||
To: hackers@postgreSQL.org |
||||
Subject: Re: [HACKERS] Progress report: buffer refcount bugs and SQL functions |
||||
Content-Type: text/plain; charset=us-ascii |
||||
Content-Transfer-Encoding: 7bit |
||||
Sender: owner-pgsql-hackers@postgreSQL.org |
||||
Precedence: bulk |
||||
Status: RO |
||||
|
||||
> Is the regression test's expected output wrong, or am I |
||||
> misunderstanding |
||||
> what this query is supposed to do? Is there any |
||||
> documentation anywhere |
||||
> about how SQL functions returning multiple tuples are supposed to |
||||
> behave? |
||||
|
||||
They are supposed to behave somewhat like a view. |
||||
Not all rows are necessarily fetched. |
||||
If used in a context that needs a single row answer, |
||||
and the answer has multiple rows it is supposed to |
||||
runtime elog. Like in: |
||||
|
||||
select * from tbl where col=funcreturningmultipleresults(); |
||||
-- this must elog |
||||
|
||||
while this is ok: |
||||
select * from tbl where col in (select funcreturningmultipleresults()); |
||||
|
||||
But the caller could only fetch the first row if he wanted. |
||||
|
||||
The nested notation is supposed to call the function passing it the tuple |
||||
as the first argument. This is what can be used to "fake" a column |
||||
onto a table (computed column). |
||||
That is what I use it for. I have never used it with a |
||||
returns setof function, but reading the comments in the regression test, |
||||
-- mike needs advil and peet's coffee, |
||||
-- joe and sally need hightops, and |
||||
-- everyone else is fine. |
||||
it looks like the results you expected are correct, and currently the |
||||
wrong result is given. |
||||
|
||||
But I think this query could also elog whithout removing substantial |
||||
functionality. |
||||
|
||||
SELECT p.name, p.hobbies.name, p.hobbies.equipment.name FROM person p; |
||||
|
||||
Actually for me it would be intuitive, that this query return one row per |
||||
person, but elog on those that have more than one hobbie or a hobbie that |
||||
needs more than one equipment. Those that don't have a hobbie should |
||||
return name|NULL|NULL. A hobbie that does'nt need equipment name|hobbie|NULL. |
||||
|
||||
Andreas |
||||
|
||||
************ |
||||
|
||||
|
||||
From owner-pgsql-hackers@hub.org Wed Sep 22 22:01:07 1999 |
||||
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id WAA16360 |
||||
for <maillist@candle.pha.pa.us>; Wed, 22 Sep 1999 22:01:05 -0400 (EDT) |
||||
Received: from hub.org (hub.org [216.126.84.1]) by renoir.op.net (o1/$ Revision: 1.18 $) with ESMTP id VAA08386 for <maillist@candle.pha.pa.us>; Wed, 22 Sep 1999 21:37:24 -0400 (EDT) |
||||
Received: from hub.org (hub.org [216.126.84.1]) |
||||
by hub.org (8.9.3/8.9.3) with ESMTP id VAA88083; |
||||
Wed, 22 Sep 1999 21:28:11 -0400 (EDT) |
||||
(envelope-from owner-pgsql-hackers@hub.org) |
||||
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 22 Sep 1999 21:27:48 +0000 (EDT) |
||||
Received: (from majordom@localhost) |
||||
by hub.org (8.9.3/8.9.3) id VAA87938 |
||||
for pgsql-hackers-outgoing; Wed, 22 Sep 1999 21:26:52 -0400 (EDT) |
||||
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||
Received: from orion.SAPserv.Hamburg.dsh.de (Tpolaris2.sapham.debis.de [53.2.131.8]) |
||||
by hub.org (8.9.3/8.9.3) with SMTP id VAA87909 |
||||
for <pgsql-hackers@postgresql.org>; Wed, 22 Sep 1999 21:26:36 -0400 (EDT) |
||||
(envelope-from wieck@debis.com) |
||||
Received: by orion.SAPserv.Hamburg.dsh.de |
||||
for pgsql-hackers@postgresql.org |
||||
id m11TxXw-0003kLC; Thu, 23 Sep 99 03:19 MET DST |
||||
Message-Id: <m11TxXw-0003kLC@orion.SAPserv.Hamburg.dsh.de> |
||||
From: wieck@debis.com (Jan Wieck) |
||||
Subject: Re: [HACKERS] Progress report: buffer refcount bugs and SQL functions |
||||
To: tgl@sss.pgh.pa.us (Tom Lane) |
||||
Date: Thu, 23 Sep 1999 03:19:39 +0200 (MET DST) |
||||
Cc: pgsql-hackers@postgreSQL.org |
||||
Reply-To: wieck@debis.com (Jan Wieck) |
||||
In-Reply-To: <6408.938045139@sss.pgh.pa.us> from "Tom Lane" at Sep 22, 99 08:05:39 pm |
||||
X-Mailer: ELM [version 2.4 PL25] |
||||
Content-Type: text |
||||
Sender: owner-pgsql-hackers@postgreSQL.org |
||||
Precedence: bulk |
||||
Status: RO |
||||
|
||||
Tom Lane wrote: |
||||
|
||||
> [...] |
||||
> |
||||
> What I am wondering, though, is whether this addition is actually |
||||
> necessary, or is it a bug that the functions aren't run to completion |
||||
> in the first place? I don't really understand the semantics of this |
||||
> "nested dot notation". I suppose it is a Berkeleyism; I can't find |
||||
> anything about it in the SQL92 document. The test cases shown in the |
||||
> misc regress test seem peculiar, not to say wrong. For example: |
||||
> |
||||
> [...] |
||||
> |
||||
> Is the regression test's expected output wrong, or am I misunderstanding |
||||
> what this query is supposed to do? Is there any documentation anywhere |
||||
> about how SQL functions returning multiple tuples are supposed to |
||||
> behave? |
||||
|
||||
I've said some time (maybe too long) ago, that SQL functions |
||||
returning tuple sets are broken in general. This nested dot |
||||
notation (which I think is an artefact from the postquel |
||||
querylanguage) is implemented via set functions. |
||||
|
||||
Set functions have total different semantics from all other |
||||
functions. First they don't really return a tuple set as |
||||
someone might think - all that screwed up code instead |
||||
simulates that they return something you could consider a |
||||
scan of the last SQL statement in the function. Then, on |
||||
each subsequent call inside of the same command, they return |
||||
a "tupletable slot" containing the next found tuple (that's |
||||
why their Func node is mangled up after the first call). |
||||
|
||||
Second they have a targetlist what I think was originally |
||||
intended to extract attributes out of the tuples returned |
||||
when the above scan is asked to get the next tuple. But as I |
||||
read the code it invokes the function again and this might |
||||
cause the resource leakage you see. |
||||
|
||||
Third, all this seems to never have been implemented |
||||
(thought?) to the end. A targetlist doesn't make sense at |
||||
this place because it could at max contain a single attribute |
||||
- so a single attno would have the same power. And if set |
||||
functions could appear in the rangetable (FROM clause), than |
||||
they would be treated as that and regular Var nodes in the |
||||
query would do it. |
||||
|
||||
I think you shouldn't really care for that regression test |
||||
and maybe we should disable set functions until we really |
||||
implement stored procedures returning sets in the rangetable. |
||||
|
||||
Set functions where planned by Stonebraker's team as |
||||
something that today is called stored procedures. But AFAIK |
||||
they never reached the useful state because even in Postgres |
||||
4.2 you haven't been able to get more than one attribute out |
||||
of a set function. It was a feature of the postquel |
||||
querylanguage that you could get one attribute from a set |
||||
function via |
||||
|
||||
RETRIEVE (attributename(setfuncname())) |
||||
|
||||
While working on the constraint triggers I've came across |
||||
another regression test (triggers :-) that's errorneous too. |
||||
The funny_dup17 trigger proc executes an INSERT into the same |
||||
relation where it get fired for by a previous INSERT. And it |
||||
stops this recursion only if it reaches a nesting level of |
||||
17, which could only occur if it is fired DURING the |
||||
execution of it's own SPI_exec(). After Vadim quouted some |
||||
SQL92 definitions about when constraint checks and triggers |
||||
are to be executed, I decided to fire regular triggers at the |
||||
end of a query too. Thus, there is absolutely no nesting |
||||
possible for AFTER triggers resulting in an endless loop. |
||||
|
||||
|
||||
Jan |
||||
|
||||
-- |
||||
|
||||
#======================================================================# |
||||
# It's easier to get forgiveness for being wrong than for being right. # |
||||
# Let's break this rule - forgive me. # |
||||
#========================================= wieck@debis.com (Jan Wieck) # |
||||
|
||||
|
||||
|
||||
************ |
||||
|
||||
|
||||
From owner-pgsql-hackers@hub.org Thu Sep 23 11:01:06 1999 |
||||
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id LAA16162 |
||||
for <maillist@candle.pha.pa.us>; Thu, 23 Sep 1999 11:01:04 -0400 (EDT) |
||||
Received: from hub.org (hub.org [216.126.84.1]) by renoir.op.net (o1/$ Revision: 1.18 $) with ESMTP id KAA28544 for <maillist@candle.pha.pa.us>; Thu, 23 Sep 1999 10:45:54 -0400 (EDT) |
||||
Received: from hub.org (hub.org [216.126.84.1]) |
||||
by hub.org (8.9.3/8.9.3) with ESMTP id KAA52943; |
||||
Thu, 23 Sep 1999 10:20:51 -0400 (EDT) |
||||
(envelope-from owner-pgsql-hackers@hub.org) |
||||
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Thu, 23 Sep 1999 10:19:58 +0000 (EDT) |
||||
Received: (from majordom@localhost) |
||||
by hub.org (8.9.3/8.9.3) id KAA52472 |
||||
for pgsql-hackers-outgoing; Thu, 23 Sep 1999 10:19:03 -0400 (EDT) |
||||
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||
Received: from sss.sss.pgh.pa.us (sss.pgh.pa.us [209.114.166.2]) |
||||
by hub.org (8.9.3/8.9.3) with ESMTP id KAA52431 |
||||
for <pgsql-hackers@postgresql.org>; Thu, 23 Sep 1999 10:18:47 -0400 (EDT) |
||||
(envelope-from tgl@sss.pgh.pa.us) |
||||
Received: from sss.sss.pgh.pa.us (localhost [127.0.0.1]) |
||||
by sss.sss.pgh.pa.us (8.9.1/8.9.1) with ESMTP id KAA13253; |
||||
Thu, 23 Sep 1999 10:18:02 -0400 (EDT) |
||||
To: wieck@debis.com (Jan Wieck) |
||||
cc: pgsql-hackers@postgreSQL.org |
||||
Subject: Re: [HACKERS] Progress report: buffer refcount bugs and SQL functions |
||||
In-reply-to: Your message of Thu, 23 Sep 1999 03:19:39 +0200 (MET DST) |
||||
<m11TxXw-0003kLC@orion.SAPserv.Hamburg.dsh.de> |
||||
Date: Thu, 23 Sep 1999 10:18:01 -0400 |
||||
Message-ID: <13251.938096281@sss.pgh.pa.us> |
||||
From: Tom Lane <tgl@sss.pgh.pa.us> |
||||
Sender: owner-pgsql-hackers@postgreSQL.org |
||||
Precedence: bulk |
||||
Status: RO |
||||
|
||||
wieck@debis.com (Jan Wieck) writes: |
||||
> Tom Lane wrote: |
||||
>> What I am wondering, though, is whether this addition is actually |
||||
>> necessary, or is it a bug that the functions aren't run to completion |
||||
>> in the first place? |
||||
|
||||
> I've said some time (maybe too long) ago, that SQL functions |
||||
> returning tuple sets are broken in general. |
||||
|
||||
Indeed they are. Try this on for size (using the regression database): |
||||
|
||||
SELECT p.name, p.hobbies.equipment.name FROM person p; |
||||
SELECT p.hobbies.equipment.name, p.name FROM person p; |
||||
|
||||
You get different result sets!? |
||||
|
||||
The problem in this example is that ExecTargetList returns the isDone |
||||
flag from the last targetlist entry, regardless of whether there are |
||||
incomplete iterations in previous entries. More generally, the buffer |
||||
leak problem that I started with only occurs if some Iter nodes are not |
||||
run to completion --- but execQual.c has no mechanism to make sure that |
||||
they have all reached completion simultaneously. |
||||
|
||||
What we really need to make functions-returning-sets work properly is |
||||
an implementation somewhat like aggregate functions. We need to make |
||||
a list of all the Iter nodes present in a targetlist and cycle through |
||||
the values returned by each in a methodical fashion (run the rightmost |
||||
through its full cycle, then advance the next-to-rightmost one value, |
||||
run the rightmost through its cycle again, etc etc). Also there needs |
||||
to be an understanding of the hierarchy when an Iter appears in the |
||||
arguments of another Iter's function. (You cycle the upper one for |
||||
*each* set of arguments created by cycling its sub-Iters.) |
||||
|
||||
I am not particularly interested in working on this feature right now, |
||||
since AFAIK it's a Berkeleyism not found in SQL92. What I've done |
||||
is to hack ExecTargetList so that it behaves semi-sanely when there's |
||||
more than one Iter at the top level of the target list --- it still |
||||
doesn't really give the right answer, but at least it will keep |
||||
generating tuples until all the Iters are done at the same time. |
||||
It happens that that's enough to give correct answers for the examples |
||||
shown in the misc regress test. Even when it fails to generate all |
||||
the possible combinations, there will be no buffer leaks. |
||||
|
||||
So, I'm going to declare victory and go home ;-). We ought to add a |
||||
TODO item along the lines of |
||||
* Functions returning sets don't really work right |
||||
in hopes that someone will feel like tackling this someday. |
||||
|
||||
regards, tom lane |
||||
|
||||
************ |
||||
|
||||
|
||||
@ -1,285 +0,0 @@ |
||||
From owner-pgsql-hackers@hub.org Fri Nov 13 13:24:37 1998 |
||||
Received: from hub.org (majordom@hub.org [209.47.148.200]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA13457 |
||||
for <maillist@candle.pha.pa.us>; Fri, 13 Nov 1998 13:24:35 -0500 (EST) |
||||
Received: from localhost (majordom@localhost) |
||||
by hub.org (8.9.1/8.9.1) with SMTP id NAA02464; |
||||
Fri, 13 Nov 1998 13:22:52 -0500 (EST) |
||||
(envelope-from owner-pgsql-hackers@hub.org) |
||||
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Fri, 13 Nov 1998 13:21:14 +0000 (EST) |
||||
Received: (from majordom@localhost) |
||||
by hub.org (8.9.1/8.9.1) id NAA02331 |
||||
for pgsql-hackers-outgoing; Fri, 13 Nov 1998 13:21:12 -0500 (EST) |
||||
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||
Received: from orion.SAPserv.Hamburg.dsh.de (Tpolaris2.sapham.debis.de [53.2.131.8]) |
||||
by hub.org (8.9.1/8.9.1) with SMTP id NAA02316 |
||||
for <pgsql-hackers@postgreSQL.org>; Fri, 13 Nov 1998 13:21:06 -0500 (EST) |
||||
(envelope-from wieck@sapserv.debis.de) |
||||
Received: by orion.SAPserv.Hamburg.dsh.de |
||||
for pgsql-hackers@postgreSQL.org |
||||
id m0zeOEf-000EBPC; Fri, 13 Nov 98 19:46 MET |
||||
Message-Id: <m0zeOEf-000EBPC@orion.SAPserv.Hamburg.dsh.de> |
||||
From: jwieck@debis.com (Jan Wieck) |
||||
Subject: [HACKERS] shmem limits and redolog |
||||
To: pgsql-hackers@postgreSQL.org (PostgreSQL HACKERS) |
||||
Date: Fri, 13 Nov 1998 19:46:20 +0100 (MET) |
||||
Reply-To: jwieck@debis.com (Jan Wieck) |
||||
X-Mailer: ELM [version 2.4 PL25] |
||||
Content-Type: text |
||||
Sender: owner-pgsql-hackers@postgreSQL.org |
||||
Precedence: bulk |
||||
Status: ROr |
||||
|
||||
Hi, |
||||
|
||||
I'm currently hacking around on a solution for logging all |
||||
database operations at query level that can recover a crashed |
||||
database from the last successful backup by redoing all the |
||||
commands. |
||||
|
||||
Well, I wanted it to be as flexible as can. So I decided to |
||||
make it per database configurable. One could say which |
||||
databases are logged and if a database is, if it is logged |
||||
sync or async (in sync mode, every COMMIT forces an fsync of |
||||
the actual logfile and controlfiles). |
||||
|
||||
To make async mode as fast as can, I'm using a shared memory |
||||
of 32K per database (not per backend) that is used as a wrap |
||||
around buffer from the backends to place their query |
||||
information. So the log writer can fall a little behind if |
||||
there are many backends doing different things that don't |
||||
lock each other. |
||||
|
||||
Now I'm a little in doubt about the shared memory limits |
||||
reported. Was it a good decision to use shared memory? Am I |
||||
better off using socket's? |
||||
|
||||
The bad thing in what I have up to now (it's far from |
||||
complete) is, that even if a database isn't currently logged, |
||||
a redolog writer is started and creates the 32K shmem segment |
||||
(plus a semaphore set with 5 semaphores). This is because I |
||||
plan to create commands like |
||||
|
||||
ALTER DATABASE LOG MODE=ASYNC LOGDIR='/somewhere/dbname'; |
||||
|
||||
and the like that can be used at runtime (while more than one |
||||
backend is connected to the database) to turn logging on/off, |
||||
switch to/from backup mode (all other activity is stopped) |
||||
etc. |
||||
|
||||
So every 32 databases will require another megabyte of shared |
||||
memory. The logging master controls which databases have |
||||
activity and kills redolog writers after some time of |
||||
inactivity, and the shmem is freed then. But it can hurt if |
||||
someone really has many many databases that are all used at |
||||
the same time. |
||||
|
||||
What do the others say? |
||||
|
||||
|
||||
Jan |
||||
|
||||
-- |
||||
|
||||
#======================================================================# |
||||
# It's easier to get forgiveness for being wrong than for being right. # |
||||
# Let's break this rule - forgive me. # |
||||
#======================================== jwieck@debis.com (Jan Wieck) # |
||||
|
||||
|
||||
|
||||
|
||||
From owner-pgsql-hackers@hub.org Wed Dec 16 15:46:41 1998 |
||||
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id PAA00521 |
||||
for <maillist@candle.pha.pa.us>; Wed, 16 Dec 1998 15:46:40 -0500 (EST) |
||||
Received: from hub.org (majordom@hub.org [209.47.145.100]) by renoir.op.net (o1/$ Revision: 1.18 $) with ESMTP id PAA08772 for <maillist@candle.pha.pa.us>; Wed, 16 Dec 1998 15:10:01 -0500 (EST) |
||||
Received: from localhost (majordom@localhost) |
||||
by hub.org (8.9.1/8.9.1) with SMTP id PAA01254; |
||||
Wed, 16 Dec 1998 15:06:56 -0500 (EST) |
||||
(envelope-from owner-pgsql-hackers@hub.org) |
||||
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 16 Dec 1998 14:58:11 +0000 (EST) |
||||
Received: (from majordom@localhost) |
||||
by hub.org (8.9.1/8.9.1) id OAA00660 |
||||
for pgsql-hackers-outgoing; Wed, 16 Dec 1998 14:58:10 -0500 (EST) |
||||
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||
Received: from orion.SAPserv.Hamburg.dsh.de (Tpolaris2.sapham.debis.de [53.2.131.8]) |
||||
by hub.org (8.9.1/8.9.1) with SMTP id OAA00643 |
||||
for <pgsql-hackers@postgreSQL.org>; Wed, 16 Dec 1998 14:58:05 -0500 (EST) |
||||
(envelope-from wieck@sapserv.debis.de) |
||||
Received: by orion.SAPserv.Hamburg.dsh.de |
||||
for pgsql-hackers@postgreSQL.org |
||||
id m0zqNDo-000EBTC; Wed, 16 Dec 98 21:07 MET |
||||
Message-Id: <m0zqNDo-000EBTC@orion.SAPserv.Hamburg.dsh.de> |
||||
From: jwieck@debis.com (Jan Wieck) |
||||
Subject: Re: [HACKERS] redolog - for discussion |
||||
To: vadim@krs.ru (Vadim Mikheev) |
||||
Date: Wed, 16 Dec 1998 21:07:00 +0100 (MET) |
||||
Cc: jwieck@debis.com, pgsql-hackers@postgreSQL.org |
||||
Reply-To: jwieck@debis.com (Jan Wieck) |
||||
In-Reply-To: <3677B71D.C67462B3@krs.ru> from "Vadim Mikheev" at Dec 16, 98 08:35:25 pm |
||||
X-Mailer: ELM [version 2.4 PL25] |
||||
Content-Type: text |
||||
Sender: owner-pgsql-hackers@postgreSQL.org |
||||
Precedence: bulk |
||||
Status: RO |
||||
|
||||
Vadim wrote: |
||||
|
||||
> |
||||
> Jan Wieck wrote: |
||||
> > |
||||
> > RECOVER DATABASE {ALL | UNTIL 'datetime' | RESET}; |
||||
> > |
||||
> ... |
||||
> > |
||||
> > For the others, the backend starts the recovery program |
||||
> > which reads the redolog files, establishes database |
||||
> > connections as required and reruns all the commands in |
||||
> ^^^^^^^^^^^^^^^^^^^^^^^^^^ |
||||
> > them. If a required logfile isn't found, it tells the |
||||
> ^^^^^ |
||||
> |
||||
> I foresee problems with using _commands_ logging for |
||||
> recovery/replication -:(( |
||||
> |
||||
> Let's consider two concurrent updates in READ COMMITTED mode: |
||||
> |
||||
> update test set x = 2 where y = 1; |
||||
> |
||||
> and |
||||
> |
||||
> update test set x = 3 where y = 1; |
||||
> |
||||
> The result of both committed transaction will be x = 2 |
||||
> if the 1st transaction updated row _after_ 2nd transaction |
||||
> and x = 3 if the 2nd transaction gets row after 1st one. |
||||
> Order of updates is not defined by order in which commands |
||||
> begun and so order in which commands should be rerun |
||||
> will be unknown... |
||||
|
||||
Yepp, the order in which commands begun is absolutely not of |
||||
interest. Locking could already delay the execution of one |
||||
command until another one started later has finished and |
||||
released the lock. It's a classic race condition. |
||||
|
||||
Thus, my plan was to log the queries just before the call to |
||||
CommitTransactionCommand() in tcop. This has the advantage, |
||||
that queries which bail out with errors don't get into the |
||||
log at all and must not get rerun. And I can set a static |
||||
flag to false before starting the command, which is set to |
||||
true in the buffer manager when a buffer is written (marked |
||||
dirty), so filtering out queries that do no updates at all is |
||||
easy. |
||||
|
||||
Unfortunately query level logging get's hit by the current |
||||
implementation of sequence numbers. If a query that get's |
||||
aborted somewhere in the middle (maybe by a trigger) called |
||||
nextval() for rows processed earlier, the sequence number |
||||
isn't advanced at recovery time, because the query is |
||||
suppressed at all. And sequences aren't locked, so for |
||||
concurrently running queries getting numbers from the same |
||||
sequence, the results aren't reproduceable. If some |
||||
application selects a value resulting from a sequence and |
||||
uses that later in another query, how could the redolog know |
||||
that this has changed? It's a Const in the query logged, and |
||||
all that corrupts the whole thing. |
||||
|
||||
All that is painful and I don't see another solution yet than |
||||
to hook into nextval(), log out the numbers generated in |
||||
normal operation and getting back the same numbers in redo |
||||
mode. |
||||
|
||||
The whole thing gets more and more complicated :-( |
||||
|
||||
|
||||
Jan |
||||
|
||||
-- |
||||
|
||||
#======================================================================# |
||||
# It's easier to get forgiveness for being wrong than for being right. # |
||||
# Let's break this rule - forgive me. # |
||||
#======================================== jwieck@debis.com (Jan Wieck) # |
||||
|
||||
|
||||
|
||||
|
||||
From owner-pgsql-hackers@hub.org Wed Jun 16 09:29:31 1999 |
||||
Received: from hub.org (hub.org [209.167.229.1]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id JAA22504 |
||||
for <maillist@candle.pha.pa.us>; Wed, 16 Jun 1999 09:29:29 -0400 (EDT) |
||||
Received: from hub.org (hub.org [209.167.229.1]) |
||||
by hub.org (8.9.3/8.9.3) with ESMTP id JAA02132; |
||||
Wed, 16 Jun 1999 09:18:20 -0400 (EDT) |
||||
(envelope-from owner-pgsql-hackers@hub.org) |
||||
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 16 Jun 1999 09:14:07 +0000 (EDT) |
||||
Received: (from majordom@localhost) |
||||
by hub.org (8.9.3/8.9.3) id JAA01318 |
||||
for pgsql-hackers-outgoing; Wed, 16 Jun 1999 09:14:06 -0400 (EDT) |
||||
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||
X-Authentication-Warning: hub.org: majordom set sender to owner-pgsql-hackers@postgreSQL.org using -f |
||||
Received: from sunpine.krs.ru (SunPine.krs.ru [195.161.16.37]) |
||||
by hub.org (8.9.3/8.9.3) with ESMTP id JAA01278 |
||||
for <hackers@postgreSQL.org>; Wed, 16 Jun 1999 09:13:48 -0400 (EDT) |
||||
(envelope-from vadim@krs.ru) |
||||
Received: from krs.ru (dune.krs.ru [195.161.16.38]) |
||||
by sunpine.krs.ru (8.8.8/8.8.8) with ESMTP id VAA06276 |
||||
for <hackers@postgreSQL.org>; Wed, 16 Jun 1999 21:12:49 +0800 (KRSS) |
||||
Message-ID: <3767A2CF.E6E4A5F9@krs.ru> |
||||
Date: Wed, 16 Jun 1999 21:12:47 +0800 |
||||
From: Vadim Mikheev <vadim@krs.ru> |
||||
Organization: OJSC Rostelecom (Krasnoyarsk) |
||||
X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 3.0-RELEASE i386) |
||||
X-Accept-Language: ru, en |
||||
MIME-Version: 1.0 |
||||
To: PostgreSQL Developers List <hackers@postgreSQL.org> |
||||
Subject: [HACKERS] Savepoints... |
||||
Content-Type: text/plain; charset=us-ascii |
||||
Content-Transfer-Encoding: 7bit |
||||
Sender: owner-pgsql-hackers@postgreSQL.org |
||||
Precedence: bulk |
||||
Status: ROr |
||||
|
||||
To have them I need to add tuple id (6 bytes) to heap tuple |
||||
header. Are there objections? Though it's not good to increase |
||||
tuple header size, subj is, imho, very nice feature... |
||||
|
||||
Implementation is , hm, "easy": |
||||
|
||||
- heap_insert/heap_delete/heap_replace/heap_mark4update will |
||||
remember updated tid (and current command id) in relation cache |
||||
and store previously updated tid (remembered in relation cache) |
||||
in additional heap header tid; |
||||
- lmgr will remember command id when lock was acquired; |
||||
- for a savepoint we will just store command id when |
||||
the savepoint was setted; |
||||
- when going to sleep due to concurrent the-same-row update, |
||||
backend will store MyProc and tuple id in shmem hash table. |
||||
|
||||
When rolling back to a savepoint, backend will: |
||||
|
||||
- release locks acquired after savepoint; |
||||
- for a relation updated after savepoint, get last updated tid |
||||
from relation cache, walk through relation, set |
||||
HEAP_XMIN_INVALID/HEAP_XMAX_INVALID in all tuples updated |
||||
after savepoint and wake up concurrent writers blocked |
||||
on these tuples (using shmem hash table mentioned above). |
||||
|
||||
The last feature (waking up of concurrent writers) is most hard |
||||
part to implement. AFAIK, Oracle 7.3 was not able to do it. |
||||
Can someone comment is this feature implemented in Oracle 8.X, |
||||
other DBMSes? |
||||
|
||||
Now about implicit savepoints. Backend will place them before |
||||
user statements execution. In the case of failure, transaction |
||||
state will be rolled back to the one before execution of query. |
||||
As side-effect, this means that we'll get rid of complaints |
||||
about entire transaction abort in the case of mistyping |
||||
causing abort due to parser errors... |
||||
|
||||
Comments? |
||||
|
||||
Vadim |
||||
|
||||
|
||||
@ -1,392 +0,0 @@ |
||||
From lockhart@alumni.caltech.edu Thu Jan 7 13:31:08 1999 |
||||
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA07771 |
||||
for <maillist@candle.pha.pa.us>; Thu, 7 Jan 1999 13:31:06 -0500 (EST) |
||||
Received: from golem.jpl.nasa.gov (IDENT:root@hectic-2.jpl.nasa.gov [128.149.68.204]) by renoir.op.net (o1/$ Revision: 1.18 $) with ESMTP id NAA14597 for <maillist@candle.pha.pa.us>; Thu, 7 Jan 1999 13:27:37 -0500 (EST) |
||||
Received: from alumni.caltech.edu (localhost [127.0.0.1]) |
||||
by golem.jpl.nasa.gov (8.8.5/8.8.5) with ESMTP id SAA13416; |
||||
Thu, 7 Jan 1999 18:26:56 GMT |
||||
Sender: tgl@mythos.jpl.nasa.gov |
||||
Message-ID: <3694FC70.FAD67BC3@alumni.caltech.edu> |
||||
Date: Thu, 07 Jan 1999 18:26:56 +0000 |
||||
From: "Thomas G. Lockhart" <lockhart@alumni.caltech.edu> |
||||
Organization: Caltech/JPL |
||||
X-Mailer: Mozilla 4.07 [en] (X11; I; Linux 2.0.30 i686) |
||||
MIME-Version: 1.0 |
||||
To: Bruce Momjian <maillist@candle.pha.pa.us> |
||||
CC: Postgres Hackers List <hackers@postgresql.org> |
||||
Subject: Outer Joins (and need CASE help) |
||||
References: <199901071747.MAA07054@candle.pha.pa.us> |
||||
Content-Type: text/plain; charset=us-ascii |
||||
Content-Transfer-Encoding: 7bit |
||||
Status: RO |
||||
|
||||
> Thomas, do you need help on outer joins? |
||||
|
||||
Yes. I'm going slowly partly because I get distracted with other |
||||
Postgres stuff like docs, and partly because I don't understand all of |
||||
the pieces I'm working with. |
||||
|
||||
I've identified the place in the MergeJoin code where the null filling |
||||
for outer joins needs to happen, and have the "merge walk" code done. |
||||
But I don't have the supporting code which actually would know how to |
||||
null-fill a result tuple from the left or right. I thought you might be |
||||
interested in that? |
||||
|
||||
I've done some work in the parser, and can now do things like: |
||||
|
||||
postgres=> select * from t1 join t2 using (i); |
||||
NOTICE: JOIN not yet implemented |
||||
i|j|i|k |
||||
-+-+-+- |
||||
1|2|1|3 |
||||
(1 row) |
||||
|
||||
But this is just an inner join, and the result isn't quite right since |
||||
the second "i" column should probably be omitted. At the moment I |
||||
transform it from the syntax above into existing parse nodes, and |
||||
everything from there on works. |
||||
|
||||
I don't yet pass an explicit join node into the planner/optimizer, and |
||||
that will be the hardest part I assume. Perhaps we can work on that |
||||
together. |
||||
|
||||
So, what I'll try to do (soon, in the next few days?) is put in |
||||
|
||||
#ifdef ENABLE_OUTER_JOINS |
||||
|
||||
conditional code into the parser area (already there for the executor) |
||||
and commit everything to the development tree. Does that sound OK? |
||||
|
||||
Oh, and if anyone is looking for something to do, I've got a couple of |
||||
CASE statements in the case.sql regression test which are commented out |
||||
because they crash the backend. They involve references to multiple |
||||
tables within a single result column, and in other contexts that |
||||
construct works. It would be great if someone had time to track it |
||||
down... |
||||
|
||||
- Tom |
||||
|
||||
From lockhart@alumni.caltech.edu Mon Feb 22 02:01:13 1999 |
||||
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id CAA22073 |
||||
for <maillist@candle.pha.pa.us>; Mon, 22 Feb 1999 02:01:12 -0500 (EST) |
||||
Received: from golem.jpl.nasa.gov (IDENT:root@hectic-2.jpl.nasa.gov [128.149.68.204]) by renoir.op.net (o1/$ Revision: 1.18 $) with ESMTP id BAA26054 for <maillist@candle.pha.pa.us>; Mon, 22 Feb 1999 01:57:00 -0500 (EST) |
||||
Received: from alumni.caltech.edu (localhost [127.0.0.1]) |
||||
by golem.jpl.nasa.gov (8.8.5/8.8.5) with ESMTP id GAA04715; |
||||
Mon, 22 Feb 1999 06:56:36 GMT |
||||
Sender: tgl@mythos.jpl.nasa.gov |
||||
Message-ID: <36D0FFA4.32ADB75C@alumni.caltech.edu> |
||||
Date: Mon, 22 Feb 1999 06:56:36 +0000 |
||||
From: "Thomas G. Lockhart" <lockhart@alumni.caltech.edu> |
||||
Organization: Caltech/JPL |
||||
X-Mailer: Mozilla 4.07 [en] (X11; I; Linux 2.0.36 i686) |
||||
MIME-Version: 1.0 |
||||
To: Bruce Momjian <maillist@candle.pha.pa.us> |
||||
CC: hackers@postgreSQL.org |
||||
Subject: Re: start on outer join |
||||
References: <199902220304.WAA10066@candle.pha.pa.us> |
||||
Content-Type: text/plain; charset=us-ascii |
||||
Content-Transfer-Encoding: 7bit |
||||
Status: ROr |
||||
|
||||
Bruce Momjian wrote: |
||||
> |
||||
> > Will apply ... some other changes laying a bit of |
||||
> > groundwork for outer joins so you can start on the planner/optimizer |
||||
> > parts :) |
||||
> Those will be a synch now that I understand the optimizer. In fact, I |
||||
> think it all will happen in the executor. |
||||
|
||||
I've modified executor/nodeMergeJoin.c to walk a left/right/both outer |
||||
join, but didn't fill in the part which actually creates the result |
||||
tuple (which will be the current left- or right-side tuple plus nulls |
||||
for filler). I hope this is up your alley :) |
||||
|
||||
So far, I'm not certain what to pass to the planner. The syntax leads me |
||||
to pass a select structure from gram.y with a "JoinExpr" structure in |
||||
the "fromClause" list. I need to expand that with a combination of |
||||
column names and qualifications, but at the time I see the JoinExpr I |
||||
don't have access to the top query structure itself. So I may just keep |
||||
a modestly transformed JoinExpr to expand later or to pass to the |
||||
planner. |
||||
|
||||
btw, the EXCEPT/INTERSECT stuff from Stefan has some ugliness in gram.y |
||||
which needs to be fixed (the shift/reduce conflict is not acceptable for |
||||
our release version) and some of that code clearly needs to move to |
||||
analyze.c or some other module. |
||||
|
||||
- Tom |
||||
|
||||
From maillist Wed Feb 24 05:27:08 1999 |
||||
Received: (from maillist@localhost) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) id FAA09648; |
||||
Wed, 24 Feb 1999 05:27:08 -0500 (EST) |
||||
From: Bruce Momjian <maillist> |
||||
Message-Id: <199902241027.FAA09648@candle.pha.pa.us> |
||||
Subject: Re: [HACKERS] OUTER joins |
||||
In-Reply-To: <199902240953.EAA08561@candle.pha.pa.us> from Bruce Momjian at "Feb 24, 1999 4:53:21 am" |
||||
To: maillist@candle.pha.pa.us (Bruce Momjian) |
||||
Date: Wed, 24 Feb 1999 05:27:07 -0500 (EST) |
||||
Cc: lockhart@alumni.caltech.edu, hackers@postgreSQL.org |
||||
X-Mailer: ELM [version 2.4ME+ PL47 (25)] |
||||
MIME-Version: 1.0 |
||||
Content-Type: text/plain; charset=US-ASCII |
||||
Content-Transfer-Encoding: 7bit |
||||
Status: RO |
||||
|
||||
> |
||||
> How do you propose doing outer joins in non-mergejoin situations? |
||||
> Mergejoins can only be used currently in equal joins. |
||||
|
||||
Is your solution going to be to make sure the OUTER table is always a |
||||
MergeJoin, or on the outside of a join loop? That could work. |
||||
|
||||
That could get tricky if the table is joined to _two_ other tables. |
||||
With the cleaned-up optimizer, we can disable non-merge joins in certain |
||||
circumstances, and prevent OUTER tables from being inner in the others. |
||||
Is that the plan? |
||||
|
||||
-- |
||||
Bruce Momjian | http://www.op.net/~candle |
||||
maillist@candle.pha.pa.us | (610) 853-3000 |
||||
+ If your life is a hard drive, | 830 Blythe Avenue |
||||
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026 |
||||
|
||||
From lockhart@alumni.caltech.edu Mon Mar 1 13:01:08 1999 |
||||
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id NAA21672 |
||||
for <maillist@candle.pha.pa.us>; Mon, 1 Mar 1999 13:01:06 -0500 (EST) |
||||
Received: from golem.jpl.nasa.gov (IDENT:root@hectic-2.jpl.nasa.gov [128.149.68.204]) by renoir.op.net (o1/$ Revision: 1.18 $) with ESMTP id MAA12756 for <maillist@candle.pha.pa.us>; Mon, 1 Mar 1999 12:14:16 -0500 (EST) |
||||
Received: from alumni.caltech.edu (localhost [127.0.0.1]) |
||||
by golem.jpl.nasa.gov (8.8.5/8.8.5) with ESMTP id RAA09406; |
||||
Mon, 1 Mar 1999 17:10:49 GMT |
||||
Sender: tgl@mythos.jpl.nasa.gov |
||||
Message-ID: <36DACA19.E6DBE7D8@alumni.caltech.edu> |
||||
Date: Mon, 01 Mar 1999 17:10:49 +0000 |
||||
From: "Thomas G. Lockhart" <lockhart@alumni.caltech.edu> |
||||
Organization: Caltech/JPL |
||||
X-Mailer: Mozilla 4.07 [en] (X11; I; Linux 2.0.36 i686) |
||||
MIME-Version: 1.0 |
||||
To: Bruce Momjian <maillist@candle.pha.pa.us> |
||||
CC: PostgreSQL-development <hackers@postgreSQL.org> |
||||
Subject: Re: OUTER joins |
||||
References: <199902240953.EAA08561@candle.pha.pa.us> |
||||
Content-Type: text/plain; charset=us-ascii |
||||
Content-Transfer-Encoding: 7bit |
||||
Status: ROr |
||||
|
||||
(back from a short vacation...) |
||||
|
||||
> How do you propose doing outer joins in non-mergejoin situations? |
||||
> Mergejoins can only be used currently in equal joins. |
||||
|
||||
Hadn't thought about it, other than figuring that implementing the |
||||
equi-join first was a good start. There is a class of outer join syntax |
||||
(the USING clause) which is implicitly an equi-join... |
||||
|
||||
- Tom |
||||
|
||||
From lockhart@alumni.caltech.edu Mon Mar 8 21:55:02 1999 |
||||
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id VAA15978 |
||||
for <maillist@candle.pha.pa.us>; Mon, 8 Mar 1999 21:54:57 -0500 (EST) |
||||
Received: from golem.jpl.nasa.gov (IDENT:root@hectic-1.jpl.nasa.gov [128.149.68.203]) by renoir.op.net (o1/$ Revision: 1.18 $) with ESMTP id VAA15837 for <maillist@candle.pha.pa.us>; Mon, 8 Mar 1999 21:48:33 -0500 (EST) |
||||
Received: from alumni.caltech.edu (localhost [127.0.0.1]) |
||||
by golem.jpl.nasa.gov (8.8.5/8.8.5) with ESMTP id CAA06996; |
||||
Tue, 9 Mar 1999 02:46:40 GMT |
||||
Sender: tgl@mythos.jpl.nasa.gov |
||||
Message-ID: <36E48B90.F3E902B7@alumni.caltech.edu> |
||||
Date: Tue, 09 Mar 1999 02:46:40 +0000 |
||||
From: "Thomas G. Lockhart" <lockhart@alumni.caltech.edu> |
||||
Organization: Caltech/JPL |
||||
X-Mailer: Mozilla 4.07 [en] (X11; I; Linux 2.0.36 i686) |
||||
MIME-Version: 1.0 |
||||
To: Bruce Momjian <maillist@candle.pha.pa.us> |
||||
CC: hackers@postgreSQL.org |
||||
Subject: Re: OUTER joins |
||||
References: <199903070325.WAA10357@candle.pha.pa.us> |
||||
Content-Type: text/plain; charset=us-ascii |
||||
Content-Transfer-Encoding: 7bit |
||||
Status: ROr |
||||
|
||||
> > Hadn't thought about it, other than figuring that implementing the |
||||
> > equi-join first was a good start. There is a class of outer join |
||||
> > syntax (the USING clause) which is implicitly an equi-join... |
||||
> Not that easy. You don't automatically get a mergejoin from an |
||||
> equijoin. I will have to force outer's to be either mergejoins, or |
||||
> inners of non-merge joins. Can you add code to non-merge joins in the |
||||
> executor to throw out a null row if it does not find an inner match |
||||
> for the outer row, and I will handle the optimizer so it doesn't throw |
||||
> a non-conforming plan to the executor. |
||||
|
||||
So far I don't have enough info in the parser to get the |
||||
planner/optimizer going. Should we work from the front to the back, or |
||||
should I go ahead and look at the non-merge joins? It's painfully |
||||
obvious that I don't know anything about the middle parts of this to |
||||
proceed without lots more research. |
||||
|
||||
- Tom |
||||
|
||||
From lockhart@alumni.caltech.edu Tue Mar 9 22:47:57 1999 |
||||
Received: from golem.jpl.nasa.gov (IDENT:root@hectic-1.jpl.nasa.gov [128.149.68.203]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id WAA07869 |
||||
for <maillist@candle.pha.pa.us>; Tue, 9 Mar 1999 22:47:54 -0500 (EST) |
||||
Received: from alumni.caltech.edu (localhost [127.0.0.1]) |
||||
by golem.jpl.nasa.gov (8.8.5/8.8.5) with ESMTP id DAA14761; |
||||
Wed, 10 Mar 1999 03:46:43 GMT |
||||
Sender: tgl@mythos.jpl.nasa.gov |
||||
Message-ID: <36E5EB23.F5CD959B@alumni.caltech.edu> |
||||
Date: Wed, 10 Mar 1999 03:46:43 +0000 |
||||
From: "Thomas G. Lockhart" <lockhart@alumni.caltech.edu> |
||||
Organization: Caltech/JPL |
||||
X-Mailer: Mozilla 4.07 [en] (X11; I; Linux 2.0.36 i686) |
||||
MIME-Version: 1.0 |
||||
To: Bruce Momjian <maillist@candle.pha.pa.us>, tgl@mythos.jpl.nasa.gov |
||||
Subject: Re: SQL outer |
||||
References: <199903100112.UAA05772@candle.pha.pa.us> |
||||
Content-Type: text/plain; charset=us-ascii |
||||
Content-Transfer-Encoding: 7bit |
||||
Status: RO |
||||
|
||||
> select * |
||||
> from outer tab1, tab2, tab3 |
||||
> where tab1.col1 = tab2.col1 and |
||||
> tab1.col1 = tab3.col1 |
||||
|
||||
select * |
||||
from t1 left join t2 using (c1) |
||||
join t3 on (c1 = t3.c1) |
||||
|
||||
Result: |
||||
t1.c1 t1.c2 t2.c2 t3.c1 |
||||
2 12 NULL 32 |
||||
|
||||
t1: |
||||
c1 c2 |
||||
1 11 |
||||
2 12 |
||||
3 13 |
||||
4 14 |
||||
|
||||
t2: |
||||
c1 c2 |
||||
1 21 |
||||
3 23 |
||||
|
||||
t3: |
||||
c1 c2 |
||||
2 32 |
||||
|
||||
From lockhart@alumni.caltech.edu Wed Mar 10 10:48:54 1999 |
||||
Received: from golem.jpl.nasa.gov (IDENT:root@hectic-1.jpl.nasa.gov [128.149.68.203]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA16741 |
||||
for <maillist@candle.pha.pa.us>; Wed, 10 Mar 1999 10:48:51 -0500 (EST) |
||||
Received: from alumni.caltech.edu (localhost [127.0.0.1]) |
||||
by golem.jpl.nasa.gov (8.8.5/8.8.5) with ESMTP id PAA17723; |
||||
Wed, 10 Mar 1999 15:48:31 GMT |
||||
Sender: tgl@mythos.jpl.nasa.gov |
||||
Message-ID: <36E6944F.1F93B08@alumni.caltech.edu> |
||||
Date: Wed, 10 Mar 1999 15:48:31 +0000 |
||||
From: "Thomas G. Lockhart" <lockhart@alumni.caltech.edu> |
||||
Organization: Caltech/JPL |
||||
X-Mailer: Mozilla 4.07 [en] (X11; I; Linux 2.0.36 i686) |
||||
MIME-Version: 1.0 |
||||
To: Bruce Momjian <maillist@candle.pha.pa.us> |
||||
CC: Thomas Lockhart <lockhart@alumni.caltech.edu> |
||||
Subject: Re: SQL outer |
||||
References: <199903100112.UAA05772@candle.pha.pa.us> <36E5EB23.F5CD959B@alumni.caltech.edu> |
||||
Content-Type: text/plain; charset=us-ascii |
||||
Content-Transfer-Encoding: 7bit |
||||
Status: ROr |
||||
|
||||
Just thinking... |
||||
|
||||
If the initial RelOptInfo groupings are derived from the WHERE clause |
||||
expressions, how about marking the "outer" property in those expressions |
||||
in the parser? istm that is where the parser knows about two tables in |
||||
one place, and I'm generating those expressions anyway. We could add a |
||||
field(s) to the expression structure, or pass along a slightly different |
||||
structure... |
||||
|
||||
- Tom |
||||
|
||||
From owner-pgsql-hackers@hub.org Wed Jul 21 02:35:13 1999 |
||||
Received: from hub.org (hub.org [216.126.84.1]) |
||||
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id CAA13837 |
||||
for <maillist@candle.pha.pa.us>; Wed, 21 Jul 1999 02:35:12 -0400 (EDT) |
||||
Received: from hub.org (hub.org [216.126.84.1]) |
||||
by hub.org (8.9.3/8.9.3) with ESMTP id CAA88539; |
||||
Wed, 21 Jul 1999 02:27:41 -0400 (EDT) |
||||
(envelope-from owner-pgsql-hackers@hub.org) |
||||
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 21 Jul 1999 02:24:08 +0000 (EDT) |
||||
Received: (from majordom@localhost) |
||||
by hub.org (8.9.3/8.9.3) id CAA87850 |
||||
for pgsql-hackers-outgoing; Wed, 21 Jul 1999 02:23:13 -0400 (EDT) |
||||
(envelope-from owner-pgsql-hackers@postgreSQL.org) |
||||
Received: from localhost (IDENT:root@hectic-2.jpl.nasa.gov [128.149.68.204]) |
||||
by hub.org (8.9.3/8.9.3) with ESMTP id CAA87810 |
||||
for <pgsql-hackers@postgreSQL.org>; Wed, 21 Jul 1999 02:22:52 -0400 (EDT) |
||||
(envelope-from lockhart@alumni.caltech.edu) |
||||
Received: from alumni.caltech.edu (lockhart@localhost [127.0.0.1]) |
||||
by localhost (8.8.7/8.8.7) with ESMTP id GAA14480; |
||||
Wed, 21 Jul 1999 06:20:22 GMT |
||||
Message-ID: <379566A6.A4CDF97F@alumni.caltech.edu> |
||||
Date: Wed, 21 Jul 1999 06:20:22 +0000 |
||||
From: Thomas Lockhart <lockhart@alumni.caltech.edu> |
||||
X-Mailer: Mozilla 4.6 [en] (X11; I; Linux 2.0.36 i686) |
||||
X-Accept-Language: en |
||||
MIME-Version: 1.0 |
||||
To: Tom Lane <tgl@sss.pgh.pa.us> |
||||
CC: Bruce Momjian <maillist@candle.pha.pa.us>, pgsql-hackers@postgreSQL.org |
||||
Subject: Re: [HACKERS] Another reason to redesign querytree representation |
||||
References: <591.932505751@sss.pgh.pa.us> |
||||
Content-Type: text/plain; charset=us-ascii |
||||
Content-Transfer-Encoding: 7bit |
||||
Sender: owner-pgsql-hackers@postgreSQL.org |
||||
Precedence: bulk |
||||
Status: RO |
||||
|
||||
> Thomas, what do you think is needed for outer joins? |
||||
|
||||
Bruce and I have talked about it some already: |
||||
|
||||
For outer joins, tables must be combined in a particular order. For |
||||
example, a left outer join requires that any entries in the left-side |
||||
table which do not have a corresponding entry in the right-side table |
||||
be expanded with nulls during the join. The information on the outer |
||||
join can't be carried by the rte since the same table can appear twice |
||||
in an outer join expression: |
||||
|
||||
select * from t1 left join t2 using (i) |
||||
left join t1 on (i = t1.j); |
||||
|
||||
For a query like |
||||
|
||||
select * from t1 left join t2 using (i) where t2.j = 3; |
||||
|
||||
istm that the outer join must be done before the t2 qualification is |
||||
applied, and that another ordering may produce the wrong result. |
||||
|
||||
>From what I understand Bruce to say, the planner/optimizer is allowed |
||||
to try all kinds of permutations of plans, choosing the one with the |
||||
lowest cost. But if the info for the join is carried in a |
||||
qualification node, then the planner/optimizer must know that it can't |
||||
reorder the query as freely as it does now. |
||||
|
||||
I was thinking of having a new qualification node to carry this info, |
||||
and it could be transformed into a mergejoin node which has a couple |
||||
of new fields indicating left and/or right outer join behavior. |
||||
|
||||
A hashjoin method may be possible for queries which are structured as |
||||
a left outer join; other outer joins will need to use the mergejoin |
||||
method. Also, some poorly-qualified outer joins reduce to inner joins, |
||||
and perhaps the optimizer can be smart enough to realize this. |
||||
|
||||
- Thomas |
||||
|
||||
-- |
||||
Thomas Lockhart lockhart@alumni.caltech.edu |
||||
South Pasadena, California |
||||
|
||||
|
||||
Loading…
Reference in new issue