open-dsi
/
postgres
mirror of https://github.com/postgres/postgres
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add to threads.

Browse Source
REL7_2_STABLE
Bruce Momjian 25 years ago
parent 3676e121f0
commit abb6134095
1 changed files with 60 additions and 634 deletions
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Show Stats Download Patch File Download Diff File
  1. 694
      doc/TODO.detail/thread

694
doc/TODO.detail/thread
Unescape View File

@ -1,642 +1,68 @@
From mscott@sacadia.com Wed Nov 15 14:50:19 2000 From pgsql-hackers-owner+M13599=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 17:25:32 2001
Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) Return-path: <pgsql-hackers-owner+M13599=candle.pha.pa.us=pgman@postgresql.org>
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id OAA11583 Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged))
for <pgman@candle.pha.pa.us>; Wed, 15 Nov 2000 14:50:13 -0500 (EST) by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QLPWo07589
Received: from localhost (localhost [127.0.0.1]) for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 17:25:32 -0400 (EDT)
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id LAA09998; Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
Wed, 15 Nov 2000 11:35:33 -0800 (PST) by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QLPf405606
Date: Wed, 15 Nov 2000 11:35:33 -0800 (PST) for <pgman@candle.pha.pa.us>; Wed, 26 Sep 2001 16:25:41 -0500 (CDT)
From: Myron Scott <mscott@sacadia.com> (envelope-from pgsql-hackers-owner+M13599=candle.pha.pa.us=pgman@postgresql.org)
X-Sender: mscott@goldengate.kojoworldwide.com. Received: from gromit.dotclick.com (ipn9-f8366.net-resource.net [216.204.83.66])
To: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>, by postgresql.org (8.11.3/8.11.4) with ESMTP id f8QKj3h82020
Bruce Momjian <pgman@candle.pha.pa.us>, Tom Lane <tgl@sss.pgh.pa.us> for <pgsql-hackers@postgresql.org>; Wed, 26 Sep 2001 16:45:03 -0400 (EDT)
Subject: Please help with some advice (envelope-from markw@mohawksoft.com)
Message-ID: <Pine.GSO.4.10.10011151053260.9940-100000@goldengate.kojoworldwide.com.> Received: from mohawksoft.com (IDENT:markw@localhost.localdomain [127.0.0.1])
MIME-Version: 1.0 by gromit.dotclick.com (8.9.3/8.9.3) with ESMTP id QAA23693;
Content-Type: TEXT/PLAIN; charset=US-ASCII Wed, 26 Sep 2001 16:43:02 -0400
Status: ORr Message-ID: <3BB23DD6.E86AF327@mohawksoft.com>
Date: Wed, 26 Sep 2001 16:43:02 -0400
Dear Sirs, From: mlw <markw@mohawksoft.com>
X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.2 i686)
I have been lurking on the PostgreSQL hackers list for about 3 months now
and your names comes up more than any with helpful info about the project
so I was hoping you could help me.
Let me cut to the chase. I have been experimenting with 7.0.2 source to
see if I could create a mutlti-threaded version of the backend so
I could link directly from java ( I have a fe<->be protocol that I use for
my apps). Needless to say I got into much more than I bargained for. I
now have a version that works and it has some nice benefits that are very
helpful to a project that I am working on. What I gained was
prepared statements outside of spi
batched commits (fsync)
one connection per thread
multiple threads per process
multiple processes per installation
I never really intended for anyone else to see the work so I drifted
pretty far from the original code. I also ended up using Solaris threads
rather than pthreads, I did my own implementation of the bufmgr.c and
gram.y, and used Solaris implementation of mutex in place of S_LOCK and
TAS. I grabbed all global variables and put them in an environment
variable that is thread local. I also did some really stupid
things like making TransactionId uint64 and making all my inserts use the
same oid.
My question is this. I would like to get some critical feedback and
suggestions about the work from others. What is the best way to go about
this? I thought about trying to create a project on greatbridge.org
but I am rather new to open source and the code needs commented properly
and cleaned up before too many try and look at it.
Any suggestions would be greatly appreciated.
Thanks in advance,
Myron Scott
From mscott@sacadia.com Thu Nov 16 17:19:45 2000
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id RAA04315
for <pgman@candle.pha.pa.us>; Thu, 16 Nov 2000 17:19:43 -0500 (EST)
Received: from localhost (localhost [127.0.0.1])
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id OAA11449;
Thu, 16 Nov 2000 14:05:15 -0800 (PST)
Date: Thu, 16 Nov 2000 14:05:15 -0800 (PST)
From: Myron Scott <mscott@sacadia.com>
X-Sender: mscott@goldengate.kojoworldwide.com.
To: Bruce Momjian <pgman@candle.pha.pa.us>
cc: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>, Tom Lane <tgl@sss.pgh.pa.us>
Subject: Re: Please help with some advice
In-Reply-To: <200011160533.AAA27886@candle.pha.pa.us>
Message-ID: <Pine.GSO.4.10.10011161401570.11441-100000@goldengate.kojoworldwide.com.>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: OR
Bruce Momjian wrote:
>I am curious how you isolated each thread. It seems we pretty much
>assume all our memory is controlled by a single query in the process.
I moved all global variables to a thread global variable which is accessed
by the method GetEnv(). Which looks like this
Env* GetEnv(void) {
Env* env;
thr_getspecific(*envkey,(void*)&env);
return env;
}
The Env struct includes the CurrentMemoryContext, TopMemoryContext,
PortalHeapMemory for each instance of a connection (one thread per
connection). So, for example,
EndPortalAllocMode uses GetEnv()->CurrentMemoryContext
void
EndPortalAllocMode()
{
PortalHeapMemory context;
AssertState(PortalManagerEnabled);
AssertState(IsA(GetEnv()->CurrentMemoryContext,
PortalHeapMemory));
context = (PortalHeapMemory) GetEnv()->CurrentMemoryContext;
AssertState(PointerIsValid(context->block)); /* XXX
Trap(...) */
/* free current mode */
AllocSetReset(&HEAPMEMBLOCK(context)->setData);
MemoryContextFree((MemoryContext)
PortalHeapMemoryGetVariableMemory(context),
context->block);
/* restore previous mode */
context->block = FixedStackPop(&context->stackData);
}
From vmikheev@SECTORBASE.COM Thu Nov 16 17:23:22 2000
Received: from sectorbase2.sectorbase.com ([208.48.122.131])
by candle.pha.pa.us (8.9.0/8.9.0) with SMTP id RAA04562
for <pgman@candle.pha.pa.us>; Thu, 16 Nov 2000 17:23:21 -0500 (EST)
Received: by sectorbase2.sectorbase.com with Internet Mail Service (5.5.2650.21)
id <V8XQB5RW>; Thu, 16 Nov 2000 14:05:24 -0800
Message-ID: <8F4C99C66D04D4118F580090272A7A234D318D@sectorbase1.sectorbase.com>
From: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>
To: "'Myron Scott'" <mscott@sacadia.com>,
Bruce Momjian
<pgman@candle.pha.pa.us>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Subject: RE: Please help with some advice
Date: Thu, 16 Nov 2000 14:09:30 -0800
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2650.21)
Content-Type: text/plain;
charset="iso-8859-1"
Status: ORr
I think the question do we want to make backend multy-threaded
should be discussed in hackers.
Vadim
> -----Original Message-----
> From: Myron Scott [mailto:mscott@sacadia.com]
> Sent: Thursday, November 16, 2000 2:05 PM
> To: Bruce Momjian
> Cc: Mikheev, Vadim; Tom Lane
> Subject: Re: Please help with some advice
>
>
> Bruce Momjian wrote:
>
> >I am curious how you isolated each thread. It seems we pretty much
> >assume all our memory is controlled by a single query in the process.
>
>
>
> I moved all global variables to a thread global variable
> which is accessed
> by the method GetEnv(). Which looks like this
>
> Env* GetEnv(void) {
> Env* env;
> thr_getspecific(*envkey,(void*)&env);
> return env;
> }
>
> The Env struct includes the CurrentMemoryContext, TopMemoryContext,
> PortalHeapMemory for each instance of a connection (one thread per
> connection). So, for example,
> EndPortalAllocMode uses GetEnv()->CurrentMemoryContext
>
> void
> EndPortalAllocMode()
> {
> PortalHeapMemory context;
>
> AssertState(PortalManagerEnabled);
> AssertState(IsA(GetEnv()->CurrentMemoryContext,
> PortalHeapMemory));
>
> context = (PortalHeapMemory) GetEnv()->CurrentMemoryContext;
> AssertState(PointerIsValid(context->block)); /* XXX
> Trap(...) */
>
> /* free current mode */
> AllocSetReset(&HEAPMEMBLOCK(context)->setData);
> MemoryContextFree((MemoryContext)
> PortalHeapMemoryGetVariableMemory(context),
> context->block);
>
> /* restore previous mode */
> context->block = FixedStackPop(&context->stackData);
> }
>
>
>
From mscott@sacadia.com Thu Nov 16 22:16:38 2000
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id WAA14638
for <pgman@candle.pha.pa.us>; Thu, 16 Nov 2000 22:16:36 -0500 (EST)
Received: from localhost (localhost [127.0.0.1])
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id TAA11874;
Thu, 16 Nov 2000 19:04:48 -0800 (PST)
Date: Thu, 16 Nov 2000 19:04:48 -0800 (PST)
From: Myron Scott <mscott@sacadia.com>
X-Sender: mscott@goldengate.kojoworldwide.com.
To: Bruce Momjian <pgman@candle.pha.pa.us>
cc: "Mikheev, Vadim" <vmikheev@SECTORBASE.COM>, Tom Lane <tgl@sss.pgh.pa.us>
Subject: Re: Please help with some advice
In-Reply-To: <200011170156.UAA11438@candle.pha.pa.us>
Message-ID: <Pine.GSO.4.10.10011161904140.11870-100000@goldengate.kojoworldwide.com.>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: ORr
Thanks very much, I will post to hackers.
Myron
From pgsql-hackers-owner+M2691@postgresql.org Tue Jan 2 00:30:20 2001
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id AAA08195
for <pgman@candle.pha.pa.us>; Tue, 2 Jan 2001 00:30:19 -0500 (EST)
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f025UjL33335;
Tue, 2 Jan 2001 00:30:45 -0500 (EST)
(envelope-from pgsql-hackers-owner+M2691@postgresql.org)
Received: from mailsys01.intnet.net (tmail.wwc.com [198.252.32.143] (may be forged))
by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f025UTL33232
for <pgsql-hackers@postgresql.org>; Tue, 2 Jan 2001 00:30:32 -0500 (EST)
(envelope-from mscott@sacadia.com)
Received: from [206.112.108.0] (HELO sacadia.com)
by mailsys01.intnet.net (CommuniGate Pro SMTP 3.3.2)
with ESMTP id 2214231; Tue, 02 Jan 2001 00:29:47 -0500
Message-ID: <3A5167DB.3050807@sacadia.com>
Date: Mon, 01 Jan 2001 21:32:11 -0800
From: Myron Scott <mscott@sacadia.com>
Reply-To: mscott@sacadia.com
User-Agent: Mozilla/5.0 (Windows; U; Win98; en-US; m18) Gecko/20001108 Netscape6/6.0
X-Accept-Language: en X-Accept-Language: en
MIME-Version: 1.0 MIME-Version: 1.0
To: "Ross J. Reedstrom" <reedstrm@rice.edu> To: "D. Hageman" <dhageman@dracken.com>,
CC: pgsql-hackers@postgresql.org "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Using Threads? Subject: Re: [HACKERS] Spinlock performance improvement proposal
References: <004401c058fd$fd498d40$f2356880@tracy> <Pine.GSO.4.10.10012032351040.28161-100000@goldengate.kojoworldwide.com.> <20001204113307.B5871@rice.edu> References: <Pine.LNX.4.33.0109261330030.1906-100000@typhon.dracken.com>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR
For anyone interested,
I have posted my multi-threaded version of PostgreSQL here.
http://www.sacadia.com/mtpg.html
It is based on 7.0.2 and the TAO CORBA ORB which is here.
http://www.cs.wustl.edu/~schmidt/TAO.html
Myron Scott
mkscott@sacadia.com
From bright@fw.wintelcom.net Tue Jan 2 03:02:28 2001
Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20])
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA16169
for <pgman@candle.pha.pa.us>; Tue, 2 Jan 2001 03:02:27 -0500 (EST)
Received: (from bright@localhost)
by fw.wintelcom.net (8.10.0/8.10.0) id f0282Vm10623;
Tue, 2 Jan 2001 00:02:31 -0800 (PST)
Date: Tue, 2 Jan 2001 00:02:31 -0800
From: Alfred Perlstein <bright@wintelcom.net>
To: Bruce Momjian <pgman@candle.pha.pa.us>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Assuming that TAS() will succeed the first time is verboten
Message-ID: <20010102000230.C19572@fw.wintelcom.net>
References: <9850.978067943@sss.pgh.pa.us> <200101020759.CAA15836@candle.pha.pa.us>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline Content-Transfer-Encoding: 7bit
User-Agent: Mutt/1.2.5i
In-Reply-To: <200101020759.CAA15836@candle.pha.pa.us>; from pgman@candle.pha.pa.us on Tue, Jan 02, 2001 at 02:59:20AM -0500
Status: OR
* Bruce Momjian <pgman@candle.pha.pa.us> [010101 23:59] wrote:
> > Alfred Perlstein <bright@wintelcom.net> writes:
> > > One trick that may help is calling sched_yield(2) on a lock miss,
> > > it's a POSIX call and quite new so you'd need a 'configure' test
> > > for it.
> >
> > The author of the current s_lock code seems to have thought that
> > select() with a zero delay would do the equivalent of sched_yield().
> > I'm not sure if that's true on very many kernels, if indeed any...
> >
> > I doubt we could buy much by depending on sched_yield(); if you want
> > to assume POSIX facilities, ISTM you might as well go for user-space
> > semaphores and forget the whole TAS mechanism.
>
>
> Another issue is that sched_yield brings in the pthreads library/hooks
> on some OS's, which we certainly want to avoid.
I know it's a major undertaking, but since the work is sort of done,
have you guys considered the port to solaris threads and seeing about
making a pthreads port of that?
I know it would probably get you considerable gains under Windows
at the expense of dropping some really really legacy system.
Or you could do what apache (is rumored) does and have it do either
threads or processes or both...
--
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
"I have the heart of a child; I keep it in a jar on my desk."
From pgsql-hackers-owner+M4275@postgresql.org Mon Feb 5 21:45:00 2001
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id VAA09262
for <pgman@candle.pha.pa.us>; Mon, 5 Feb 2001 21:44:59 -0500 (EST)
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f162ixx00920;
Mon, 5 Feb 2001 21:44:59 -0500 (EST)
(envelope-from pgsql-hackers-owner+M4275@postgresql.org)
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f162fSx00595
for <pgsql-hackers@postgresql.org>; Mon, 5 Feb 2001 21:41:29 -0500 (EST)
(envelope-from mscott@sacadia.com)
Received: from localhost (localhost [127.0.0.1])
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id SAA03298
for <pgsql-hackers@postgresql.org>; Mon, 5 Feb 2001 18:25:05 -0800 (PST)
Date: Mon, 5 Feb 2001 18:25:05 -0800 (PST)
From: Myron Scott <mscott@sacadia.com>
X-Sender: mscott@goldengate.kojoworldwide.com.
To: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Using Threads?
Message-ID: <Pine.GSO.4.10.10102051823210.3289-100000@goldengate.kojoworldwide.com.>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR
I have put a new version of my multi-threaded
postgresql experiment at
http://www.sacadia.com/mtpg.html
This one actually works. I have added a server
based on omniORB, a CORBA 2.3 ORB from ATT. It
is much smaller than TAO and uses the thread per
connection model. I haven't added the java side
of the JNI interface yet but the C++ side is there.
It's still not stable but it is much better than
the last.
Myron Scott
mkscott@sacadia.com
From pgsql-hackers-owner+M4304@postgresql.org Tue Feb 6 10:24:21 2001
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA22027
for <pgman@candle.pha.pa.us>; Tue, 6 Feb 2001 10:24:20 -0500 (EST)
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16FOBx97182;
Tue, 6 Feb 2001 10:24:11 -0500 (EST)
(envelope-from pgsql-hackers-owner+M4304@postgresql.org)
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16FLWx96814
for <pgsql-hackers@postgresql.org>; Tue, 6 Feb 2001 10:21:33 -0500 (EST)
(envelope-from mscott@sacadia.com)
Received: from localhost (localhost [127.0.0.1])
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id HAA04170;
Tue, 6 Feb 2001 07:05:04 -0800 (PST)
Date: Tue, 6 Feb 2001 07:05:04 -0800 (PST)
From: Myron Scott <mscott@sacadia.com>
X-Sender: mscott@goldengate.kojoworldwide.com.
To: Karel Zak <zakkr@zf.jcu.cz>
cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Using Threads
In-Reply-To: <Pine.LNX.3.96.1010206101030.20355B-100000@ara.zf.jcu.cz>
Message-ID: <Pine.GSO.4.10.10102060650250.4153-100000@goldengate.kojoworldwide.com.>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR
>
> Sorry I haven't time to see and test your experiment,
> but I have a question. How you solve memory management?
> The current mmgr is based on global variable
> CurrentMemoryContext that is very often changed and used.
> Use you for this locks? If yes it is probably problematic
> point for perfomance.
>
> Karel
>
There are many many globals I had to work around including all the memory
management stuff. I basically threw everything into and "environment"
variable which I stored in a thread specific using thr_setspecific.
Performance is acually very good for what I am doing. I was able to batch
commit transactions which cuts down on fsync calls, use prepared
statements from my client using CORBA, and the various locking calls for
the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did
some performance tests for inserts
20 clients, 900 inserts per client, 1 insert per transaction, 4 different
tables.
7.0.2 About 10:52 average completion
multi-threaded 2:42 average completion
7.1beta3 1:13 average completion
If I increased the number of inserts per transaction, multi-threaded got
closer to 7.1 for inserts. I haven't tested other other types of
commands
yet.
Myron Scott
mkscott@sacadia.com
From pgsql-hackers-owner+M4313@postgresql.org Tue Feb 6 12:32:00 2001
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id MAA29163
for <pgman@candle.pha.pa.us>; Tue, 6 Feb 2001 12:31:59 -0500 (EST)
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16HVox17454;
Tue, 6 Feb 2001 12:31:51 -0500 (EST)
(envelope-from pgsql-hackers-owner+M4313@postgresql.org)
Received: from ara.zf.jcu.cz (ara.zf.jcu.cz [160.217.161.4])
by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16HV6x17323
for <pgsql-hackers@postgresql.org>; Tue, 6 Feb 2001 12:31:06 -0500 (EST)
(envelope-from zakkr@zf.jcu.cz)
Received: from localhost (zakkr@localhost)
by ara.zf.jcu.cz (8.9.3/8.9.3/Debian 8.9.3-21) with SMTP id SAA03980;
Tue, 6 Feb 2001 18:31:02 +0100
Date: Tue, 6 Feb 2001 18:31:02 +0100 (CET)
From: Karel Zak <zakkr@zf.jcu.cz>
To: Myron Scott <mscott@sacadia.com>
cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Using Threads
In-Reply-To: <Pine.GSO.4.10.10102060650250.4153-100000@goldengate.kojoworldwide.com.>
Message-ID: <Pine.LNX.3.96.1010206182112.3799B-100000@ara.zf.jcu.cz>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR
On Tue, 6 Feb 2001, Myron Scott wrote:
> There are many many globals I had to work around including all the memory
> management stuff. I basically threw everything into and "environment"
> variable which I stored in a thread specific using thr_setspecific.
Yes, it's good. I working on multi-thread application server
(http://mape.jcu.cz) and I use for this project some things from PG (like
mmgr), I planning use same solution.
> Performance is acually very good for what I am doing. I was able to batch
> commit transactions which cuts down on fsync calls, use prepared
> statements from my client using CORBA, and the various locking calls for
> the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did
> some performance tests for inserts
>
> 20 clients, 900 inserts per client, 1 insert per transaction, 4 different
> tables.
>
> 7.0.2 About 10:52 average completion
> multi-threaded 2:42 average completion
> 7.1beta3 1:13 average completion
It is very very good for time for 7.1, already look forward to 7.2! :-)
BTW, I not sure if you anytime in future will see threads in
official PostgreSQL and if you spending time on relevant things (IMHO).
Karel
From pgsql-hackers-owner+M4304@postgresql.org Tue Feb 6 10:24:21 2001
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA22027
for <pgman@candle.pha.pa.us>; Tue, 6 Feb 2001 10:24:20 -0500 (EST)
Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28])
by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16FOBx97182;
Tue, 6 Feb 2001 10:24:11 -0500 (EST)
(envelope-from pgsql-hackers-owner+M4304@postgresql.org)
Received: from goldengate.kojoworldwide.com. ([216.133.4.130])
by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16FLWx96814
for <pgsql-hackers@postgresql.org>; Tue, 6 Feb 2001 10:21:33 -0500 (EST)
(envelope-from mscott@sacadia.com)
Received: from localhost (localhost [127.0.0.1])
by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id HAA04170;
Tue, 6 Feb 2001 07:05:04 -0800 (PST)
Date: Tue, 6 Feb 2001 07:05:04 -0800 (PST)
From: Myron Scott <mscott@sacadia.com>
X-Sender: mscott@goldengate.kojoworldwide.com.
To: Karel Zak <zakkr@zf.jcu.cz>
cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Using Threads
In-Reply-To: <Pine.LNX.3.96.1010206101030.20355B-100000@ara.zf.jcu.cz>
Message-ID: <Pine.GSO.4.10.10102060650250.4153-100000@goldengate.kojoworldwide.com.>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Precedence: bulk Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org Sender: pgsql-hackers-owner@postgresql.org
Status: OR Status: OR
"D. Hageman" wrote:
>
> Sorry I haven't time to see and test your experiment, > The plan for the new spinlocks does look like it has some potential. My
> but I have a question. How you solve memory management? > only comment in regards to permformance when we start looking at SMP
> The current mmgr is based on global variable > machines is ... it is my belief that getting a true threaded backend may
> CurrentMemoryContext that is very often changed and used. > be the only way to get the full potential out of SMP machines. I see that
> Use you for this locks? If yes it is probably problematic > is one of the things to experiment with on the TODO list and I have seen
> point for perfomance. > some people have messed around already with this using Solaris threads.
> > It should probably be attempted with pthreads if PostgreSQL is going to
> Karel > keep some resemblance of cross-platform compatibility. At that time, it
> > would probably be easier to go in and clean up some stuff for the
> implementation of other TODO items (put in the base framework for more
There are many many globals I had to work around including all the memory > complex future items) as threading the backend would take a little bit of
management stuff. I basically threw everything into and "environment" > ideology shift.
variable which I stored in a thread specific using thr_setspecific.
I can only think of two objectives for threading. (1) running the various
Performance is acually very good for what I am doing. I was able to batch connections in their own thread instead of their own process. (2) running
commit transactions which cuts down on fsync calls, use prepared complex queries across multiple threads.
statements from my client using CORBA, and the various locking calls for
the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did For item (1) I see no value to this. It is a lot of work with no tangible
some performance tests for inserts benefit. If you have an old fashion pthreads implementation, it will hurt
performance because are scheduled within the single process's time slice.. If
20 clients, 900 inserts per client, 1 insert per transaction, 4 different you have a newer kernel scheduled implementation, then you will have the same
tables. scheduling as separate processes. The only thing you will need to do is
switch your brain from figuring out how to share data, to trying to figure
7.0.2 About 10:52 average completion out how to isolate data. A multithreaded implementation lacks many of the
multi-threaded 2:42 average completion benefits and robustness of a multiprocess implementation.
7.1beta3 1:13 average completion
For item (2) I can see how that could speed up queries in a low utilization
If I increased the number of inserts per transaction, multi-threaded got system, and that would be cool, but in a server that is under load, threading
closer to 7.1 for inserts. I haven't tested other other types of the queries probably be less efficient.
commands
yet.
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
Myron Scott
mkscott@sacadia.com
From lamar.owen@wgcr.org Thu Jun 28 11:14:10 2001
Return-path: <lamar.owen@wgcr.org>
Received: from www.wgcr.org (IDENT:root@www.wgcr.org [206.74.232.194])
by candle.pha.pa.us (8.10.1/8.10.1) with ESMTP id f5SFE9U18758
for <pgman@candle.pha.pa.us>; Thu, 28 Jun 2001 11:14:09 -0400 (EDT)
Received: from lowen.wgcr.org (IDENT:lowen@[10.1.2.3])
by www.wgcr.org (8.9.3/8.9.3/WGCR) with SMTP id LAA11879;
Thu, 28 Jun 2001 11:14:14 -0400
Content-Type: text/plain;
charset="iso-8859-1"
From: Lamar Owen <lamar.owen@wgcr.org>
To: Bruce Momjian <pgman@candle.pha.pa.us>
Subject: Process weight (was:Re: [GENERAL] Re: Red Hat to support PostgreSQL)
Date: Thu, 28 Jun 2001 11:14:09 -0400
X-Mailer: KMail [version 1.2]
References: <200106272258.f5RMwIb26959@candle.pha.pa.us>
In-Reply-To: <200106272258.f5RMwIb26959@candle.pha.pa.us>
MIME-Version: 1.0
Message-ID: <01062811140902.01118@lowen.wgcr.org>
Content-Transfer-Encoding: 8bit
Status: ORr
On Wednesday 27 June 2001 18:58, Bruce Momjian wrote:
> > I had almost given up on using Postgres for this system because under
> > Solaris, it just couldn't cut it (MySQL could do the work with one CPU
> > while Postgres took up even more CPU and required *both* CPUs to be
> > enabled), but when we moved the system to a Linux box, things worked
> > much better.
> Ah, back to a PostgreSQL topic. :-)
> My guess on this one is that Solaris is slower for PostgreSQL because
> process switching is _much_ heavier on Solaris than other OS's. This is
> because of the way they implemented processes in SVr4. They got quite
> heavy, almost requiring kernel threads so you weren't switching
> processes all the time.
Now, the question of the week:
Is supporting a thread model for an inefficient OS a desirable thing to do,
when more efficient OS kernels are available such as FreeBSD 4.x and Linux
2.4? My opinion is that our existing model, when used with a
connection-pooling frontend, is rather efficient. (Yes, I use a
connection-pooling frontend. Performance is rather nice, and I don't have to
have a full backend spawned for every page hit.)
In fact, on a Linux box threads show as processes. While I know that the
kernel actually supports themin a slightly different manner than processes,
they have more similarities than differences.
However, even on OS's where threads are supported, the mechanism to support
those threads must be an efficient one -- not all pthreads libraries are
created equal. Many are frontends (expensive ones, at that) for plain old
processes.
Does anyone know of a resource that details the 'weight' of processes for our
supported platforms? [reply off-list -- I'll be glad to summarize responses
to HACKERS, ADMIN, or PORTS, as appropriate, if desired.]
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Write Preview
Loading…
Cancel
Save