httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From r..@covalent.net
Subject Re: linux and SINGLE_LISTEN_UNSERIALIZED_ACCEPT
Date Sun, 05 Nov 2000 00:13:04 GMT

I'll do it later today or tomorrow sometime.  :-)

Ryan

On Sat, 4 Nov 2000, dean gaudet wrote:

> hi, a followup to an earlier post i made.  according to the information
> given below i'm going to change apache-1.3.x on linux 2.2.x and later to:
> 
> SINGLE_LISTEN_UNSERIALIZED_ACCEPT 
> USE_SYSVSEM_SERIALIZED_ACCEPT
> 
> i would really appreciate it if someone could forward these changes to
> apache-2.0.
> 
> thanks
> -dean
> 
> p.s. linux 2.2.14 is what's shipped with redhat 6.2, so it's probably the
> most common 2.2.x kernel out there.  there's security reasons for folks to
> upgrade all the way to 2.2.17, so it's a fine answer for us to suggest
> that upgrade should we find problems with earlier 2.2.x kernels.
> 
> ---------- Forwarded message ----------
> Date: Sat, 04 Nov 2000 16:08:29 +1100
> From: Andrew Morton <andrewm@uow.edu.au>
> To: dean gaudet <dean-list-linux-kernel@arctic.org>
> Cc: kumon@flab.fujitsu.co.jp, linux-kernel@vger.kernel.org
> Subject: Re: [PATCH] Re: Negative scalability by removal of 
>     lock_kernel()?(Was:Strange performance behavior of 2.4.0-test9)
> 
> dean gaudet wrote:
> > 
> > On Tue, 31 Oct 2000, Andrew Morton wrote:
> > 
> > > Dean,  it looks like the same problem will occur with flock()-based
> > > serialisation.  Does Apache/Linux ever use that option?
> > 
> > from apache/src/include/ap_config.h in the linux section there's
> > this:
> > 
> > /* flock is faster ... but hasn't been tested on 1.x systems */
> > /* PR#3531 indicates flock() may not be stable, probably depends on
> >  * kernel version.  Go back to using fcntl, but provide a way for
> >  * folks to tweak their Configuration to get flock.
> >  */
> > #ifndef USE_FLOCK_SERIALIZED_ACCEPT
> > #define USE_FCNTL_SERIALIZED_ACCEPT
> > #endif
> > 
> > so you should be able to -DUSE_FLOCK_SERIALIZED_ACCEPT to try it.
> > 
> 
> Dean,
> 
> neither flock() nor fcntl() serialisation are effective
> on linux 2.2 or linux 2.4.  This is because the file
> locking code still wakes up _all_ waiters.  In my testing
> with fcntl serialisation I have seen a single Apache
> instance get woken and put back to sleep 1,500 times
> before the poor thing actually got to service a request.
> 
> For kernel 2.2 I recommend that Apache consider using
> sysv semaphores for serialisation. They use wake-one. 
> 
> For kernel 2.4 I recommend that Apache use unserialised
> accept.
> 
> This means that you'll need to make a runtime decision
> on whether to use unserialised, serialised with sysv or
> serialised with fcntl (if sysv IPC isn't installed).
> 
> 
> In my testing I launched 3, 10, 30 or 150 Apache instances and then used
> 
> 	httperf --num-conns=2000 --num-calls=1 --uri=/index.html
> 
> to open, use and close 2000 connections.
> 
> Here are the (terrible) results on 2.4 SMP with fcntl
> serialisation:
> 
> fcntl accept, 3 servers, vanilla: 938.0 req/s
> fcntl accept, 30 servers, vanilla: 697.1 req/s
> fcntl accept, 150 servers, vanilla: 99.9 req/s         (sic)
> 
> 2.4 SMP with no serialisation:
> 
> unserialised accept, 3 servers, vanilla: 1049.0 req/s
> unserialised accept, 10 servers, vanilla: 968.8 req/s
> unserialised accept, 30 servers, vanilla: 1040.2 req/s
> unserialised accept, 150 servers, vanilla: 1091.4 req/s
> 
> 2.4 SMP with no serialisation and my patch to the
> wakeup and waitqueue code:
> 
> unserialised accept, 3 servers, task_exclusive: 1117.4 req/s
> unserialised accept, 10 servers, task_exclusive: 1118.6 req/s
> unserialised accept, 30 servers, task_exclusive: 1105.6 req/s
> unserialised accept, 150 servers, task_exclusive: 1077.1 req/s
> 
> 2.4 SMP with sysv semaphore serialisation:
> 
> sysvsem accept, 3 servers: 1001.2 req/s
> sysvsem accept, 10 servers: 1061.0 req/s
> sysvsem accept, 30 servers: 1021.2 req/s
> sysvsem accept, 150 servers: 943.6 req/s
> 
> 2.2.14 SMP with fcntl serialisation:
> 
> fcntl accept, 3 servers: 1053.8 req/s
> fcntl accept, 10 servers: 996.2 req/s
> fcntl accept, 30 servers: 934.3 req/s
> fcntl accept, 150 servers: 141.4 req/s                (sic)
> 
> 2.2.14 SMP with no serialisation:
> 
> unserialised accept, 3 servers: 1039.9 req/s
> unserialised accept, 10 servers: 983.1 req/s
> unserialised accept, 30 servers: 775.7 req/s
> unserialised accept, 150 servers: 220.7 req/s         (sic)
> 
> 2.2.14 SMP with sysv sem serialisation:
> 
> sysv accept, 3 servers: 932.2 req/s
> sysv accept, 10 servers: 910.6 req/s
> sysv accept, 30 servers: 1026.6 req/s
> sysv accept, 150 servers: 927.2 req/s
> 
> 
> Note that the first test (2.4 with fcntl serialisation) was
> with an unpatched 2.4.0-test10-pre5.  Once the simple
> flock.patch is applied, the performance with 150 servers
> doubles.  But it's still sucky.  The flock.patch change
> is effective in increasing scalability wiht a large number
> of CPUs, not a large number of httpd's.
> 
> Here's the silly patch I used to turn on sysv sem serialisation
> in Apache.  There's probably a better way than this :)
> 
> --- apache_1.3.14.orig/src/main/http_main.c	Fri Sep 29 00:32:36 2000
> +++ apache_1.3.14/src/main/http_main.c	Sat Nov  4 15:01:41 2000
> @@ -172,6 +172,13 @@
>  
>  #include "explain.h"
>  
> +/* AKPM */
> +#if 1
> +#define NEED_UNION_SEMUN
> +#define USE_SYSVSEM_SERIALIZED_ACCEPT
> +#define USE_FCNTL_SERIALIZED_ACCEPT
> +#endif
> +
>  #if !defined(max)
>  #define max(a,b)        (a > b ? a : b)
>  #endif
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> Please read the FAQ at http://www.tux.org/lkml/
> 
> 
> 
> ---------- Forwarded message ----------
> To: linux-kernel@vger.kernel.org
> From: Linus Torvalds <torvalds@transmeta.com>
> Subject: Re: [PATCH] Re: Negative scalability by removal of 
>     lock_kernel()?(Was:Strange performance behavior of 2.4.0-test9)
> Date: 3 Nov 2000 22:23:15 -0800
> Organization: Transmeta Corporation
> 
> In article <3A0399CD.8B080698@uow.edu.au>,
> Andrew Morton  <andrewm@uow.edu.au> wrote:
> >
> >neither flock() nor fcntl() serialisation are effective
> >on linux 2.2 or linux 2.4.  This is because the file
> >locking code still wakes up _all_ waiters.  In my testing
> >with fcntl serialisation I have seen a single Apache
> >instance get woken and put back to sleep 1,500 times
> >before the poor thing actually got to service a request.
> 
> Indeed.
> 
> flock() is the absolute worst case, and always has been.  I guess nobody
> every actually bothered to benchmark it.
> 
> >For kernel 2.2 I recommend that Apache consider using
> >sysv semaphores for serialisation. They use wake-one. 
> >
> >For kernel 2.4 I recommend that Apache use unserialised
> >accept.
> 
> No.
> 
> Please use unserialized accept() _always_, because we can fix that. 
> 
> Even 2.2.x can be fixed to do the wake-one for accept(), if required. 
> It's not going to be any worse than the current apache config, and
> basically the less games apache plays, the better the kernel can try to
> accomodate what apache _really_ wants done.  When playing games, you
> hide what you really want done, and suddenly kernel profiles etc end up
> being completely useless, because they no longer give the data we needed
> to fix the problem. 
> 
> Basically, the whole serialization crap is all about the Apache people
> saying the equivalent of "the OS does a bad job on something we consider
> to be incredibly important, so we do something else instead to hide it".
> 
> And regardless of _what_ workaround Apache does, whether it is the sucky
> fcntl() thing or using SysV semaphores, it's going to hide the real
> issue and mean that it never gets fixed properly.
> 
> And in the end it will result in really really bad performance. 
> 
> Instead, if apache had just done the thing it wanted to do in the first
> place, the wake-one accept() semantics would have happened a hell of a
> lot earlier. 
> 
> Now it's there in 2.4.x. Please use it. PLEASE PLEASE PLEASE don't play
> games trying to outsmart the OS, it will just hurt Apache in the long run.
> 
> 		Linus
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> Please read the FAQ at http://www.tux.org/lkml/
> 
> 
> 
> ---------- Forwarded message ----------
> Date: Sat, 4 Nov 2000 12:03:06 -0800 (PST)
> From: dean gaudet <dean-list-linux-kernel@arctic.org>
> To: Linus Torvalds <torvalds@transmeta.com>
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: [PATCH] Re: Negative scalability by removal of 
>     lock_kernel()?(Was:Strange performance behavior of 2.4.0-test9)
> X-comment: visit http://arctic.org/~dean/legal for information regarding
>     copyright and disclaimer.
> 
> On Fri, 3 Nov 2000, Linus Torvalds wrote:
> 
> > Please use unserialized accept() _always_, because we can fix that.
> 
> i can unserialise the single socket case, but the multiple socket case is
> not so simple.
> 
> the executive summary is that when you've got multiple sockets you have to
> use select().  select is necessarily wake-all.  remember there's N
> children trying to do select/accept.  if the listening socket is
> non-blocking then you spin in N-1 children; if it's blocking then you
> starve other sockets.
> 
> see http://www.apache.org/docs/misc/perf-tuning.html, search for "multiple
> sockets" for my full analysis of the problem.
> 
> > Instead, if apache had just done the thing it wanted to do in the first
> > place, the wake-one accept() semantics would have happened a hell of a
> > lot earlier.
> 
> counter-example:  freebsd had wake-one semantics a few years before linux.
> 
> revision 1.237
> date: 1998/09/29 01:22:57;  author: marc;  state: Exp;  lines: +1 -0
> Unserialized accept() should be safe (in all versions) and efficient
> (in anything vaguely recent) on FreeBSD.
> 
> ok, we done finger pointing? :)
> 
> -dean
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> Please read the FAQ at http://www.tux.org/lkml/
> 
> 


_______________________________________________________________________________
Ryan Bloom                        	rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------


Mime
View raw message