httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul J. Reder" <rede...@raleigh.ibm.com>
Subject Re: Suggested direction for fixing threaded mpm thread startup.
Date Fri, 20 Apr 2001 15:48:38 GMT
Greg Ames wrote:
>                                                         One theme
> I heard over and over was that heavyweight web workloads (database,
> mod_perl) should be isolated from lightweight workloads in order to
> minimize the memory footprint.  Makes a lot of sense to me.

This is more an argument for moving heavy computation (mod_perl, mod_include,
mod_rewrite) to another process. Big files, especially ones that don't
require any processing - just sending, aren't a problem per-se. They just
take a noticeable amount of communications bandwidth. But this doesn't
impact the processor that much, especially if you have multiple processes/threads
to divide the request processing workload over.

> 
> But in order to have our problem here, a prerequisite is that
> heavyweight and lightweight requests are being served by the same
> process.  hmmmm...seems like a sharp sysadmin would start by separating
> the long running stuff onto a different server instance or some such,
> and therefore would be much less likely to hit this situation.

Actually this problem can happen with just lightweight, or just heavyweight
processes as well (especially with just heavyweight). The problem has to
do with a request still being processed when max_requests_per_child is hit.
It may take more or less time, but this will happen in all of the cases.
The variance in size or in the max_requests_per_child setting will only
aggravate or ameliorate the problem, not eliminate it (unless max_requests_
per_child is set to 0).

> OK, with the 3 zillion users we have out there, we can't expect that all
> the admins will figure this out (especially judging by some of the
> recent questions on this list...sheesh!), or that the clever admins have
> the time/resources to shuffle things around.  My point is that I don't
> see this as being much of a problem for high end shops, so perhaps we
> can focus on something really simple for the low end folks for starters.
> 
> AFAIK, Paul hit this problem because of idle_server_maintenance and/or
> MaxRequestsPerChild - true?

Not true. As stated above, I aggravated the problem with the settings (which
as far as I know are the same settings used on the apache.org site) but the
settings did not cause the problem (except for the fact that max_requests_
per_child was > 0).

>                             If that's really the extent of it, I
> believe there's a pretty easy solution.  If idle_server_maintenance
> triggers this, it is thrashing (i.e., first trying to cut back on the
> number of processes, then trying to increase them, then trying to cut
> back again while the first process with the mixed weight workloads is
> still hanging around).  So duh!, let's make it stop thrashing.  All we
> have to do is limit the number of processes that can be terminated by
> idle_server_maintenance to one at a time.  Piece of cake.

Perform_idle_server_maintenance is not thrashing. What happens is that
Apache does a pretty good job of distributing the requests evenly among
the processes/threads. This means that many processes/threads reach 
max_requests_per_child in nearly the same time frame. When many processes/
threads depart, perform_idle_server_maintenance works to increase the
number to the operational level (one-for-one replacement).

In a separate situation, after a flurry of requests have come in and been processed,
the worker count is high, perform_idle_server_maintenance starts killing off
workers (one server at a time). Since only servers are killed off, N workers will
go with it. So the threads_per_child and min/max_spare_threads should be tuned
carefully, knowing that if it will kill a server with 100 threads because there
are 10 idle we're now going to be 90 short!

The change in request volume can happen simply because of changes in request traffic
flow, or because the requesters are waiting on large requests and don't have the
bandwidth/cpu to continue the high volume of requests. So the apparent slowdown may
be artificial.

> 
> If you buy that, then why doesn't that solution work for
> MaxRequestsPerChild as well?  Think about it.  That number is basically
> for memory leak tolerance on a buggy server.  How important is it that
> we always stop a process precisely when we hit that number?  not very,
> IMO.  For the low end folks it's probably the default anyway, which we
> developers just pulled out of the air.  So if we have our mixed workload
> process basket case scenario going on, just hold off on killing any more
> processes until the first one terminates completely.

The processes get killed by perform_idle_server_maintenance (one per second),
the workers go away by themselves when they reach max_requests_per_child.
There is a little mixing of apples and oranges concepts here. We could change the
perform_idle_server_maintenance code so that it wouldn't kill another server
(even after the 1 second delay) unless the previous kill had finished, but in
fact I believe what happens now is that it would find the same process still 
there and try to kill it again - not another one.

In order for the fuzzy max_requests_per_child idea to have any impact, each
child would have to vary the fuzziness independently (since many threads hit
mr/c "simultaneously"). Alternatively, the workers would all have to communicate
to keep multiple from exiting together (since the workers make this decision
on their own). The final alternative is to not distribute the requests so evenly,
thus encouraging one set of workers to die first before the primary workload is
shifted to the next set.

> 
> Beside idle_server_maintenance and MaxRequestsPerChild, are there any
> *realistic* scenarios that trigger this problem?

As long as max_requests_per_child is set to anything other than 0 and request
volume has any variance to it, this problem will occur. It may take more or
less time, but it will still occur in every scenario (*realistic* or otherwise).

> 
> Greg

-- 
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it.  Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein

Mime
View raw message