httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Darroch <chr...@pearsoncmg.com>
Subject [PATCH] #39275 MaxClients on startup [Was: Bug in 2.0.56-dev]
Date Tue, 11 Apr 2006 20:50:57 GMT
Hi --

Alexander Lazic wrote:

>> After 'make install' i started apache, then some seconds later i got the
>> message '...MaxClients reached...' but there was no entry in the access
>> log, and nobody have make a request to this server.

Jeff Trawick wrote:

> There are problems accounting for child processes which are trying to
> initialize that result in the parent thinking it needs to create more
> children.  The less harmful flavor is when it thinks (incorrectly) it
> is already at MaxClients and issues the "reached MaxClients" message. 
> More disturbing is when MaxClients is very high and the parent keeps
> creating new children using exponential ramp-up.  That can be very
> painful.

   I have been seeing something similar with 2.2.0 using the worker
MPM, where with the following settings, I get over 10 child processes
initializing immediately (e.g., up to 15), and then they drop back to
10.  I see the "server reached MaxClients" message as well right
after httpd startup, although nothing is connecting yet.

<IfModule mpm_worker_module>
    StartServers         10
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads     100
    ThreadsPerChild      10
</IfModule>

   In my case, the problem relates to how long the child_init phase
takes to execute.  I can "tune" this by raising DBDMin (and DBDKeep)
so that mod_dbd attempts to open increasingly large numbers of
DB connections during child_init.  With DBDMin set to 0 or 1,
all is well; no funny behaviour.  Up at DBDMin and DBDKeep at 3,
that's when (for me) things go pear-shaped.

   In server/mpm/worker/worker.c, after make_child() creates a
child process it immediately sets the scoreboard parent slot's pid
value.  The main process goes into server_main_loop() and begins
executing perform_idle_server_maintenance() every second; this
looks at any process with a non-zero pid in the scoreboard and
assumes that any of its worker threads marked SERVER_DEAD are,
in fact, dead.

   However, if the child processes are starting "slowly" because
ap_run_child_init() in child_main() is taking its time, then
start_threads() hasn't even been run yet, so the threads aren't
marked SERVER_STARTING -- they're just set to 0 as the default
value.  But 0 == SERVER_DEAD, so the main process sees a lot
of dead worker threads and begins spawning new child processes,
up to MaxClients/ThreadsPerChild in the worst case.  In this case,
when no worker threads have started yet, but all possible child
processes have been spawned (and are working through their
child_init phases), then the following is true and the
"server reached MaxClients" message is printed, even though
the server hasn't started accepting connections yet:

    else if (idle_thread_count < min_spare_threads) {
        /* terminate the free list */
        if (free_length == 0) {

   I considered wedging another thread status into the
scoreboard, between SERVER_DEAD (the initial value) and
SERVER_STARTING.  The make_child() would set all the thread
slots to this value and start_threads() would later flip them
to SERVER_STARTING after actually creating the worker threads.

   That would have various ripple effects on other bits of
httpd, though, like mod_status and other MPMs, etc.  So instead
I tried adding a status field to the process_score scoreboard
structure, and making the following changes to worker.c such that
this field is set by make_child to SERVER_STARTING and then
changed to SERVER_READY once the start thread that runs
start_threads() has done its initial work.

   During this period, while the new child process is running
ap_run_child_init() and friends, perform_idle_server_maintenance()
just counts that child process's worker threads as all being
effectively in SERVER_STARTING mode.  Once the process_score.status
field changes to SERVER_READY, perform_idle_server_maintenance()
begins to look at the individual thread status values.

   Any thoughts?  The patch in Bugzilla doesn't address other
MPMs that might see the same behaviour (event, and maybe prefork?)

http://issues.apache.org/bugzilla/show_bug.cgi?id=39275

It also doesn't necessarily play ideally well with the fact that
new child processes can gradually take over thread slots in
the scoreboard from a gracefully exiting old process -- the
count of idle threads for that process will be pegged (only
by perform_idle_server_maintenance()) at ap_threads_per_child
until the new process creates its first new worker thread.
But, that may be just fine....  I'll keep poking around and
testing and maybe a better idea will present itself.

Chris.


Mime
View raw message