httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Omar W. Hannet" <ohan...@allez-oop.net>
Subject [users@httpd] Graceful restart
Date Tue, 03 Jul 2007 23:27:50 GMT
My cron issues a nightly 'apachectl graceful'.  Once in a while I
will find extra Apache processes running on the server, and when
I check them via server-status, a number of them show a state of
'G' (gracefully finishing).  They remain in that state indefinitely,
and will not go away until I issue a start-and-stop or a restart.

The 'top' command shows one of these 'G' processes is stuck in
kread, with the rest are in lockf, (I believe due to the accept
mutex which allows only one process at a time to poll sockets).

I built Apache with symbols and attached a debugger to the one
in kread state.  The stack trace shows the process calling kevent()
from the apr_pollset_poll() function in srclib/apr/poll/unix/kqueue.c,
which is called from the main loop in child_main().  When I allow
the process to continue, the call to kevent() never returns.

I will often find this in my error log when the problem occurs:
[error] (9)Bad file descriptor: apr_socket_accept: (client socket)

I am able to replicate the behavior in Apache 2.2.4 on FreeBSD 6.2.
My build configuration is as follows:

./configure \
   --with-apr=/usr/local/apr-httpd/ \
   --with-apr-util=/usr/local/apr-util-httpd/ \
   --prefix=/usr/local/httpd-2.2.4 \
   --with-mpm=prefork \
   --enable-mods=info

In httpd.conf, I add an extra Listen command:

   Listen 80
   Listen 8080

The behavior will not occur without this, because it needs to
be polling multiple sockets via kevent().

I also enable server-status by uncommenting the Include line for
httpd-info.conf and adding an 'Allow from' line in the file
conf/extra/httpd-info.conf.  This is not a required step, but allows
me to monitor more easily.

Then I start Apache and run a series of 'apachectl graceful'
commands.  Eventually the behavior will occur, but it speeds
things up if I run about a dozen of them on one command line,
separated by semi-colons.  Usually the hung 'G' process will show
up within 2-3 minutes by continually repeating in this way.

Am I seeing a known race condition in the code for graceful
restart?  Or maybe a problem in kevent?  I would appreciate any
light anyone can shed on this topic.

Thanks,
-- 
Omar W. Hannet
http://www.allez-oop.net/

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message