httpd-bugs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzi...@apache.org
Subject DO NOT REPLY [Bug 42829] New: - graceful restart with multiple listeners using prefork MPM can result in hung processes on FreeBSD
Date Fri, 06 Jul 2007 21:45:17 GMT
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG·
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=42829>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND·
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=42829

           Summary: graceful restart with multiple listeners using prefork
                    MPM can result in hung processes on FreeBSD
           Product: Apache httpd-2
           Version: 2.2.4
          Platform: PC
        OS/Version: FreeBSD
            Status: NEW
          Keywords: PatchAvailable
          Severity: normal
          Priority: P2
         Component: prefork
        AssignedTo: bugs@httpd.apache.org
        ReportedBy: nmower@verio.net


Intermittently after a graceful restart (apachectl graceful) extra Apache
processes are left running on the server.  A check to server-status shows a
number of them to be in state 'G' (gracefully finishing).  They remain in that
state indefinitely, and will not go away until a stop-and-start or a restart is
issued.

The 'top' command shows one of these 'G' processes is stuck in kqread, while the
rest are in lockf.  (I believe the processes in lockf are blocking on the accept
mutex, which is held by the process in kqread and therefore is never cleared.)

A stack trace of the process in kqread state shows the process calling kevent()
from the apr_pollset_poll() function in srclib/apr/poll/unix/kqueue.c, which is
called from the main loop in child_main() of server/mpm/prefork/profork.c.  When
the process is allowed to  continue, it never returns from kevent().

Frequently the error log contains the following message when the problem has
occurred:
[error] (9)Bad file descriptor: apr_socket_accept: (client socket)

The behavior can be replicated in the Apache 2.2.4 release version on FreeBSD
6.2 stable.  The build configuration is as follows:

./configure \
  --with-apr=/usr/local/apr-httpd/ \
  --with-apr-util=/usr/local/apr-util-httpd/ \
  --prefix=/usr/local/httpd-2.2.4 \
  --with-mpm=prefork \
  --enable-mods=info

The problem only occurs when at least one extra listener is configured in
httpd.conf:
  Listen 80
  Listen 8080

When replicating the problem it is useful to enable server-status by
uncommenting the Include line for httpd-info.conf and adding an 'Allow from'
line in the file conf/extra/httpd-info.conf.

After Apache has been restarted a series of 'apachectl graceful' commands will
usually trigger the problem.  It speeds things up to run several of them on one
command line (separated by semi-colons), but care is required not to run into
another Apache bug with similar steps to duplicate (ID 39311).  Usually the hung
'G' process will show up within 2-3 minutes by continually repeating in this way.

Here is my take on what may happening, along with a patch that works for me:

There appears to be a race condition between Apache prefork and kevent() that
depends upon the timing of the receipt of SIGUSR1 in the child process.  When
SIGUSR1 is received, the current signal handler for prefork closes all listeners
immediately.  I believe if the close occurs before the process gets into
kevent() to poll the sockets, it can cause kevent() to hang.

My solution was to wait until after the !die_now loop to close listeners.  The
signal handler merely sets the die_now flag, nothing more.  Here is my proposed
patch:

--- prefork.c.orig      Fri Jul  6 11:12:53 2007
+++ prefork.c   Fri Jul  6 11:18:24 2007
@@ -330,8 +330,6 @@
 
 static void stop_listening(int sig)
 {
-    ap_close_listeners();
-
     /* For a graceful stop, we want the child to exit when done */
     die_now = 1;
 }
@@ -657,6 +655,7 @@
             die_now = 1;
         }
     }
+    ap_close_listeners();
     clean_child_exit(0);
 }

-- 
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org


Mime
View raw message