httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Darroch <chr...@pearsoncmg.com>
Subject Re: shutdown and linux poll()
Date Mon, 20 Feb 2006 22:21:46 GMT
Hi --

   I've crafted what seems to me like a reasonably minimal set of
patches to deal with the issue I described in this thread:

http://marc.theaimsgroup.com/?l=apache-httpd-dev&m=113986864730305&w=2

   The crux of the problem is that on Linux, when using httpd with
the worker MPM (and probably the event MPM too), hard restarts and
shutdowns often end up sending SIGKILL to httpd child processes
because those processes are waiting for their worker threads to
finish polling on Keep-Alive connections.

   Apparently, on most OSes, if one thread closes a socket descriptor
then other threads polling on it immediately get a return value.
This certainly seems to be the case on Solaris.  But on Linux,
worker threads polling on their sockets in apr_wait_for_io_or_timeout()
don't get an error return value until the full (usually 15 second)
Keep-Alive timeout period is up.  The main httpd process deems that
too long, and issues SIGKILL to the child processes.

   For me personally, the consequence is that all my nice cleanup
handlers registered against the memory pool that's passed during the
child_init stage never get called.  This is particularly painful
if one is hoping to, for example, cleanly shut down DB connections
that one has opened with mod_dbd/apr_dbd.  In the case of mod_dbd,
it opens its reslist of apr_dbd connections against the pool
passed in the child_init stage, which with the worker MPM is its
pchild pool.  When SIGKILL is applied, the apr_pool_destroy(pchild)
call is often not reached, so DB disconnections don't occur;
even if I'm trying to shut down httpd in a hurry, I don't really
want that to happen if at all possible.

   Without further ado, then, my initial patches.  These are
Unix-only at the moment; I have little experience with other OSes.
If anyone wants to propose something better, and/or suggest
changes, that would be superb.  In the meantime, since these
work for me, I'll start applying them against APR and httpd for
my own use.

   First, the APR patches (against trunk):

===================================
--- include/apr_network_io.h.orig       2006-02-20 16:20:44.841609000
-0500
+++ include/apr_network_io.h    2006-02-20 16:24:19.993533339 -0500

@@ -99,6 +99,7 @@

                                     * until data is available.

                                     * @see apr_socket_accept_filter

                                     */

+#define APR_INTERRUPT_WAIT  65536 /**< Return from IO wait on interrupt
*/

 /** @} */


--- network_io/unix/sockopt.c.orig      2006-02-17 11:24:13.058691778 -0500
+++ network_io/unix/sockopt.c   2006-02-17 11:28:08.910410867 -0500

@@ -318,6 +318,9 @@

         return APR_ENOTIMPL;

 #endif

         break;

+    case APR_INTERRUPT_WAIT:

+        apr_set_option(sock, APR_INTERRUPT_WAIT, on);

+        break;

     default:

         return APR_EINVAL;

     }

--- support/unix/waitio.c.orig  2005-07-09 03:07:17.000000000 -0400
+++ support/unix/waitio.c       2006-02-17 11:23:42.620856949 -0500

@@ -49,7 +49,8 @@


     do {

         rc = poll(&pfd, 1, timeout);

-    } while (rc == -1 && errno == EINTR);

+    } while (rc == -1 && errno == EINTR &&

+             (f || !apr_is_option_set(s, APR_INTERRUPT_WAIT)));

     if (rc == 0) {

         return APR_TIMEUP;

     }

===================================

   Second, the httpd patches (also against trunk):

===================================
--- server/mpm/worker/worker.c.orig	2006-02-20 16:26:55.302701000 -0500
+++ server/mpm/worker/worker.c	2006-02-20 16:46:44.764980568 -0500
@@ -213,6 +213,19 @@
  */
 #define LISTENER_SIGNAL     SIGHUP

+/* The WORKER_SIGNAL signal will be sent from the main thread to the
+ * worker threads after APR_INTERRUPT_WAIT is set true on their sockets.
+ * This ensures that on systems (i.e., Linux) where closing the worker
+ * socket doesn't awake the worker thread when it is polling on the socket
+ * (especially after in apr_wait_for_io_or_timeout() when handling
+ * Keep-Alive connections), close_worker_sockets() and join_workers()
+ * still function in timely manner and allow ungraceful shutdowns to
+ * proceed to completion.  Otherwise join_workers() doesn't return
+ * before the main process decides the child process is non-responsive
+ * and sends a SIGKILL.
+ */
+#define WORKER_SIGNAL       AP_SIG_GRACEFUL
+
 /* An array of socket descriptors in use by each thread used to
  * perform a non-graceful (forced) shutdown of the server. */
 static apr_socket_t **worker_sockets;
@@ -222,6 +235,7 @@
     int i;
     for (i = 0; i < ap_threads_per_child; i++) {
         if (worker_sockets[i]) {
+            apr_socket_opt_set(worker_sockets[i], APR_INTERRUPT_WAIT, 1);
             apr_socket_close(worker_sockets[i]);
             worker_sockets[i] = NULL;
         }
@@ -822,6 +836,11 @@
     ap_scoreboard_image->servers[process_slot][thread_slot].generation
= ap_my_generation;
     ap_update_child_status_from_indexes(process_slot, thread_slot,
SERVER_STARTING, NULL);

+#ifdef HAVE_PTHREAD_KILL
+    unblock_signal(WORKER_SIGNAL);
+    apr_signal(WORKER_SIGNAL, dummy_signal_handler);
+#endif
+
     while (!workers_may_exit) {
         if (!is_idle) {
             rv = ap_queue_info_set_idle(worker_queue_info, last_ptrans);
@@ -1077,6 +1096,13 @@

     for (i = 0; i < ap_threads_per_child; i++) {
         if (threads[i]) { /* if we ever created this thread */
+#ifdef HAVE_PTHREAD_KILL
+            apr_os_thread_t *worker_os_thread;
+
+            apr_os_thread_get(&worker_os_thread, threads[i]);
+            pthread_kill(*worker_os_thread, WORKER_SIGNAL);
+#endif
+
             rv = apr_thread_join(&thread_rv, threads[i]);
             if (rv != APR_SUCCESS) {
                 ap_log_error(APLOG_MARK, APLOG_CRIT, rv, ap_server_conf,
===================================

Chris.

-- 
GPG Key ID: 366A375B
GPG Key Fingerprint: 485E 5041 17E1 E2BB C263  E4DE C8E3 FA36 366A 375B


Mime
View raw message