Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 06C50200BE7 for ; Tue, 20 Dec 2016 13:53:23 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 053DE160B29; Tue, 20 Dec 2016 12:53:23 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A7382160B1B for ; Tue, 20 Dec 2016 13:53:21 +0100 (CET) Received: (qmail 39255 invoked by uid 500); 20 Dec 2016 12:53:15 -0000 Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm Precedence: bulk Reply-To: dev@httpd.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@httpd.apache.org Received: (qmail 39243 invoked by uid 99); 20 Dec 2016 12:53:15 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Dec 2016 12:53:15 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id F206D184905 for ; Tue, 20 Dec 2016 12:53:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=6.31 tests=[none] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id RXYws3v03Gap for ; Tue, 20 Dec 2016 12:53:06 +0000 (UTC) Received: from cloud1-vm154.de-nserver.de (cloud1-vm154.de-nserver.de [178.250.10.56]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id E98925FC87 for ; Tue, 20 Dec 2016 12:53:05 +0000 (UTC) Received: (qmail 16579 invoked from network); 20 Dec 2016 13:52:26 +0100 X-Fcrdns: No Received: from phoffice.de-nserver.de (HELO [10.11.11.165]) (185.39.223.5) (smtp-auth username hostmaster@profihost.com, mechanism plain) by cloud1-vm154.de-nserver.de (qpsmtpd/0.92) with (ECDHE-RSA-AES256-SHA encrypted) ESMTPSA; Tue, 20 Dec 2016 13:52:26 +0100 Subject: Re: Frequent wake-ups for mpm_event To: dev@httpd.apache.org References: <8610c372-7ac4-c1bc-64fc-4de981267f22@profihost.ag> <9413e65e-b037-c07d-267e-dcd7011386ff@profihost.ag> From: Stefan Priebe - Profihost AG Message-ID: Date: Tue, 20 Dec 2016 13:52:26 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-User-Auth: Auth by hostmaster@profihost.com through 185.39.223.5 archived-at: Tue, 20 Dec 2016 12:53:23 -0000 Hi Yann, here is a follow up on this old thread with some problems again. See below. Am 23.09.2016 um 21:26 schrieb Stefan Priebe - Profihost AG: > Hi Yann, > Hi Luca, > > right now i had another server which shows: > AH00485: scoreboard is full, not at MaxRequestWorkers msg. > > I never saw that before applying the event patch. > > A fullstatus view is not answered - so the whole apache hangs. Also it > has 84 running processes instead of normally just 4-8. > # ps aux|grep httpd | wc -l > 84 > > There are also not many connections: > # netstat -na | egrep ":80|:443"|wc -l > 110 > > Even after a restart all and even more httpd processes are there: > # ps aux|grep httpd | wc -l > 89 > > ps aux also tell me that there a lot of pretty old processes: > # ps aux|grep httpd | grep Sep22 | wc -l > 40 > > and even a lot of processes are from Sep 23 many hours ago. So i don't > think this is bug #53555. > > It just seems a lot of threads are hanging / deadlocked. > > gdb says: > (gdb) bt > #0 0x00007f9235b634db in pthread_join () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00007f9235d9e9bb in apr_thread_join () from > /usr/lib/x86_64-linux-gnu/libapr-1.so.0 > #2 0x000000000055b283 in join_workers () > #3 0x000000000055b8cc in child_main () > #4 0x000000000055ba3f in make_child () > #5 0x000000000055c21e in perform_idle_server_maintenance () > #6 0x000000000055c619 in server_main_loop () > #7 0x000000000055c959 in event_run () > #8 0x0000000000445f91 in ap_run_mpm () > #9 0x000000000043dd19 in main () > > > bt from all threads of an old httpd process: > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > 0x00007f9235b634db in pthread_join () from > /lib/x86_64-linux-gnu/libpthread.so.0 > > Thread 52 (Thread 0x7f92325ee700 (LWP 13704)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 51 (Thread 0x7f9231ded700 (LWP 13705)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 50 (Thread 0x7f92315ec700 (LWP 13706)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 49 (Thread 0x7f9230deb700 (LWP 13707)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 48 (Thread 0x7f92305ea700 (LWP 13708)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 47 (Thread 0x7f922fde9700 (LWP 13709)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 46 (Thread 0x7f922f5e8700 (LWP 13710)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 45 (Thread 0x7f922ede7700 (LWP 13711)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 44 (Thread 0x7f922e5e6700 (LWP 13712)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 43 (Thread 0x7f922dde5700 (LWP 13713)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 42 (Thread 0x7f922d5e4700 (LWP 13714)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 41 (Thread 0x7f922cde3700 (LWP 13715)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 40 (Thread 0x7f922c5e2700 (LWP 13716)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 39 (Thread 0x7f922bde1700 (LWP 13717)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 38 (Thread 0x7f922b5e0700 (LWP 13718)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 37 (Thread 0x7f922addf700 (LWP 13719)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 36 (Thread 0x7f922a5de700 (LWP 13720)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 35 (Thread 0x7f9229ddd700 (LWP 13721)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 34 (Thread 0x7f92295dc700 (LWP 13722)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 33 (Thread 0x7f9228ddb700 (LWP 13723)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 32 (Thread 0x7f92285da700 (LWP 13724)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 31 (Thread 0x7f9227dd9700 (LWP 13725)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 30 (Thread 0x7f92275d8700 (LWP 13726)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 29 (Thread 0x7f9226dd7700 (LWP 13727)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 28 (Thread 0x7f92265d6700 (LWP 13728)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000005286f9 in get_mplx_next () > #2 0x000000000052fb34 in execute () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 27 (Thread 0x7f92255d4700 (LWP 13783)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 26 (Thread 0x7f9224dd3700 (LWP 13784)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 25 (Thread 0x7f921ffff700 (LWP 13785)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 24 (Thread 0x7f921f7fe700 (LWP 13786)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 23 (Thread 0x7f921effd700 (LWP 13787)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 22 (Thread 0x7f921e7fc700 (LWP 13788)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 21 (Thread 0x7f921dffb700 (LWP 13789)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 20 (Thread 0x7f921d7fa700 (LWP 13790)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 19 (Thread 0x7f921cff9700 (LWP 13791)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 18 (Thread 0x7f921c7f8700 (LWP 13792)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 17 (Thread 0x7f921bff7700 (LWP 13793)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 16 (Thread 0x7f921b7f6700 (LWP 13794)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 15 (Thread 0x7f921aff5700 (LWP 13795)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 14 (Thread 0x7f921a7f4700 (LWP 13796)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 13 (Thread 0x7f9219ff3700 (LWP 13797)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 12 (Thread 0x7f92197f2700 (LWP 13798)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 11 (Thread 0x7f9218ff1700 (LWP 13799)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 10 (Thread 0x7f92187f0700 (LWP 13800)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 9 (Thread 0x7f9217fef700 (LWP 13801)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 8 (Thread 0x7f92177ee700 (LWP 13802)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 7 (Thread 0x7f9216fed700 (LWP 13803)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 6 (Thread 0x7f92167ec700 (LWP 13804)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 5 (Thread 0x7f9215feb700 (LWP 13805)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 4 (Thread 0x7f92157ea700 (LWP 13806)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 3 (Thread 0x7f9214fe9700 (LWP 13807)): > #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x000000000055fa26 in ap_queue_pop_something () > #2 0x000000000055a633 in worker_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 2 (Thread 0x7f92147e8700 (LWP 13808)): > #0 0x00007f9235897bb3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007f9235d99fc3 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0 > #2 0x00000000005595bd in listener_thread () > #3 0x00007f9235b620a4 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > Thread 1 (Thread 0x7f9237159780 (LWP 13703)): > #0 0x00007f9235b634db in pthread_join () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00007f9235d9e9bb in apr_thread_join () from > /usr/lib/x86_64-linux-gnu/libapr-1.so.0 > #2 0x000000000055b283 in join_workers () > #3 0x000000000055b8cc in child_main () > #4 0x000000000055ba3f in make_child () > #5 0x000000000055c21e in perform_idle_server_maintenance () > #6 0x000000000055c619 in server_main_loop () > #7 0x000000000055c959 in event_run () > #8 0x0000000000445f91 in ap_run_mpm () > #9 0x000000000043dd19 in main () > > > Stefan > > Am 23.09.2016 um 16:53 schrieb Yann Ylavic: >> With the patch this time... >> >> On Fri, Sep 23, 2016 at 4:53 PM, Yann Ylavic wrote: >>> On Fri, Sep 23, 2016 at 3:13 PM, Yann Ylavic wrote: >>>> >>>> If you can try another (additive) patch on the faulty server, it would >>>> be awesome to test the one from [1], not sure it applies correctly >>>> with my 2.4.x patch though, possibly better with the trunk version >>>> over [1] (let me know otherwise). >>> >>> Not a big deal actually, the attached patch includes both [1] and [2] combined. >>> >>> Regards, >>> Yann. >>> >>> [1] https://bz.apache.org/bugzilla/show_bug.cgi?id=53555#c55 >>> [2] https://bz.apache.org/bugzilla/show_bug.cgi?id=57399#c9 Today i had another server giving no answers to any requests. apache fullstatus did not respond. [ ~]# netstat -na | grep ":80" | wc -l 29 [ ~]# netstat -na | grep ":443" | wc -l 57 ps aux|grep -i httpd root 575 0.0 0.2 161060 40092 ? Ss Dec07 1:47 /usr/local/apache2/bin/httpd -k start wwwrun 17894 3.1 1.6 4891824 278748 ? Sl 11:52 3:42 /usr/local/apache2/bin/httpd -k start wwwrun 25574 0.0 0.0 108316 6684 ? S 06:00 0:00 /usr/local/apache2/bin/httpd -k start wwwrun 25575 0.0 0.0 108788 6504 ? S 06:00 0:00 /usr/local/apache2/bin/httpd -k start wwwrun 25741 2.1 0.6 4778760 110512 ? Sl 06:00 10:02 /usr/local/apache2/bin/httpd -k start wwwrun 28094 3.1 1.5 4916324 252428 ? Sl 06:31 13:59 /usr/local/apache2/bin/httpd -k start wwwrun 29575 5.2 2.6 5081956 443192 ? Sl 06:53 22:00 /usr/local/apache2/bin/httpd -k start an strace only shows: [pid 28100] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL [pid 28099] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6192, NULL [pid 28101] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL [pid 28099] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable) [pid 28098] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6187, NULL [pid 28099] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL [pid 28098] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable) [pid 28097] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6206, NULL [pid 28098] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL [pid 28097] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable) [pid 28096] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6200, NULL [pid 28097] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL [pid 28096] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable) [pid 28095] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6168, NULL [pid 28096] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL [pid 28095] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable) [pid 28094] read(8, [pid 28095] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL [pid 28542] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 28542] futex(0x173d344, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51873, {1482238254, 882451000}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 28542] futex(0x173d344, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51875, {1482238255, 883132000}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 28542] futex(0x173d344, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51877, {1482238256, 883880000}, ffffffff) = -1 ETIMEDOUT (Connection timed out) [pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 28542] futex(0x173d344, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51879, {1482238257, 884511000}, ffffffff [pid 28146] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 28146] futex(0x7fc9ed2540a8, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 28146] futex(0x7fc9ca61a28c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 4671, {1482238262, 784057000}, ffffffff [pid 28542] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0 [pid 28542] futex(0x173d344, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51881, {1482238258, 885329000}, ffffffffq) = -1 ETIMEDOUT (Connection timed out) [pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0 gdb bt shows this for all httpd childs: (gdb) bt #0 0x00007fc9f2b7dadd in read () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x000000000047cd89 in ap_mpm_podx_check () #2 0x000000000055b98f in child_main () #3 0x000000000055bc2e in make_child () #4 0x000000000055c5a0 in perform_idle_server_maintenance () #5 0x000000000055c940 in server_main_loop () #6 0x000000000055cc80 in event_run () #7 0x0000000000445ff1 in ap_run_mpm () #8 0x000000000043dd79 in main () greets, Stefan