httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Priebe - Profihost AG <s.pri...@profihost.ag>
Subject Re: Frequent wake-ups for mpm_event
Date Tue, 20 Dec 2016 12:52:26 GMT
Hi Yann,

here is a follow up on this old thread with some problems again. See below.

Am 23.09.2016 um 21:26 schrieb Stefan Priebe - Profihost AG:
> Hi Yann,
>   Hi Luca,
> 
> right now i had another server which shows:
> AH00485: scoreboard is full, not at MaxRequestWorkers msg.
> 
> I never saw that before applying the event patch.
> 
> A fullstatus view is not answered - so the whole apache hangs. Also it
> has 84 running processes instead of normally just 4-8.
> # ps aux|grep httpd | wc -l
> 84
> 
> There are also not many connections:
> # netstat -na | egrep ":80|:443"|wc -l
> 110
> 
> Even after a restart all and even more httpd processes are there:
> # ps aux|grep httpd | wc -l
> 89
> 
> ps aux also tell me that there a lot of pretty old processes:
> # ps aux|grep httpd | grep Sep22 | wc -l
> 40
> 
> and even a lot of processes are from Sep 23 many hours ago. So i don't
> think this is bug #53555.
> 
> It just seems a lot of threads are hanging / deadlocked.
> 
> gdb says:
> (gdb) bt
> #0  0x00007f9235b634db in pthread_join () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00007f9235d9e9bb in apr_thread_join () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2  0x000000000055b283 in join_workers ()
> #3  0x000000000055b8cc in child_main ()
> #4  0x000000000055ba3f in make_child ()
> #5  0x000000000055c21e in perform_idle_server_maintenance ()
> #6  0x000000000055c619 in server_main_loop ()
> #7  0x000000000055c959 in event_run ()
> #8  0x0000000000445f91 in ap_run_mpm ()
> #9  0x000000000043dd19 in main ()
> 
> 
> bt from all threads of an old httpd process:
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> 0x00007f9235b634db in pthread_join () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> 
> Thread 52 (Thread 0x7f92325ee700 (LWP 13704)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 51 (Thread 0x7f9231ded700 (LWP 13705)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 50 (Thread 0x7f92315ec700 (LWP 13706)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 49 (Thread 0x7f9230deb700 (LWP 13707)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 48 (Thread 0x7f92305ea700 (LWP 13708)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 47 (Thread 0x7f922fde9700 (LWP 13709)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 46 (Thread 0x7f922f5e8700 (LWP 13710)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 45 (Thread 0x7f922ede7700 (LWP 13711)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 44 (Thread 0x7f922e5e6700 (LWP 13712)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 43 (Thread 0x7f922dde5700 (LWP 13713)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 42 (Thread 0x7f922d5e4700 (LWP 13714)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 41 (Thread 0x7f922cde3700 (LWP 13715)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 40 (Thread 0x7f922c5e2700 (LWP 13716)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 39 (Thread 0x7f922bde1700 (LWP 13717)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 38 (Thread 0x7f922b5e0700 (LWP 13718)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 37 (Thread 0x7f922addf700 (LWP 13719)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 36 (Thread 0x7f922a5de700 (LWP 13720)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 35 (Thread 0x7f9229ddd700 (LWP 13721)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 34 (Thread 0x7f92295dc700 (LWP 13722)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 33 (Thread 0x7f9228ddb700 (LWP 13723)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 32 (Thread 0x7f92285da700 (LWP 13724)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 31 (Thread 0x7f9227dd9700 (LWP 13725)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 30 (Thread 0x7f92275d8700 (LWP 13726)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 29 (Thread 0x7f9226dd7700 (LWP 13727)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 28 (Thread 0x7f92265d6700 (LWP 13728)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00000000005286f9 in get_mplx_next ()
> #2  0x000000000052fb34 in execute ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 27 (Thread 0x7f92255d4700 (LWP 13783)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 26 (Thread 0x7f9224dd3700 (LWP 13784)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 25 (Thread 0x7f921ffff700 (LWP 13785)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 24 (Thread 0x7f921f7fe700 (LWP 13786)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 23 (Thread 0x7f921effd700 (LWP 13787)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 22 (Thread 0x7f921e7fc700 (LWP 13788)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 21 (Thread 0x7f921dffb700 (LWP 13789)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 20 (Thread 0x7f921d7fa700 (LWP 13790)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 19 (Thread 0x7f921cff9700 (LWP 13791)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 18 (Thread 0x7f921c7f8700 (LWP 13792)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 17 (Thread 0x7f921bff7700 (LWP 13793)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 16 (Thread 0x7f921b7f6700 (LWP 13794)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 15 (Thread 0x7f921aff5700 (LWP 13795)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 14 (Thread 0x7f921a7f4700 (LWP 13796)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 13 (Thread 0x7f9219ff3700 (LWP 13797)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 12 (Thread 0x7f92197f2700 (LWP 13798)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 11 (Thread 0x7f9218ff1700 (LWP 13799)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 10 (Thread 0x7f92187f0700 (LWP 13800)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 9 (Thread 0x7f9217fef700 (LWP 13801)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 8 (Thread 0x7f92177ee700 (LWP 13802)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 7 (Thread 0x7f9216fed700 (LWP 13803)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 6 (Thread 0x7f92167ec700 (LWP 13804)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 5 (Thread 0x7f9215feb700 (LWP 13805)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 4 (Thread 0x7f92157ea700 (LWP 13806)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 3 (Thread 0x7f9214fe9700 (LWP 13807)):
> #0  0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x000000000055fa26 in ap_queue_pop_something ()
> #2  0x000000000055a633 in worker_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 2 (Thread 0x7f92147e8700 (LWP 13808)):
> #0  0x00007f9235897bb3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x00007f9235d99fc3 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2  0x00000000005595bd in listener_thread ()
> #3  0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> 
> Thread 1 (Thread 0x7f9237159780 (LWP 13703)):
> #0  0x00007f9235b634db in pthread_join () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00007f9235d9e9bb in apr_thread_join () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2  0x000000000055b283 in join_workers ()
> #3  0x000000000055b8cc in child_main ()
> #4  0x000000000055ba3f in make_child ()
> #5  0x000000000055c21e in perform_idle_server_maintenance ()
> #6  0x000000000055c619 in server_main_loop ()
> #7  0x000000000055c959 in event_run ()
> #8  0x0000000000445f91 in ap_run_mpm ()
> #9  0x000000000043dd19 in main ()
> 
> 
> Stefan
> 
> Am 23.09.2016 um 16:53 schrieb Yann Ylavic:
>> With the patch this time...
>>
>> On Fri, Sep 23, 2016 at 4:53 PM, Yann Ylavic <ylavic.dev@gmail.com> wrote:
>>> On Fri, Sep 23, 2016 at 3:13 PM, Yann Ylavic <ylavic.dev@gmail.com> wrote:
>>>>
>>>> If you can try another (additive) patch on the faulty server, it would
>>>> be awesome to test the one from [1], not sure it applies correctly
>>>> with my 2.4.x patch though, possibly better with the trunk version
>>>> over [1] (let me know otherwise).
>>>
>>> Not a big deal actually, the attached patch includes both [1] and [2] combined.
>>>
>>> Regards,
>>> Yann.
>>>
>>> [1] https://bz.apache.org/bugzilla/show_bug.cgi?id=53555#c55
>>> [2] https://bz.apache.org/bugzilla/show_bug.cgi?id=57399#c9

Today i had another server giving no answers to any requests. apache
fullstatus did not respond.

[ ~]# netstat -na | grep ":80" | wc -l
29
[ ~]# netstat -na | grep ":443" | wc -l
57

 ps aux|grep -i httpd
root       575  0.0  0.2 161060 40092 ?        Ss   Dec07   1:47
/usr/local/apache2/bin/httpd -k start
wwwrun   17894  3.1  1.6 4891824 278748 ?      Sl   11:52   3:42
/usr/local/apache2/bin/httpd -k start
wwwrun   25574  0.0  0.0 108316  6684 ?        S    06:00   0:00
/usr/local/apache2/bin/httpd -k start
wwwrun   25575  0.0  0.0 108788  6504 ?        S    06:00   0:00
/usr/local/apache2/bin/httpd -k start
wwwrun   25741  2.1  0.6 4778760 110512 ?      Sl   06:00  10:02
/usr/local/apache2/bin/httpd -k start
wwwrun   28094  3.1  1.5 4916324 252428 ?      Sl   06:31  13:59
/usr/local/apache2/bin/httpd -k start
wwwrun   29575  5.2  2.6 5081956 443192 ?      Sl   06:53  22:00
/usr/local/apache2/bin/httpd -k start

an strace only shows:
[pid 28100] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL
<unfinished ...>
[pid 28099] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6192, NULL
<unfinished ...>
[pid 28101] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL
<unfinished ...>
[pid 28099] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 28098] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6187, NULL
<unfinished ...>
[pid 28099] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL
<unfinished ...>
[pid 28098] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 28097] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6206, NULL
<unfinished ...>
[pid 28098] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL
<unfinished ...>
[pid 28097] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 28096] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6200, NULL
<unfinished ...>
[pid 28097] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL
<unfinished ...>
[pid 28096] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 28095] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6168, NULL
<unfinished ...>
[pid 28096] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL
<unfinished ...>
[pid 28095] <... futex resumed> )       = -1 EAGAIN (Resource
temporarily unavailable)
[pid 28094] read(8,  <unfinished ...>
[pid 28095] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL
<unfinished ...>
[pid 28542] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28542] futex(0x173d344,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51873, {1482238254,
882451000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28542] futex(0x173d344,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51875, {1482238255,
883132000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28542] futex(0x173d344,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51877, {1482238256,
883880000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28542] futex(0x173d344,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51879, {1482238257,
884511000}, ffffffff <unfinished ...>
[pid 28146] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid 28146] futex(0x7fc9ed2540a8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28146] futex(0x7fc9ca61a28c,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 4671, {1482238262,
784057000}, ffffffff <unfinished ...>
[pid 28542] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed
out)
[pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28542] futex(0x173d344,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51881, {1482238258,
885329000}, ffffffffq) = -1 ETIMEDOUT (Connection timed out)
[pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0

gdb bt shows this for all httpd childs:
(gdb) bt
#0  0x00007fc9f2b7dadd in read () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x000000000047cd89 in ap_mpm_podx_check ()
#2  0x000000000055b98f in child_main ()
#3  0x000000000055bc2e in make_child ()
#4  0x000000000055c5a0 in perform_idle_server_maintenance ()
#5  0x000000000055c940 in server_main_loop ()
#6  0x000000000055cc80 in event_run ()
#7  0x0000000000445ff1 in ap_run_mpm ()
#8  0x000000000043dd79 in main ()

greets,
Stefan

Mime
View raw message