Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C7544200C5A for ; Tue, 18 Apr 2017 15:06:27 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id C5CEB160BA1; Tue, 18 Apr 2017 13:06:27 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id BEC0B160B90 for ; Tue, 18 Apr 2017 15:06:26 +0200 (CEST) Received: (qmail 18477 invoked by uid 500); 18 Apr 2017 13:06:20 -0000 Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm Precedence: bulk Reply-To: dev@httpd.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@httpd.apache.org Received: (qmail 18466 invoked by uid 99); 18 Apr 2017 13:06:20 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Apr 2017 13:06:20 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E97001809FC for ; Tue, 18 Apr 2017 13:06:19 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1 X-Spam-Level: * X-Spam-Status: No, score=1 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id mFWQwR1yA9jz for ; Tue, 18 Apr 2017 13:06:17 +0000 (UTC) Received: from cloud1-vm154.de-nserver.de (cloud1-vm154.de-nserver.de [178.250.10.56]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 6BF175FDF6 for ; Tue, 18 Apr 2017 13:06:16 +0000 (UTC) Received: (qmail 5189 invoked from network); 18 Apr 2017 15:06:15 +0200 X-Fcrdns: No Received: from phoffice.de-nserver.de (HELO [10.11.11.165]) (185.39.223.5) (smtp-auth username hostmaster@profihost.com, mechanism plain) by cloud1-vm154.de-nserver.de (qpsmtpd/0.92) with (ECDHE-RSA-AES256-SHA encrypted) ESMTPSA; Tue, 18 Apr 2017 15:06:15 +0200 Subject: Re: hanging apache httpd processes To: dev@httpd.apache.org References: <40de5e44-5cb2-6c78-35dc-1baa5b478a79@profihost.ag> <0d92da3c-e9c9-1b47-3516-5c55f20615e7@profihost.ag> <30BF83B8-A4B6-4509-A3C9-A56BCDBDB7DA@greenbytes.de> From: Stefan Priebe - Profihost AG Message-ID: <769ad50c-623c-2f99-5497-5988eebdc1cb@profihost.ag> Date: Tue, 18 Apr 2017 15:06:15 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <30BF83B8-A4B6-4509-A3C9-A56BCDBDB7DA@greenbytes.de> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-User-Auth: Auth by hostmaster@profihost.com through 185.39.223.5 archived-at: Tue, 18 Apr 2017 13:06:28 -0000 Am 18.04.2017 um 15:03 schrieb Stefan Eissing: > Stefan, > > that is a 1.10.0, right? That was the first version without nested locking and I fixed 2 possible dead locks in 1.10.1. > > I am about to release a 1.10.2 with added conformity checks and a fix for client omitting EOF flags. Could you give that one a try? Yes sure where is 1.10.2? Stefan > > -Stefan > >> Am 18.04.2017 um 14:57 schrieb Stefan Priebe - Profihost AG : >> >> Hi, >> >> i saw that all of them are still serving one h2 connection. >> >> server-status: >> 0-3 32375 42/64/181 G 30.09 1020776 214 1285.1 2.40 5.81 >> h081217236127.dyn.cm.kabsi.at h2 XXXX:443 GET >> /wp-content/uploads/0000Bloglr212222-8003asd.jpg HTTP/2.0 >> >> And they all have Mode of operation => G >> >> thread apply all bt full shows: >> #0 0x00007f8e80b687fc in __lll_lock_wait () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f8e80b6b6f8 in _L_cond_lock_886 () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #2 0x00007f8e80b6b464 in __pthread_mutex_cond_lock () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #3 0x00007f8e80b6650a in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #4 0x00007f8e80d92322 in apr_thread_cond_timedwait >> (cond=0x7f8d7846bc70, mutex=0x7f8d7846bc20, timeout=60000000) at >> locks/unix/thread_cond.c:89 >> rv = >> then = >> abstime = {tv_sec = 1491843095, tv_nsec = 700198000} >> #5 0x000055e62c0951c5 in r_wait_space (premain=, >> pbl=0x7f8de853da60, block=APR_BLOCK_READ, beam=0x7f8d7846bb08) at >> h2_bucket_beam.c:337 >> status = >> #6 append_bucket (pbl=0x7f8de853da60, block=APR_BLOCK_READ, >> b=0x7f8d7846dd38, beam=0x7f8d7846bb08) at h2_bucket_beam.c:776 >> data = 0x7f8de853dac8 "\240\037" >> len = 0 >> space_left = 0 >> status = >> #7 h2_beam_send (beam=0x7f8d7846bb08, sender_bb=0x7f8d7864ba90, >> block=APR_BLOCK_READ) at h2_bucket_beam.c:906 >> force_report = 1 >> b = 0x7f8d7846dd38 >> status = 0 >> bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250 >> , leave_ctx = 0x0} >> #8 0x000055e62c08f3aa in send_out (task=0x7f8d7846f740, >> bb=0x7f8d7864ba90, block=1) at h2_task.c:100 >> written = 8096 >> left = 140245585033024 >> status = 1 >> #9 0x000055e62c08f976 in slave_out (task=0x7f8d7846f740, >> f=0x7f8d7846bdd8, bb=0x7f8d7864ba90) at h2_task.c:177 >> status = -512 >> flush = 0 >> blocking = 1 >> #10 0x000055e62c08fbfd in h2_filter_slave_output (filter=0x7f8d7846bdd8, >> brigade=0x7f8d7864ba90) at h2_task.c:370 >> task = 0x7f8d7846f740 >> status = >> #11 0x000055e62bffc01c in ap_content_length_filter (f=0x7f8d78472ca8, >> b=0x7f8d7864ba90) at protocol.c:1713 >> r = 0x7f8d78471750 >> ctx = 0x7f8d7864bc90 >> e = 0x7f8d7864ba98 >> eblock = (unknown: 2159446012) >> #12 0x000055e62c04853d in deflate_out_filter (f=0x7f8d7864b650, >> bb=0x7f8d7864b8f8) at mod_deflate.c:961 >> rv = -512 >> r = 0x7f8d78471750 >> ctx = 0x7f8d7864b9a8 >> len = 8096 >> blen = 1157663 >> data = 0x7f8d7e40b000 "/*! jQuery v2.1.4 | (c) 2005, 2015 jQuery >> Foundation, Inc. | jquery.org/license >> */\n!function(a,b){\"object\"==typeof module&&\"object\"==typeof >> module.exports?module.exports=a.document?b(a,!0):function(a)"... >> c = 0x55e62c7a7498 >> #13 0x000055e62c044b1d in filter_harness (f=0x7f8d7846bc28, bb=0x80) at >> mod_filter.c:323 >> ret = -512 >> cachecontrol = 0xfffffffffffffe00 > at address 0xfffffffffffffe00> >> filter = 0xe >> #14 0x000055e62c002242 in default_handler (r=0x7f8d78471750) at core.c:4746 >> c = 0x7f8d7846b720 >> bb = 0x7f8d7864b8f8 >> e = 0xfffffffffffffe00 >> d = 0x7f8d7864dfa0 >> fd = 0x7f8d7864b7b8 >> bld_content_md5 = 2019866872 >> #15 0x000055e62c014130 in ap_run_handler (r=r@entry=0x7f8d78471750) at >> config.c:170 >> pHook = >> n = 11 >> rv = -1 >> #16 0x000055e62c014679 in ap_invoke_handler (r=0x7f8d78471750) at >> config.c:434 >> handler = >> p = >> result = >> old_handler = 0x0 >> ignore = >> #17 0x000055e62c04dcb2 in ap_process_async_request (r=0x7f8d78471750) at >> http_request.c:436 >> c = 0x7f8d7846b720 >> access_status = 0 >> #18 0x000055e62c04de50 in ap_process_request (r=0x7f8d78471750) at >> http_request.c:471 >> bb = 0x7f8d7846bd80 >> c = 0x7f8d7846b720 >> rv = -512 >> #19 0x000055e62c08ec32 in h2_task_process_request (c=, >> task=) at h2_task.c:666 >> req = 0x7f8d78471750 >> cs = 0x7f8d7846bd80 >> r = 0x7f8d78471750 >> #20 h2_task_process_conn (c=0x7f8d7846bc28) at h2_task.c:713 >> ctx = 0x7f8d7846bd80 >> #21 0x000055e62c01e130 in ap_run_process_connection (c=0x7f8d7846b720) >> at connection.c:42 >> pHook = >> n = 0 >> rv = -1 >> #22 0x000055e62c09012b in h2_task_do (task=0x7f8d7846f740, >> thread=0x55e62cb0ce70, worker_id=0) at h2_task.c:623 >> c = 0x7f8d7846b720 >> #23 0x000055e62c093619 in slot_run (thread=0x55e62cb0ce70, >> wctx=0x55e62c8b9268) at h2_workers.c:217 >> slot = 0x55e62c8b9268 >> #24 0x00007f8e80b620a4 in start_thread () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #25 0x00007f8e8089762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> No symbol table info available. >> >> Thread 3 (Thread 0x7f8da678c700 (LWP 28251)): >> #0 0x00007f8e80b687fc in __lll_lock_wait () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f8e80b644b9 in _L_lock_909 () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #2 0x00007f8e80b642e0 in pthread_mutex_lock () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #3 0x000055e62c0942a8 in enter_yellow (pbl=0x7f8da678a8d0, >> beam=0x7f8d7846bb08) at h2_bucket_beam.c:217 >> ctx = >> enter = >> #4 h2_beam_emitted (proxy=0x7f8cd435bd38, beam=0x7f8d7846bb08) at >> h2_bucket_beam.c:352 >> bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250 >> , leave_ctx = 0x7f8c9806b3ed} >> b = >> next = >> #5 beam_bucket_destroy (data=0x7f8cd435bd38) at h2_bucket_beam.c:106 >> data = 0x7f8cd435bd38 >> d = 0x7f8cd435bd38 >> #6 0x00007f8e80fb584c in apr_brigade_cleanup (data=0x7f8d1002d220) at >> buckets/apr_brigade.c:44 >> b = 0x7f8d1002d220 >> e = 0x7f8cd435bab8 >> #7 0x000055e62c094be4 in h2_beam_leave (beam=0x7f8d7846bb08) at >> h2_bucket_beam.c:701 >> bl = {mutex = 0x7f8d7846bc20, leave = 0x55e62c094250 >> , leave_ctx = 0x4} >> #8 0x000055e62c08a7e2 in h2_stream_rst (stream=0x7f8d1002cc00, >> error_code=8) at h2_stream.c:608 >> No locals. >> #9 0x000055e62c084870 in on_stream_close_cb (ngh2=, >> stream_id=, error_code=8, userp=0x7f8d10135620) at >> h2_session.c:266 >> #10 0x00007f8e81666c22 in ?? () from >> /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 >> No symbol table info available. >> #11 0x00007f8e8166823d in ?? () from >> /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 >> No symbol table info available. >> #12 0x00007f8e8166c7cb in nghttp2_session_mem_recv () from >> /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 >> No symbol table info available. >> #13 0x000055e62c079695 in recv_RAW_DATA (c=, >> cin=0x7f8d10135cb0, block=APR_NONBLOCK_READ, b=0x7f8ca80e4a88) at >> h2_filter.c:59 >> status = 0 >> len = 13 >> data = 0x7f8c9806b3e0 "" >> session = 0x7f8d10135620 >> n = >> #14 recv_RAW_brigade (c=, cin=0x7f8d10135cb0, >> bb=0x7f8d10135e18, block=APR_NONBLOCK_READ) at h2_filter.c:100 >> status = 0 >> b = 0x7f8ca80e4a88 >> consumed = >> #15 0x000055e62c07a29c in h2_filter_core_input (f=0x7f8c9806d738, >> brigade=0x7f8d10135ce0, mode=AP_MODE_READBYTES, block=APR_NONBLOCK_READ, >> readbytes=0) at h2_filter.c:182 >> cin = 0x7f8d10135cb0 >> saved_timeout = -1 >> #16 0x000055e62c084b82 in session_read (block=, >> readlen=491520, session=0x7f8d10135620) at h2_session.c:1481 >> status = >> rstatus = 0 >> c = 0x7f8d7808e858 >> read_start = 1919 >> #17 h2_session_read (session=0x7f8d10135620, block=) at >> h2_session.c:1538 >> No locals. >> #18 0x000055e62c08857f in h2_session_process (session=0x7f8d10135620, >> async=128) at h2_session.c:2126 >> c = 0x7f8d7808e858 >> mpm_state = 1 >> trace = 0 >> #19 0x000055e62c0789ce in h2_conn_run (ctx=, >> c=0x7f8d7808e858) at h2_conn.c:203 >> status = >> mpm_state = 0 >> session = 0x7f8d10135620 >> #20 0x000055e62c07bb21 in h2_h2_process_conn (c=0x7f8d7808e858) at >> h2_h2.c:659 >> ctx = 0x7f8c9806d668 >> #21 0x000055e62c01e130 in ap_run_process_connection (c=0x7f8d7808e858) >> at connection.c:42 >> pHook = >> n = 2 >> rv = -1 >> #22 0x000055e62c0b8830 in process_socket (my_thread_num=, >> my_child_num=, cs=0x7f8d7808e7c8, sock=, >> p=, thd=) at event.c:1134 >> c = 0x55e62c0e3748 >> sbh = 0x7f8d7808e7b0 >> #23 worker_thread (thd=0x7f8d7846bc28, dummy=0x80) at event.c:2137 >> process_slot = 5 >> thread_slot = 121 >> csd = 0x7f8d7808e5c0 >> cs = 0x0 >> ptrans = 0x7f8d7808e538 >> is_idle = 2013849544 >> te = 0x0 >> #24 0x00007f8e80b620a4 in start_thread () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #25 0x00007f8e8089762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> No symbol table info available. >> >> Thread 2 (Thread 0x7f8d7ef3d700 (LWP 28405)): >> #0 0x00007f8e80897c03 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 >> No symbol table info available. >> #1 0x00007f8e80d99a93 in impl_pollset_poll (pollset=0x55e62c8459f0, >> timeout=, num=0x7f8d7ef3ce6c, descriptors=0x7f8d7ef3ce88) >> at poll/unix/epoll.c:266 >> ret = >> i = >> j = >> rv = 0 >> fdptr = >> #2 0x000055e62c0ba297 in listener_thread (thd=0x2c, dummy=0x0) at >> event.c:1738 >> workers_were_busy = 0 >> ep = 0x54d70667c9d88 >> te = 0x0 >> rc = 4 >> tpool = 0x55e62c328eb8 >> csd = 0x7f8d78006ce0 >> ptrans = 0x7f8d78006c58 >> lr = 0x54d70667c9d83 >> have_idle_worker = 0 >> out_pfd = 0x7f8de0002918 >> num = 0 >> timeout_interval = 140247325214952 >> pt = 0x1 >> closed = 1 >> #3 0x00007f8e80b620a4 in start_thread () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #4 0x00007f8e8089762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> No symbol table info available. >> >> Thread 1 (Thread 0x7f8e8215a780 (LWP 27400)): >> #0 0x00007f8e80b634db in pthread_join () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> No symbol table info available. >> #1 0x00007f8e80d9e4ab in apr_thread_join (retval=0x7ffda735e794, >> thd=0x7f8de00dbb48) at threadproc/unix/thread.c:217 >> stat = >> thread_stat = 0x1 >> #2 0x000055e62bfedeb3 in join_workers (listener=0x7f8de00dbb48, >> threads=0x55e62cb2f960) at event.c:2376 >> thread_rv = 21990 >> #3 0x000055e62bfee25a in child_main (child_num_arg=5, >> child_bucket=749926752) at event.c:2574 >> threads = 0x55e62cb2f960 >> ts = 0x55e62c844430 >> thread_attr = 0x55e62c844450 >> start_thread_id = 0x55e62c8444b0 >> #4 0x000055e62c0b8fe0 in make_child (s=0x7f8d7ef3d9d0, slot=5, >> bucket=0) at event.c:2646 >> No locals. >> #5 0x000055e62c0b950c in server_main_loop (num_buckets=, >> remaining_children_to_start=) at event.c:2916 >> ps = 0xfffffffffffffe00 >> status = 0 >> pid = {pid = 21894, in = 0x7f8e80d89d76 , out = >> 0x0, err = 0xc} >> child_slot = 5 >> exitwhy = APR_PROC_EXIT >> processed_status = 0 >> #6 event_run (_pconf=0x7f8d7ef3d9d0, plog=0x5, s=0xc8) at event.c:3035 >> remaining_children_to_start = 0 >> #7 0x000055e62bff642e in ap_run_mpm (pconf=0x55e62c74f138, >> plog=0x55e62c790bb8, s=0x55e62c7805a8) at mpm_common.c:94 >> pHook = >> n = 0 >> rv = -1 >> #8 0x000055e62bfeed15 in main (argc=2, argv=0x7ffda735eb98) at main.c:783 >> c = 68 'D' >> error = 0xfffffffffffffe00 > address 0xfffffffffffffe00> >> process = 0x55e62c74d218 >> pconf = 0x55e62c74f138 >> plog = 0x55e62c790bb8 >> ptemp = 0x55e62c7a8d98 >> pcommands = 0x55e62c771248 >> opt = 0x55e62c771338 >> mod = 0x55e62c3235d0 >> opt_arg = 0x7ffda735eef1 "FOREGROUND" >> signal_server = 0xfffffffffffffe00 >> >> Greets, >> Stefan >> >> >> Am 18.04.2017 um 14:47 schrieb Eric Covener: >>> On Tue, Apr 18, 2017 at 8:43 AM, Stefan Priebe - Profihost AG >>> wrote: >>>> bt of such a process shows: >>>> (gdb) bt >>>> #0 0x00007f5df74f64db in pthread_join () from >>>> /lib/x86_64-linux-gnu/libpthread.so.0 >>> >>> Need to see the other threads in the process, this one is just waiting >>> for the others to complete. >>> >>> "thread apply all bt full" >>> >>> >