httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Beckett <paul_beck...@outlook.com>
Subject RE: [users@httpd] Segmentation Fault - too many proxy balancers
Date Sat, 26 Jul 2014 17:37:54 GMT
Just realised I should probably have typed where in gdb after that, which additionally yields:
(gdb) where#0  apr_global_mutex_lock (mutex=0x0) at locks/unix/global_mutex.c:97#1  0x00007ffe08a90e83 in sec_audit_logger (msr=0x7ffdac006328) at msc_logging.c:579#2  0x00007ffe08a966f7 in modsecurity_process_phase_logging (msr=0x7ffdac006328, phase=<value optimized out>) at modsecurity.c:689#3  modsecurity_process_phase (msr=0x7ffdac006328, phase=<value optimized out>) at modsecurity.c:795#4  0x00007ffe08a6b68b in hook_log_transaction (r=0x7ffdac004980) at mod_security2.c:1202#5  0x00000000004343f0 in ap_run_log_transaction (r=0x7ffdac004980) at protocol.c:1788#6  0x000000000044571f in eor_bucket_cleanup (data=<value optimized out>) at eor_bucket.c:35#7  0x00007ffe10040c6e in run_cleanups (pool=0x7ffdac004908) at memory/unix/apr_pools.c:2352#8  apr_pool_destroy (pool=0x7ffdac004908) at memory/unix/apr_pools.c:804#9  0x00007ffe0cb3b9b9 in ssl_io_filter_output (f=0x7ffdb4001240, bb=0x7ffd5c003d98) at ssl_engine_io.c:1659#10 0x00007ffe0cb3c0f5 in ssl_io_filter_coalesce (f=0x7ffdb4001218, bb=0x7ffd5c003d98) at ssl_engine_io.c:1558#11 0x000000000045fa5d in ap_process_request_after_handler (r=0x7ffdac004980) at http_request.c:256#12 0x000000000045d040 in ap_process_http_async_connection (c=0x7ffdb4000cc0) at http_core.c:143#13 ap_process_http_connection (c=0x7ffdb4000cc0) at http_core.c:228#14 0x0000000000454b30 in ap_run_process_connection (c=0x7ffdb4000cc0) at connection.c:41#15 0x00007ffe0eda17e1 in process_socket (thd=0x2047658, dummy=<value optimized out>) at event.c:970#16 worker_thread (thd=0x2047658, dummy=<value optimized out>) at event.c:1815#17 0x00007ffe0f9cf851 in start_thread () from /lib64/libpthread.so.0#18 0x00007ffe0f71d90d in clone () from /lib64/libc.so.6

From: paul_beckett@outlook.com
To: users@httpd.apache.org
Date: Sat, 26 Jul 2014 18:35:03 +0100
Subject: RE: [users@httpd] Segmentation Fault - too many proxy balancers




Jeff,Thanks for the response. I'm not very familiar with the process of creating a backtrace - I've not had to debug apache at this level before (trying to follow info at http://httpd.apache.org/dev/debugging.html#backtrace ). 
Initially I created a backtrace using pstack, but this was about 1 second prior to the segmentation fault, so I guess this is probably not useful (included at end of email just in case)
I then enabled core dumps, and used gdb to analyse it, output below. Not knowing what to expect from the backtrace, is this info what you were expecting when you requested I post the backtrace?
Thanks again,Paul
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)Copyright (C) 2010 Free Software Foundation, Inc.License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>This is free software: you are free to change and redistribute it.There is NO WARRANTY, to the extent permitted by law.  Type "show copying"and "show warranty" for details.This GDB was configured as "x86_64-redhat-linux-gnu".For bug reporting instructions, please see:<http://www.gnu.org/software/gdb/bugs/>...Reading symbols from /usr/local/apache/bin/httpd...done.[New Thread 10172][New Thread 10174][New Thread 10175][New Thread 10180][New Thread 10148][New Thread 10186][New Thread 10184][New Thread 10181][New Thread 10200][New Thread 10209][New Thread 10173][New Thread 10178][New Thread 10155][New Thread 10167][New Thread 10171][New Thread 10208][New Thread 10212][New Thread 10159][New Thread 10182][New Thread 10158][New Thread 10149][New Thread 10185][New Thread 10183][New Thread 10203][New Thread 10160][New Thread 10162][New Thread 10213][New Thread 10164][New Thread 10188][New Thread 10150][New Thread 10187][New Thread 10177][New Thread 10189][New Thread 10168][New Thread 10154][New Thread 10215][New Thread 10165][New Thread 10156][New Thread 10214][New Thread 10163][New Thread 10198][New Thread 10170][New Thread 10190][New Thread 10179][New Thread 10217][New Thread 10216][New Thread 10202][New Thread 10196][New Thread 10191][New Thread 10205][New Thread 10195][New Thread 10192][New Thread 10207][New Thread 10204][New Thread 10251][New Thread 10201][New Thread 10194][New Thread 10199][New Thread 10193][New Thread 10161][New Thread 10219][New Thread 10153][New Thread 10249][New Thread 10166][New Thread 10218][New Thread 10197][New Thread 10222][New Thread 10247][New Thread 10157][New Thread 10220][New Thread 10245][New Thread 10221][New Thread 10224][New Thread 10232][New Thread 10243][New Thread 10235][New Thread 10250][New Thread 10241][New Thread 10230][New Thread 10206][New Thread 10233][New Thread 10237][New Thread 10248][New Thread 10228][New Thread 10239][New Thread 10211][New Thread 10231][New Thread 10252][New Thread 10238][New Thread 10236][New Thread 10226][New Thread 10210][New Thread 10229][New Thread 10240][New Thread 10234][New Thread 10227][New Thread 10223][New Thread 10242][New Thread 10152][New Thread 10225][New Thread 10176][New Thread 10169][New Thread 10246][New Thread 10244]Missing separate debuginfo for /usr/local/apache/lib/libaprutil-1.so.0Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/37/3e4c965981a9edf122d4ae60548613d9ae2336Missing separate debuginfo for /usr/local/apache/lib/libapr-1.so.0Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/8d/960ff682eb3d2312c241d82c9afbedd2577d2bMissing separate debuginfo for /usr/local/apache/lib/libexpat.so.0Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/29/61a04fd4c6f84d8dac7ea2c2e3db1d4a6f3b59Missing separate debuginfo for /usr/local/apache/modules/mod_shib_24.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/aa/ebeac19976d4611374e125ecd112bf1ade2011Missing separate debuginfo for /usr/local/shibsp/lib/libshibsp-lite.so.6Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/f1/b9a9fe4772f4c2750129e7872d3b63f870e83cMissing separate debuginfo for /usr/local/apache/modules/mod_mpm_event.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6d/324996bc4cd28c75ad3e9106c2893491e3efcfMissing separate debuginfo for /usr/local/apache/modules/mod_unixd.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/f9/57d49fd65d7561bf944f507da9d7dad2e513a9Missing separate debuginfo for /usr/local/apache/modules/mod_dir.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/bd/7d85b4dd733009451d18b3414c1fd76d6b8a4dMissing separate debuginfo for /usr/local/apache/modules/mod_log_config.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/5d/620abb41bd1b466156aac66e3a2200714f6f53Missing separate debuginfo for /usr/local/apache/modules/mod_mime.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/eb/b8f8bf64db622502d39c625e3a9d88a0ae1e82Missing separate debuginfo for /usr/local/apache/modules/mod_mime_magic.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6f/f24d8f77831d3206d541e587578af86ab2a519Missing separate debuginfo for /usr/local/apache/modules/mod_authn_core.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/8f/bec33c7e6f4a3cacdece68f32fff862d0f7b30Missing separate debuginfo for /usr/local/apache/modules/mod_authz_host.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9c/ad185c4b505641acc2217c6bcb8532974dd489Missing separate debuginfo for /usr/local/apache/modules/mod_authz_user.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/4b/07ae5e53f6c9d42455589a56cc52dc21e022a9Missing separate debuginfo for /usr/local/apache/modules/mod_authz_core.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/fa/d69123abc0b4e1e6bb7338e7754c5dae94dd70Missing separate debuginfo for /usr/local/apache/modules/mod_access_compat.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a3/2700ac62cd51565e1563c6852c6ff996573576Missing separate debuginfo for /usr/local/apache/modules/mod_reqtimeout.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/d6/9d6cf839eceb72c55235064dee28fc9e0d01b0Missing separate debuginfo for /usr/local/apache/modules/mod_filter.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/17/56caae1845d0adc08fb069208ef3101ae718daMissing separate debuginfo for /usr/local/apache/modules/mod_env.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/4e/36fea0939efac30d11bf5cec681fc7d171ee45Missing separate debuginfo for /usr/local/apache/modules/mod_include.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b6/380ff50a323d5711c065a70f17052d3ba2da29Missing separate debuginfo for /usr/local/apache/modules/mod_headers.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/0e/10b837e15f98e4ecdb7dc785dcc0381f57e8f6Missing separate debuginfo for /usr/local/apache/modules/mod_setenvif.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/dc/08448a6e9c167ceb128981105847af62175e12Missing separate debuginfo for /usr/local/apache/modules/mod_ssl.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/ab/2e79aba60c6bc949754958f36ca9899d7e0c0fMissing separate debuginfo for /usr/local/apache/modules/mod_socache_shmcb.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/6a/a31378be3a2741f0dcc17abc413ca31300cd7dMissing separate debuginfo for /usr/local/apache/modules/mod_rewrite.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/75/1362a102f672d1ded8dd8ea09e597a152b8f3eMissing separate debuginfo for /usr/local/apache/modules/mod_status.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/bd/1f9b60389f1e4ee2ed1edf918ba03b3de1b077Missing separate debuginfo for /usr/local/apache/modules/mod_info.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/c8/6f92dfae31509c068568a4913567db8f5e083cMissing separate debuginfo for /usr/local/apache/modules/mod_cache.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/75/d7a334e208aeb7b07b7b81514b863b40117be1Missing separate debuginfo for /usr/local/apache/modules/mod_cache_disk.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/bf/2989446a71cdffd99631f310ebba25c0f24e0bMissing separate debuginfo for /usr/local/apache/modules/mod_deflate.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/58/cf80ecc07f6ce5eddf682298c9793d7755d753Missing separate debuginfo for /usr/local/apache/modules/mod_slotmem_shm.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/e4/c513d6ecd62be795c6cd32f4928f82a36d7e64Missing separate debuginfo for /usr/local/apache/modules/mod_proxy.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/17/1a1a7350d5d2f111046d77f4bc075e335695d1Missing separate debuginfo for /usr/local/apache/modules/mod_proxy_connect.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9d/9b005ecad7b6c16f5b46363efb5bf5e595a026Missing separate debuginfo for /usr/local/apache/modules/mod_proxy_http.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/98/6860aef9605a6db743ee2ae027dc36e8a77f26Missing separate debuginfo for /usr/local/apache/modules/mod_proxy_balancer.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/e6/62c31160985d77b5019b87ec10ef0b2d07f618Missing separate debuginfo for /usr/local/apache/modules/mod_lbmethod_byrequests.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/58/3071a430fd84abd068fe2aafb8d9074b0e36c1Missing separate debuginfo for /usr/local/apache/modules/mod_unique_id.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/65/a9a5a3b4aeb09caded6e5fac915c3a3533ef53Missing separate debuginfo for /usr/local/apache/modules/mod_security2.soTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/16/38ce80438b7cabfa13da67aaeb976d29a5f9acMissing separate debuginfo forTry: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/08/f634a1d22deff00461d50a7699dacdc97657bfReading symbols from /lib64/libpcre.so.0...(no debugging symbols found)...done.Loaded symbols for /lib64/libpcre.so.0Reading symbols from /usr/local/apache/lib/libaprutil-1.so.0...done.Loaded symbols for /usr/local/apache/lib/libaprutil-1.so.0Reading symbols from /lib64/libexpat.so.1...(no debugging symbols found)...done.Loaded symbols for /lib64/libexpat.so.1Reading symbols from /usr/local/apache/lib/libapr-1.so.0...done.Loaded symbols for /usr/local/apache/lib/libapr-1.so.0Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.Loaded symbols for /lib64/librt.so.1Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.Loaded symbols for /lib64/libcrypt.so.1Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.[Thread debugging using libthread_db enabled]Loaded symbols for /lib64/libpthread.so.0Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.Loaded symbols for /lib64/libc.so.6Reading symbols from /usr/local/apache/lib/libexpat.so.0...done.Loaded symbols for /usr/local/apache/lib/libexpat.so.0Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.Loaded symbols for /lib64/ld-linux-x86-64.so.2Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.Loaded symbols for /lib64/libfreebl3.soReading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.Loaded symbols for /lib64/libdl.so.2Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.Loaded symbols for /lib64/libresolv.so.2Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.Loaded symbols for /lib64/libm.so.6Reading symbols from /usr/local/apache/modules/mod_shib_24.so...done.Loaded symbols for /usr/local/apache/modules/mod_shib_24.soReading symbols from /usr/local/shibsp/lib/libshibsp-lite.so.6...done.Loaded symbols for /usr/local/shibsp/lib/libshibsp-lite.so.6Reading symbols from /usr/lib64/libxmltooling-lite.so.6...(no debugging symbols found)...done.Loaded symbols for /usr/lib64/libxmltooling-lite.so.6Reading symbols from /usr/lib64/libxerces-c-3.1.so...(no debugging symbols found)...done.Loaded symbols for /usr/lib64/libxerces-c-3.1.soReading symbols from /usr/lib64/liblog4shib.so.1...(no debugging symbols found)...done.Loaded symbols for /usr/lib64/liblog4shib.so.1Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.Loaded symbols for /lib64/libnsl.so.1Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.Loaded symbols for /usr/lib64/libstdc++.so.6Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.Loaded symbols for /lib64/libgcc_s.so.1Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.Loaded symbols for /lib64/libnss_files.so.2Reading symbols from /lib64/libnss_vas4.so.2...(no debugging symbols found)...done.Loaded symbols for /lib64/libnss_vas4.so.2Reading symbols from /opt/quest/lib64/libvtsmartcache.so.1...(no debugging symbols found)...done.Loaded symbols for /opt/quest/lib64/libvtsmartcache.so.1Reading symbols from /opt/quest/lib64/libvtutil.so.1...(no debugging symbols found)...done.Loaded symbols for /opt/quest/lib64/libvtutil.so.1Reading symbols from /opt/quest/lib64/libvtcacheipc.so.1...(no debugging symbols found)...done.Loaded symbols for /opt/quest/lib64/libvtcacheipc.so.1Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.Loaded symbols for /lib64/libnss_dns.so.2Reading symbols from /usr/lib64/gconv/UTF-16.so...(no debugging symbols found)...done.Loaded symbols for /usr/lib64/gconv/UTF-16.soReading symbols from /usr/local/apache/modules/mod_mpm_event.so...done.Loaded symbols for /usr/local/apache/modules/mod_mpm_event.soReading symbols from /usr/local/apache/modules/mod_unixd.so...done.Loaded symbols for /usr/local/apache/modules/mod_unixd.soReading symbols from /usr/local/apache/modules/mod_dir.so...done.Loaded symbols for /usr/local/apache/modules/mod_dir.soReading symbols from /usr/local/apache/modules/mod_log_config.so...done.Loaded symbols for /usr/local/apache/modules/mod_log_config.soReading symbols from /usr/local/apache/modules/mod_mime.so...done.Loaded symbols for /usr/local/apache/modules/mod_mime.soReading symbols from /usr/local/apache/modules/mod_mime_magic.so...done.Loaded symbols for /usr/local/apache/modules/mod_mime_magic.soReading symbols from /usr/local/apache/modules/mod_authn_core.so...done.Loaded symbols for /usr/local/apache/modules/mod_authn_core.soReading symbols from /usr/local/apache/modules/mod_authz_host.so...done.Loaded symbols for /usr/local/apache/modules/mod_authz_host.soReading symbols from /usr/local/apache/modules/mod_authz_user.so...done.Loaded symbols for /usr/local/apache/modules/mod_authz_user.soReading symbols from /usr/local/apache/modules/mod_authz_core.so...done.Loaded symbols for /usr/local/apache/modules/mod_authz_core.soReading symbols from /usr/local/apache/modules/mod_access_compat.so...done.Loaded symbols for /usr/local/apache/modules/mod_access_compat.soReading symbols from /usr/local/apache/modules/mod_reqtimeout.so...done.Loaded symbols for /usr/local/apache/modules/mod_reqtimeout.soReading symbols from /usr/local/apache/modules/mod_filter.so...done.Loaded symbols for /usr/local/apache/modules/mod_filter.soReading symbols from /usr/local/apache/modules/mod_env.so...done.Loaded symbols for /usr/local/apache/modules/mod_env.soReading symbols from /usr/local/apache/modules/mod_include.so...done.Loaded symbols for /usr/local/apache/modules/mod_include.soReading symbols from /usr/local/apache/modules/mod_headers.so...done.Loaded symbols for /usr/local/apache/modules/mod_headers.soReading symbols from /usr/local/apache/modules/mod_setenvif.so...done.Loaded symbols for /usr/local/apache/modules/mod_setenvif.soReading symbols from /usr/local/apache/modules/mod_ssl.so...done.Loaded symbols for /usr/local/apache/modules/mod_ssl.soReading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.Loaded symbols for /usr/lib64/libssl.so.10Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.Loaded symbols for /usr/lib64/libcrypto.so.10Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.Loaded symbols for /lib64/libz.so.1Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.Loaded symbols for /lib64/libgssapi_krb5.so.2Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.Loaded symbols for /lib64/libkrb5.so.3Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.Loaded symbols for /lib64/libcom_err.so.2Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.Loaded symbols for /lib64/libk5crypto.so.3Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.Loaded symbols for /lib64/libkrb5support.so.0Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.Loaded symbols for /lib64/libkeyutils.so.1Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.Loaded symbols for /lib64/libselinux.so.1Reading symbols from /usr/local/apache/modules/mod_socache_shmcb.so...done.Loaded symbols for /usr/local/apache/modules/mod_socache_shmcb.soReading symbols from /usr/local/apache/modules/mod_rewrite.so...done.Loaded symbols for /usr/local/apache/modules/mod_rewrite.soReading symbols from /usr/local/apache/modules/mod_status.so...done.Loaded symbols for /usr/local/apache/modules/mod_status.soReading symbols from /usr/local/apache/modules/mod_info.so...done.Loaded symbols for /usr/local/apache/modules/mod_info.soReading symbols from /usr/local/apache/modules/mod_cache.so...done.Loaded symbols for /usr/local/apache/modules/mod_cache.soReading symbols from /usr/local/apache/modules/mod_cache_disk.so...done.Loaded symbols for /usr/local/apache/modules/mod_cache_disk.soReading symbols from /usr/local/apache/modules/mod_deflate.so...done.Loaded symbols for /usr/local/apache/modules/mod_deflate.soReading symbols from /usr/local/apache/modules/mod_slotmem_shm.so...done.Loaded symbols for /usr/local/apache/modules/mod_slotmem_shm.soReading symbols from /usr/local/apache/modules/mod_proxy.so...done.Loaded symbols for /usr/local/apache/modules/mod_proxy.soReading symbols from /usr/local/apache/modules/mod_proxy_connect.so...done.Loaded symbols for /usr/local/apache/modules/mod_proxy_connect.soReading symbols from /usr/local/apache/modules/mod_proxy_http.so...done.Loaded symbols for /usr/local/apache/modules/mod_proxy_http.soReading symbols from /usr/local/apache/modules/mod_proxy_balancer.so...done.Loaded symbols for /usr/local/apache/modules/mod_proxy_balancer.soReading symbols from /usr/local/apache/modules/mod_lbmethod_byrequests.so...done.Loaded symbols for /usr/local/apache/modules/mod_lbmethod_byrequests.soReading symbols from /usr/lib64/libxml2.so...(no debugging symbols found)...done.Loaded symbols for /usr/lib64/libxml2.soReading symbols from /usr/lib64/liblua-5.1.so...(no debugging symbols found)...done.Loaded symbols for /usr/lib64/liblua-5.1.soReading symbols from /usr/local/apache/modules/mod_unique_id.so...done.Loaded symbols for /usr/local/apache/modules/mod_unique_id.soReading symbols from /usr/local/apache/modules/mod_security2.so...done.Loaded symbols for /usr/local/apache/modules/mod_security2.soCore was generated by `/usr/local/apache/bin/httpd -k start -D UEA_TEST_SERVER'.Program terminated with signal 11, Segmentation fault.#0  apr_global_mutex_lock (mutex=0x0) at locks/unix/global_mutex.c:9797          if (mutex->thread_mutex) {Missing separate debuginfos, use: debuginfo-install expat-2.0.1-11.el6_2.x86_64 glibc-2.12-1.107.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6_4.2.x86_64 libcom_err-1.41.12-14.el6.x86_64 libgcc-4.4.7-3.el6.x86_64 liblog4shib1-1.0.6-1.1.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 libstdc++-4.4.7-3.el6.x86_64 libxerces-c-3_1-3.1.1-2.4.el6.x86_64 libxml2-devel-2.7.6-12.el6_4.1.x86_64 libxmltooling6-1.5.3-1.1.el6.x86_64 lua-5.1.4-4.1.el6.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 openssl-1.0.0-27.el6_4.2.x86_64 pcre-7.8-6.el6.x86_64 zlib-1.2.3-29.el6.x86_64



Thread 104 (Thread 0x7fe663364700 (LWP 64967)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66531db79 in xmltooling::ReloadableXMLFile::reload_fn(void*) () from /usr/lib64/libxmltooling-lite.so.6#2  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#3  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 103 (Thread 0x7fe662963700 (LWP 64968)):#0  0x00007fe66ca767bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe665326bc4 in xmltooling::CondWaitImpl::timedwait(xmltooling::Mutex*, int) () from /usr/lib64/libxmltooling-lite.so.6#2  0x00007fe66562b26d in (anonymous namespace)::SSCache::cleanup_fn(void*) () from /usr/local/shibsp/lib/libshibsp-lite.so.6#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 102 (Thread 0x7fe661561700 (LWP 64970)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 101 (Thread 0x7fe660b60700 (LWP 64971)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 100 (Thread 0x7fe653fff700 (LWP 64972)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 99 (Thread 0x7fe6535fe700 (LWP 64973)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 98 (Thread 0x7fe652bfd700 (LWP 64974)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 97 (Thread 0x7fe6521fc700 (LWP 64975)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 96 (Thread 0x7fe6517fb700 (LWP 64976)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 95 (Thread 0x7fe650dfa700 (LWP 64977)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 94 (Thread 0x7fe6503f9700 (LWP 64978)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 93 (Thread 0x7fe64f9f8700 (LWP 64979)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 92 (Thread 0x7fe64eff7700 (LWP 64980)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 91 (Thread 0x7fe64e5f6700 (LWP 64981)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 90 (Thread 0x7fe64dbf5700 (LWP 64982)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 89 (Thread 0x7fe64d1f4700 (LWP 64983)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 88 (Thread 0x7fe64c7f3700 (LWP 64984)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 87 (Thread 0x7fe64bdf2700 (LWP 64985)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 86 (Thread 0x7fe64b3f1700 (LWP 64986)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 85 (Thread 0x7fe64a9f0700 (LWP 64987)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 84 (Thread 0x7fe649fef700 (LWP 64988)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 83 (Thread 0x7fe6495ee700 (LWP 64989)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 82 (Thread 0x7fe648bed700 (LWP 64990)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 81 (Thread 0x7fe6481ec700 (LWP 64991)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 80 (Thread 0x7fe6477eb700 (LWP 64992)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 79 (Thread 0x7fe646dea700 (LWP 64993)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 78 (Thread 0x7fe6463e9700 (LWP 64994)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 77 (Thread 0x7fe6459e8700 (LWP 64995)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 76 (Thread 0x7fe644fe7700 (LWP 64996)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 75 (Thread 0x7fe6445e6700 (LWP 64997)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 74 (Thread 0x7fe643be5700 (LWP 64998)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 73 (Thread 0x7fe6431e4700 (LWP 64999)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 72 (Thread 0x7fe6427e3700 (LWP 65000)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 71 (Thread 0x7fe641de2700 (LWP 65001)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 70 (Thread 0x7fe6413e1700 (LWP 65002)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 69 (Thread 0x7fe6409e0700 (LWP 65003)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 68 (Thread 0x7fe63ffdf700 (LWP 65004)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 67 (Thread 0x7fe63f5de700 (LWP 65005)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 66 (Thread 0x7fe63ebdd700 (LWP 65006)):#0  0x00007fe66c7b7253 in poll () from /lib64/libc.so.6#1  0x00007fe66d0ed367 in apr_wait_for_io_or_timeout () from /usr/local/apache/lib/libapr-1.so.0#2  0x00007fe66d0e780a in apr_socket_recv () from /usr/local/apache/lib/libapr-1.so.0#3  0x00007fe66d52c255 in socket_bucket_read () from /usr/local/apache/lib/libaprutil-1.so.0#4  0x000000000044689e in ap_core_input_filter ()#5  0x00007fe66a819e14 in reqtimeout_filter () from /usr/local/apache/modules/mod_reqtimeout.so#6  0x00007fe669be01b9 in bio_filter_in_read () from /usr/local/apache/modules/mod_ssl.so#7  0x00007fe669681419 in BIO_read () from /usr/lib64/libcrypto.so.10#8  0x00007fe669992bb8 in ssl3_read_n () from /usr/lib64/libssl.so.10#9  0x00007fe66999327d in ssl3_read_bytes () from /usr/lib64/libssl.so.10#10 0x00007fe66998f700 in ?? () from /usr/lib64/libssl.so.10#11 0x00007fe669bdf3fe in ssl_io_input_read () from /usr/local/apache/modules/mod_ssl.so#12 0x00007fe669bdfac8 in ssl_io_filter_input () from /usr/local/apache/modules/mod_ssl.so#13 0x0000000000435ec2 in ap_rgetline_core ()#14 0x0000000000436cee in ap_read_request ()#15 0x000000000045cfc9 in ap_process_http_connection ()#16 0x0000000000454b30 in ap_run_process_connection ()#17 0x00007fe66be447e1 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#18 0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#19 0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 65 (Thread 0x7fe63e1dc700 (LWP 65007)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 64 (Thread 0x7fe63d7db700 (LWP 65008)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 63 (Thread 0x7fe63cdda700 (LWP 65009)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 62 (Thread 0x7fe63c3d9700 (LWP 65010)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 61 (Thread 0x7fe63b9d8700 (LWP 65011)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 60 (Thread 0x7fe63afd7700 (LWP 65012)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 59 (Thread 0x7fe63a5d6700 (LWP 65013)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 58 (Thread 0x7fe639bd5700 (LWP 65014)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 57 (Thread 0x7fe6391d4700 (LWP 65015)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 56 (Thread 0x7fe6387d3700 (LWP 65016)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 55 (Thread 0x7fe637dd2700 (LWP 65017)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 54 (Thread 0x7fe6373d1700 (LWP 65018)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 53 (Thread 0x7fe6369d0700 (LWP 65019)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 52 (Thread 0x7fe635fcf700 (LWP 65020)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 51 (Thread 0x7fe6355ce700 (LWP 65021)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 50 (Thread 0x7fe634bcd700 (LWP 65022)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 49 (Thread 0x7fe6341cc700 (LWP 65023)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 48 (Thread 0x7fe6337cb700 (LWP 65024)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 47 (Thread 0x7fe632dca700 (LWP 65025)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 46 (Thread 0x7fe6323c9700 (LWP 65026)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 45 (Thread 0x7fe6319c8700 (LWP 65027)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 44 (Thread 0x7fe630fc7700 (LWP 65028)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 43 (Thread 0x7fe6305c6700 (LWP 65029)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 42 (Thread 0x7fe62fbc5700 (LWP 65030)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 41 (Thread 0x7fe62f1c4700 (LWP 65031)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 40 (Thread 0x7fe62e7c3700 (LWP 65032)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 39 (Thread 0x7fe62ddc2700 (LWP 65033)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 38 (Thread 0x7fe62d3c1700 (LWP 65034)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 37 (Thread 0x7fe62c9c0700 (LWP 65035)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 36 (Thread 0x7fe62bfbf700 (LWP 65036)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 35 (Thread 0x7fe62b5be700 (LWP 65037)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 34 (Thread 0x7fe62abbd700 (LWP 65038)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 33 (Thread 0x7fe62a1bc700 (LWP 65039)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 32 (Thread 0x7fe6297bb700 (LWP 65040)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 31 (Thread 0x7fe628dba700 (LWP 65041)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 30 (Thread 0x7fe6283b9700 (LWP 65042)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 29 (Thread 0x7fe6279b8700 (LWP 65043)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 28 (Thread 0x7fe626fb7700 (LWP 65044)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 27 (Thread 0x7fe6265b6700 (LWP 65045)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 26 (Thread 0x7fe625bb5700 (LWP 65046)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 25 (Thread 0x7fe6251b4700 (LWP 65047)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 24 (Thread 0x7fe6247b3700 (LWP 65048)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 23 (Thread 0x7fe623db2700 (LWP 65049)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 22 (Thread 0x7fe6233b1700 (LWP 65050)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 21 (Thread 0x7fe6229b0700 (LWP 65051)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 20 (Thread 0x7fe621faf700 (LWP 65052)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 19 (Thread 0x7fe6215ae700 (LWP 65053)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 18 (Thread 0x7fe620bad700 (LWP 65054)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 17 (Thread 0x7fe6201ac700 (LWP 65055)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 16 (Thread 0x7fe61f7ab700 (LWP 65056)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 15 (Thread 0x7fe61edaa700 (LWP 65057)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 14 (Thread 0x7fe61e3a9700 (LWP 65058)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 13 (Thread 0x7fe61d9a8700 (LWP 65059)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 12 (Thread 0x7fe61cfa7700 (LWP 65060)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 11 (Thread 0x7fe61c5a6700 (LWP 65061)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 10 (Thread 0x7fe61bba5700 (LWP 65062)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 9 (Thread 0x7fe61b1a4700 (LWP 65063)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 8 (Thread 0x7fe61a7a3700 (LWP 65064)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 7 (Thread 0x7fe619da2700 (LWP 65065)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 6 (Thread 0x7fe6193a1700 (LWP 65066)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 5 (Thread 0x7fe6189a0700 (LWP 65067)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 4 (Thread 0x7fe617f9f700 (LWP 65068)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 3 (Thread 0x7fe61759e700 (LWP 65069)):#0  0x00007fe66ca7643c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1  0x00007fe66be46e95 in ap_queue_pop_something () from /usr/local/apache/modules/mod_mpm_event.so#2  0x00007fe66be44228 in worker_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 2 (Thread 0x7fe616b9d700 (LWP 65070)):#0  0x00007fe66c7c0f03 in epoll_wait () from /lib64/libc.so.6#1  0x00007fe66d0ea271 in impl_pollset_poll () from /usr/local/apache/lib/libapr-1.so.0#2  0x00007fe66be4333b in listener_thread () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66ca72851 in start_thread () from /lib64/libpthread.so.0#4  0x00007fe66c7c090d in clone () from /lib64/libc.so.6Thread 1 (Thread 0x7fe66db80700 (LWP 64966)):#0  0x00007fe66ca7954d in read () from /lib64/libpthread.so.0#1  0x00000000004577b7 in ap_mpm_podx_check ()#2  0x00007fe66be44eec in child_main () from /usr/local/apache/modules/mod_mpm_event.so#3  0x00007fe66be45978 in make_child () from /usr/local/apache/modules/mod_mpm_event.so#4  0x00007fe66be46554 in event_run () from /usr/local/apache/modules/mod_mpm_event.so#5  0x000000000043089e in ap_run_mpm ()#6  0x000000000042ab84 in main ()

Date: Sat, 26 Jul 2014 08:28:07 -0400
From: trawick@gmail.com
To: users@httpd.apache.org
Subject: Re: [users@httpd] Segmentation Fault - too many proxy balancers

On Fri, Jul 25, 2014 at 2:00 PM, Paul Beckett <paul_beckett@outlook.com> wrote:




My apache server has started segmentation faulting all the time (seems to log a segmentation fault every few requests to the apache error log):
[Fri Jul 25 06:25:42.046752
2014] [core:notice] [pid 11226:tid 140006078953216] AH00052: child pid 11715
exit signal Segmentation fault (11)
Can you post a backtrace from the crash? 

This appears to be due to the number of proxy balancers I have configured (problem isn't related to any one specific proxy balancer, adding / removing any of the proxy balancers causes the problem to appear/disappear). I'm using Apache HTTPD as a reverse proxy for a lot of load-balanced (by apache httpd) application servers. My googling so far hasn't found any specific limit on the number of proxy, or how I can increase this. 

I am running Apache HTTPD 2.4.9, built from source on RHEL6.
I would be very grateful if anyone can shed more light on this, and assuming I'm right about a limit: point my in the right direction as to how I can increase this.

Thanks,
Paul 		 	   		  


-- 
Born in Roswell... married an alien...
http://emptyhammock.com/
http://edjective.org/


 		 	   		   		 	   		  
Mime
View raw message