Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Received-SPF: pass (athena.apache.org: domain of vinay@vys.in designates
 209.85.128.185 as permitted sender)
Message-ID: <e66331100809190347v7316308t30c9988eec279635@mail.gmail.com>
Date: Fri, 19 Sep 2008 16:17:46 +0530
From: "Vinay Y S" <vinay@vys.in>
To: dev@httpd.apache.org
Subject: Understanding apache core traces
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hi,
I've got a apache core file due to a crash from a 3rdparty module. But
the first thread shown in "thread apply all bt full" is this:

Core trace extracted from /usr/local/httpd/service/httpd/core.20252
follows:
Using host libthread_db library "/lib/i686/nosegneg/libthread_db.so.1".

warning: Can't read pathname for load map: Input/output error.

warning: Lowest section in /usr/lib/libicudata.so.36 is .gnu.hash at
000000b4
Core was generated by `httpd.worker -DNO_DETACH -f
/usr/local/httpd/config/httpd.conf'.
Program terminated with signal 11, Segmentation fault.
#0  0xbfffe402 in __kernel_vsyscall ()
#0  0xbfffe402 in __kernel_vsyscall ()
#1  0xb7d45ebb in read () from /lib/i686/nosegneg/libpthread.so.0
#2  0x8003c6c0 in ap_mpm_pod_check (pod=0x8011bcb0)
    at /usr/include/bits/unistd.h:35
#3  0x8003a017 in child_main (child_num_arg=<value optimized out>)
    at /usr/src/debug/httpd-2.2.4/server/mpm/worker/worker.c:1258
#4  0x8003a1c0 in make_child (s=0x8005f468, slot=0)
    at /usr/src/debug/httpd-2.2.4/server/mpm/worker/worker.c:1341
#5  0x8003a2ad in startup_children (number_to_start=1)
    at /usr/src/debug/httpd-2.2.4/server/mpm/worker/worker.c:1375
#6  0x8003b022 in ap_mpm_run (_pconf=0x8005d5c8, plog=0x8008b680,
    s=0x8005f468)
    at /usr/src/debug/httpd-2.2.4/server/mpm/worker/worker.c:1725
#7  0x80010958 in main (argc=-2147109312, argv=0x0)
    at /usr/src/debug/httpd-2.2.4/server/main.c:717
eax            0xfffffe00       -512
ecx            0xbfee4c53       -1074901933
edx            0x1      1
ebx            0x5      5
esp            0xbfee4c20       0xbfee4c20
ebp            0xbfee4c58       0xbfee4c58
esi            0x80130ad8       -2146235688
edi            0x0      0
eip            0xbfffe402       0xbfffe402 <__kernel_vsyscall+2>
eflags         0x200293 [ CF AF SF IF ID ]
cs             0x73     115
ss             0x7b     123
ds             0x7b     123
es             0x7b     123
fs             0x0      0
gs             0x33     51

The actual thread that caused the crash shows somewhere below after
many other threads. It looks like this.

Thread 39 (process 13697):
#0  0xbfffe402 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7bd35f6 in kill () from /lib/i686/nosegneg/libc.so.6
        _nl_category_name_idxs = {11 '\v', 32 ' ', 43 '+', 0 '\0', 20
'\024',
  51 '3', 0 '\0', 63 '?', 72 'H', 80 'P', 91 '[', 104 'h', 119 'w'}
        _nl_msg_cat_cntr = 1
#2  0x8002e91e in sig_coredump (sig=11)
    at /usr/src/debug/httpd-2.2.4/server/mpm_common.c:1188
No locals.
#3  <signal handler called>
No symbol table info available.
#4  0xb71b8e97 in PSSessionMan::terminateStaleSessions (this=0x807eb2d8)
    at services.cpp:910
        lOs = <incomplete type>
        lIR = <value optimized out>
...
...

I know the bug in the 3rdparty code and its fixed. But my question is
why does this thread not show up first in the "thread apply all bt
full" listing. Note that if I run apache attached to gdb and this
crash happens and then I do bt, I see this thread first. (Since gdb
intercepted the sig 11 and apache's sig_coredump handler didn't get
invoked). Is this expected behavior? Can I rely on the understanding
that the thread that has the sig_coredump handler invoked is the
thread that caused the crash?

Thanks,
-- 
Vinay Y S