httpd-bugs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzi...@apache.org
Subject DO NOT REPLY [Bug 39744] New: - SIGBUS on FreeBSD 4.x (semi-reproducable)
Date Wed, 07 Jun 2006 09:52:24 GMT
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=39744>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=39744

           Summary: SIGBUS on FreeBSD 4.x (semi-reproducable)
           Product: Apache httpd-2
           Version: 2.2.2
          Platform: Other
        OS/Version: FreeBSD
            Status: NEW
          Severity: normal
          Priority: P2
         Component: mod_ssl
        AssignedTo: bugs@httpd.apache.org
        ReportedBy: apache@jdc.parodius.com


This isn't going to be fun to explain.

FreeBSD pentarou.parodius.com 4.11-STABLE FreeBSD 4.11-STABLE #0: Sun May 28 
12:02:42 PDT 2006     root@pentarou.parodius.com:/usr/o
bj/usr/src/sys/PENTAROU  i386

Situation:

Occasionally our production Apache server dumps core on
a SIGBUS:

pid 31008 (httpd), uid 0: exited on signal 10 (core dumped)

The time of the problem is very suspicious: a few seconds
after midnight, which is when our newsyslog is scheduled to
run for Apache logfiles.  Note that we send Apache a signal 30,
which is SIGUSR1:

/var/log/httpd-access.log               640  13    *    @T00  
B /var/run/httpd.pid 30
/var/log/httpd-error.log                640  13    *    @T00  
B /var/run/httpd.pid 30
/var/log/httpd-cgiwrap.log              640  13    *    @T00  Z
/var/log/suphp.log                      640  13    *    @T00  Z

Apache error_log says:

[Wed Jun 07 00:00:02 2006] [notice] Graceful restart requested, doing restart

Then the log rotates.  New log says:

[Wed Jun 07 00:00:02 2006] [notice] mod_bw : Memory Allocated 112 bytes (each 
conf takes 28 bytes)
[Wed Jun 07 00:00:02 2006] [notice] mod_bw : Version 0.7 - Initialized [4 Confs]
[Wed Jun 07 00:00:03 2006] [notice] seg fault or similar nasty error detected 
in the parent process

Where things get crazy: the core backtrace shows a problem that seems
to point to the FreeBSD crypto library.  Our system libraries *are
not* built with debugging symbols (but Apache is).  We have one SSL
virtualhost (support.parodius.com). Here's the bt full:

pentarou# gdb /usr/local/sbin/httpd httpd.31008.uid_0.core
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...Deprecated bfd_read called 
at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dwarf2read.c 
line 3049 in dwarf2_read_section

Core was generated by `httpd'.
Program terminated with signal 10, Bus error.
Reading symbols from /usr/lib/libm.so.2...done.
Reading symbols from /usr/local/lib/libaprutil-1.so.2...done.
Reading symbols from /usr/local/lib/libexpat.so.6...done.
Reading symbols from /usr/local/lib/libiconv.so.3...done.
Reading symbols from /usr/local/lib/libapr-1.so.2...done.
Reading symbols from /usr/lib/libcrypt.so.2...done.
Reading symbols from /usr/lib/libc.so.4...done.
Reading symbols from /usr/local/libexec/apache22/mod_authn_file.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_authn_dbm.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_authn_anon.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_authn_default.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_authz_host.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_authz_groupfile.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_authz_user.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_authz_dbm.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_authz_owner.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_authz_default.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_auth_basic.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_include.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_filter.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_charset_lite.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_deflate.so...done.
Reading symbols from /usr/lib/libz.so.2...done.
Reading symbols from /usr/local/libexec/apache22/mod_log_config.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_logio.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_env.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_mime_magic.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_expires.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_headers.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_usertrack.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_unique_id.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_setenvif.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_version.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_ssl.so...done.
Reading symbols from /usr/lib/libssl.so.3...done.
Reading symbols from /usr/lib/libcrypto.so.3...done.
Reading symbols from /usr/local/libexec/apache22/mod_mime.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_autoindex.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_cgi.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_vhost_alias.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_negotiation.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_dir.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_actions.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_userdir.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_alias.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_rewrite.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_suphp.so...done.
Reading symbols from /usr/local/libexec/apache22/mod_bw.so...done.
Reading symbols from /usr/libexec/ld-elf.so.1...done.
#0  0x2847663c in engine_table_select () from /usr/lib/libcrypto.so.3
(gdb) bt full
#0  0x2847663c in engine_table_select () from /usr/lib/libcrypto.so.3
No symbol table info available.
#1  0x28454df4 in ENGINE_get_default_RSA () from /usr/lib/libcrypto.so.3
No symbol table info available.
#2  0x28453209 in RSA_new_method () from /usr/lib/libcrypto.so.3
No symbol table info available.
#3  0x2845309d in RSA_new () from /usr/lib/libcrypto.so.3
No symbol table info available.
#4  0x2846c6f7 in RSAPrivateKey_asn1_meth () from /usr/lib/libcrypto.so.3
No symbol table info available.
#5  0x28478688 in ASN1_item_ex_new () from /usr/lib/libcrypto.so.3
No symbol table info available.
#6  0x284784cf in ASN1_item_ex_new () from /usr/lib/libcrypto.so.3
No symbol table info available.
#7  0x28474361 in ASN1_item_ex_d2i () from /usr/lib/libcrypto.so.3
No symbol table info available.
#8  0x28473e12 in ASN1_item_d2i () from /usr/lib/libcrypto.so.3
No symbol table info available.
#9  0x2846c7ab in d2i_RSAPublicKey () from /usr/lib/libcrypto.so.3
No symbol table info available.
#10 0x28467bfe in d2i_PublicKey () from /usr/lib/libcrypto.so.3
No symbol table info available.
#11 0x284666be in X509_PUBKEY_get () from /usr/lib/libcrypto.so.3
No symbol table info available.
#12 0x284656e9 in X509_get_pubkey () from /usr/lib/libcrypto.so.3
No symbol table info available.
#13 0x28380183 in ssl_util_algotypeof (pCert=0x830b200, pKey=0x0) at 
ssl_util.c:144
        t = 0
        pFreeKey = (struct evp_pkey_st *) 0x0
#14 0x28374e41 in ssl_pphrase_Handle (s=0x80bbeb8, p=0x8146018) at 
ssl_engine_pphrase.c:216
        mc = (struct {...} *) 0x80b8320
        sc = (struct SSLSrvConfigRec *) 0x8299378
        pServ = (struct server_rec *) 0x81ada90
        cpVHostID = 0x81470c0 "support.parodius.com:443"
        szPath = "/conf/ME/certificates/support.parodius.com.crt\000ions\n\000
\001\000\000 \211)\b\200\211)\b(¿¿\000\000\000\000`.)\b\000-conference/x-
cooltalk\t\tice\000\000ltimedia\000\000\000ml\000\000eb\000\000ge+xml\000\000 
dmg\000\000.\000\000media type\000\000\000\000\211\000\000\000L\001\000\000 \002
\000\000\024\001\000\000\203\001\000\000N\000\000\000\b\000\000\000 \000\000
\000\202\000\000\000\207\000\000\000\020\000\000\000"...
        pPrivateKey = (struct evp_pkey_st *) 0x65747962
        asn1 = (struct {...} *) 0x20383220
        ucp = (unsigned char *) 0x73656b61 <Address 0x73656b61 out of bounds>
        length = 1948254218
        pX509Cert = (struct x509_st *) 0x830b200
        bReadable = 1866670132
        aPassPhrase = (struct apr_array_header_t *) 0x81470a0
        nPassPhrase = 0
        nPassPhraseCur = 1769236846
        cpPassPhraseCur = 0x49202d20 <Address 0x49202d20 out of bounds>
        nPassPhraseRetry = 925773856
        nPassPhraseDialog = 0
        nPassPhraseDialogCur = 1919243808
        bPassPhraseDialogOnce = 975206242
        cpp = (char **) 0x5f646f6d
        i = 0
        j = 0
        algoCert = 0
        algoKey = 0
        at = 842021424
        an = 0x303a3030 <Address 0x303a3030 out of bounds>
        cp = 0x20373020 <Address 0x20373020 out of bounds>
        pkey_mtime = 0
        isterm = 1
        rv = 0
#15 0x2836bbe1 in ssl_init_Module (p=0x80ba018, plog=0x80e8018, 
ptemp=0x8146018, base_server=0x80bbeb8) at ssl_engine_init.c:261
        mc = (struct {...} *) 0x80b8320
        sc = (struct SSLSrvConfigRec *) 0x8152f68
        s = (struct server_rec *) 0x0
#16 0x807602d in ap_run_post_config (pconf=0x80ba018, plog=0x80e8018, 
ptemp=0x8146018, s=0x80bbeb8) at config.c:93
        pHook = (struct ap_LINK_post_config_t *) 0x829a480
        n = 9
        rv = 0
#17 0x805f95f in main (argc=1, argv=0xbfbffc08) at main.c:706
        c = 0 '\000'
        configtestonly = 0
        confname = 0x8096599 "etc/apache22/httpd.conf"
        def_server_root = 0x80965b1 "/usr/local"
        temp_error_log = 0x0
        error = 0x0
        process = (struct process_rec *) 0x80b8098
        server_conf = (struct server_rec *) 0x80bbeb8
        pglobal = (struct apr_pool_t *) 0x80b8018
        pconf = (struct apr_pool_t *) 0x80ba018
        plog = (struct apr_pool_t *) 0x80e8018
        ptemp = (struct apr_pool_t *) 0x8146018
        pcommands = (struct apr_pool_t *) 0x80bc018
        opt = (struct apr_getopt_t *) 0x80bc0b0
        rv = 0
        mod = (struct module_struct **) 0x80a367c
        optarg = 0xbfbffbac ""
        signal_server = (int (*)(int *, struct apr_pool_t *)) 0

There was mention of this problem (or similar) on freebsd-ports
a few weeks ago, and the claim was that php5-openssl was causing
the problem.  This doesn't seem to be the case -- we build PHP as
a CGI, and we _do not_ use php5-openssl:

apache-2.2.2        Version 2.2 of Apache web server with prefork MPM.
cgiwrap-3.9_3       Securely execute Web CGI scripts
php5-5.1.4          PHP Scripting Language (Apache Module and CLI)
php5-gd-5.1.4       The gd shared extension for php
php5-iconv-5.1.4    The iconv shared extension for php
php5-mbstring-5.1.4 The mbstring shared extension for php
php5-mysql-5.1.4    The mysql shared extension for php
php5-pcre-5.1.4     The pcre shared extension for php
php5-session-5.1.4  The session shared extension for php
php5-sqlite-5.1.4   The sqlite shared extension for php
php5-xml-5.1.4      The xml shared extension for php
php5-zlib-5.1.4     The zlib shared extension for php
suphp-0.5.2_2       Securely execute PHP scripts

I can provide the coredump if need be, although I'm not just going to toss it 
out to anyone who asks for it, since it's a setuid 0 process which sig10'd and 
the core contains some sensitive information.

I say this is "semi"-reproducable because this is the 2nd or 3rd time I've seen 
it happen.  I can't get it to happen every time/day.

I'm left wondering if the fact that I'm sending SIGUSR1 to Apache -- twice very 
quickly -- due to the log rotation stuff is what's causing the problem.

Let me know if you have some ideas where I should look, or additional variables 
I should print out via `p' that'll help in the debugging process.  If you're 
*absolutely sure* it's a FreeBSD problem, then I'll take it up on the FreeBSD 
crypto/secure list.

-- 
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org


Mime
View raw message