httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Viljanen <biocro...@gmail.com>
Subject [users@httpd] Apache httpd dynamically links to wrong ssl library on macOS causing a crash
Date Wed, 14 Feb 2018 10:12:50 GMT
Hi,

I’m currently working on a new php formula in the Homebrew package manager
project where we have noticed that httpd crashes on macOS High Sierra under
certain conditions. My question now is if this is a bug in Apache httpd
that I should file in the bug tracker or if you can point me in the right
direction to get to the bottom of this.

In short what happens is that httpd (loading libphp7) crashes when the php
snmp extension in php is loaded with a reference to boringssl but when the
snmp extension is omitted it doesn’t crash.

# httpd -X
[1]    52393 abort      httpd -X
#

Using the php binary alone with the snmp extension enabled causes no issues
as far as we’ve been able to tell.

The stack trace from the crash looks like this

Process:               httpd [52393]
Path:                  /usr/local/Cellar/httpd/2.4.29_1/bin/httpd
Identifier:            httpd
Version:               0
Code Type:             X86-64 (Native)
Parent Process:        zsh [541]
Responsible:           httpd [52393]
User ID:               501

Date/Time:             2018-02-14 10:49:57.591 +0100
OS Version:            Mac OS X 10.13.3 (17D47)
Report Version:        12
Anonymous UUID:        AC6370DE-1692-0BA1-08A2-BBE6153A8410

Sleep/Wake UUID:       F545ED14-2DF3-4149-89EA-86A67EC93AB1

Time Awake Since Boot: 85000 seconds
Time Since Wake:       84000 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Application Specific Information:
Assertion failed: (ctx), function digest_update, file
/BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl/boringssl-109.40.1/apple/crypto/digests.c,
line 49.

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        0x00007fff5f670e3e __pthread_kill + 10
1   libsystem_pthread.dylib        0x00007fff5f7af150 pthread_kill + 333
2   libsystem_c.dylib              0x00007fff5f5cd312 abort + 127
3   libsystem_c.dylib              0x00007fff5f595368 __assert_rtn + 320
4   libboringssl.dylib            0x00007fff5d4b4ddc digest_update + 51
5   libcrypto.35.dylib            0x00007fff5db0fe08 EVP_DigestInit_ex + 920
6   libcrypto.35.dylib            0x00007fff5db0fa6a EVP_DigestInit + 42
7   libnetsnmp.30.dylib            0x000000010c089b8b sc_hash + 202
8   libnetsnmp.30.dylib            0x000000010c088f57 hash_engineID + 116
9   libnetsnmp.30.dylib            0x000000010c088d7f
search_enginetime_list + 40
10  libnetsnmp.30.dylib            0x000000010c08909b set_enginetime + 61
11  libnetsnmp.30.dylib            0x000000010c088a82
init_snmpv3_post_config + 150
12  libnetsnmp.30.dylib            0x000000010c08a7a5 snmp_call_callbacks +
432
13  snmp.so                        0x000000010bdcf1b0 zm_startup_snmp + 59
14  libphp7.so                    0x0000000108ee023a zend_startup_module_ex
+ 305
15  libphp7.so                    0x0000000108ee05bb
zend_startup_module_zval + 12
16  libphp7.so                    0x0000000108eec6e1 zend_hash_apply + 201
17  libphp7.so                    0x0000000108ee04d8 zend_startup_modules +
47
18  libphp7.so                    0x0000000108e82b0a php_module_startup +
2215
19  libphp7.so                    0x0000000108f9b426 php_apache2_startup +
21
20  libphp7.so                    0x0000000108f9ad33
php_apache_server_startup + 105
21  httpd                          0x00000001088bd50b ap_run_post_config +
69
22  httpd                          0x00000001088c4c2c main + 2076
23  libdyld.dylib                  0x00007fff5f521115 start + 1

Thread 1:
0   libsystem_kernel.dylib        0x00007fff5f671562 __workq_kernreturn + 10
1   libsystem_pthread.dylib        0x00007fff5f7ac06a _pthread_wqthread +
1035
2   libsystem_pthread.dylib        0x00007fff5f7abc4d start_wqthread + 13

Thread 2:
0   libsystem_kernel.dylib        0x00007fff5f671562 __workq_kernreturn + 10
1   libsystem_pthread.dylib        0x00007fff5f7ac26f _pthread_wqthread +
1552
2   libsystem_pthread.dylib        0x00007fff5f7abc4d start_wqthread + 13

Thread 3:
0   libsystem_pthread.dylib        0x00007fff5f7abc40 start_wqthread + 0
1   ???                            0x000000005e943d01 0 + 1586773249

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000000000000  rbx: 0x00007fff9857a340  rcx:
0x00007ffee7343998  rdx: 0x0000000000000000
  rdi: 0x0000000000000307  rsi: 0x0000000000000006  rbp:
0x00007ffee73439d0  rsp: 0x00007ffee7343998
   r8: 0x0000000000000000   r9: 0x0000000000000000  r10:
0x0000000000000000  r11: 0x0000000000000206
  r12: 0x0000000000000307  r13: 0x000000010bdff000  r14:
0x0000000000000006  r15: 0x000000000000002d
  rip: 0x00007fff5f670e3e  rfl: 0x0000000000000206  cr2: 0x00007fff98558148

Logical CPU:     0
Error Code:      0x02000148
Trap Number:     133

(this has been shortened and the full outtake can be found in the gist
below)

Looking at that stack trace it seems that there should be an issue with
boringssl under macOS however none of the components httpd, php nor the
snmp extension has been linked against boringssl or the macos provided ssl
binaries.

The shared libraries that the snmp extension uses looks like this
# otool -L /usr/local/opt/php/lib/php/20170718/snmp.so
/usr/local/opt/php/lib/php/20170718/snmp.so:
/usr/local/opt/net-snmp/lib/libnetsnmp.30.dylib (compatibility version
31.0.0, current version 31.3.0)
/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version
1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
1252.0.0)

(brew linkage is a homebrew command to list linked libraries across all
packaged components)
# brew linkage httpd
System libraries:
  /usr/lib/libSystem.B.dylib
  /usr/lib/libexpat.1.dylib
  /usr/lib/libiconv.2.dylib
  /usr/lib/libxml2.2.dylib
  /usr/lib/libz.1.dylib
Homebrew libraries:
  /usr/local/opt/apr/libexec/lib/libapr-1.0.dylib (apr)
  /usr/local/opt/apr-util/libexec/lib/libaprutil-1.0.dylib (apr-util)
  /usr/local/opt/nghttp2/lib/libnghttp2.14.dylib (nghttp2)
  /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (openssl)
  /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (openssl)
  /usr/local/opt/pcre/lib/libpcre.1.dylib (pcre)

# brew linkage php
System libraries:
  /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
  /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP
  /usr/lib/libSystem.B.dylib
  /usr/lib/libbz2.1.0.dylib
  /usr/lib/libc++.1.dylib
  /usr/lib/libcurl.4.dylib
  /usr/lib/libedit.3.dylib
  /usr/lib/libexpat.1.dylib
  /usr/lib/libexslt.0.dylib
  /usr/lib/libiconv.2.dylib
  /usr/lib/libicucore.A.dylib
  /usr/lib/libncurses.5.4.dylib
  /usr/lib/libpam.2.dylib
  /usr/lib/libresolv.9.dylib
  /usr/lib/libtidy.A.dylib
  /usr/lib/libxml2.2.dylib
  /usr/lib/libxslt.1.dylib
  /usr/lib/libz.1.dylib
Homebrew libraries:
  /usr/local/opt/apr/libexec/lib/libapr-1.0.dylib (apr)
  /usr/local/opt/apr-util/libexec/lib/libaprutil-1.0.dylib (apr-util)
  /usr/local/opt/argon2/lib/libargon2.1.dylib (argon2)
  /usr/local/opt/aspell/lib/libaspell.15.dylib (aspell)
  /usr/local/opt/aspell/lib/libpspell.15.dylib (aspell)
  /usr/local/opt/freetds/lib/libsybdb.5.dylib (freetds)
  /usr/local/opt/freetype/lib/libfreetype.6.dylib (freetype)
  /usr/local/opt/gettext/lib/libintl.8.dylib (gettext)
  /usr/local/opt/gmp/lib/libgmp.10.dylib (gmp)
  /usr/local/opt/icu4c/lib/libicudata.60.2.dylib (icu4c)
  /usr/local/opt/icu4c/lib/libicui18n.60.dylib (icu4c)
  /usr/local/opt/icu4c/lib/libicuio.60.dylib (icu4c)
  /usr/local/opt/icu4c/lib/libicuuc.60.dylib (icu4c)
  /usr/local/opt/jpeg/lib/libjpeg.9.dylib (jpeg)
  /usr/local/opt/libpng/lib/libpng16.16.dylib (libpng)
  /usr/local/opt/libpq/lib/libpq.5.dylib (libpq)
  /usr/local/opt/libsodium/lib/libsodium.23.dylib (libsodium)
  /usr/local/opt/libzip/lib/libzip.5.dylib (libzip)
  /usr/local/opt/net-snmp/lib/libnetsnmp.30.dylib (net-snmp)
  /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (openssl)
  /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (openssl)
  /usr/local/opt/unixodbc/lib/libodbc.2.dylib (unixodbc)
  /usr/local/opt/webp/lib/libwebp.7.dylib (webp)

➜  homebrew-core git:(master) ✗ brew linkage net-snmp
System libraries:
  /usr/lib/libSystem.B.dylib
Homebrew libraries:
  /usr/local/Cellar/net-snmp/5.7.3/lib/libnetsnmp.30.dylib (net-snmp)
  /usr/local/Cellar/net-snmp/5.7.3/lib/libnetsnmpagent.30.dylib (net-snmp)
  /usr/local/Cellar/net-snmp/5.7.3/lib/libnetsnmpmibs.30.dylib (net-snmp)
  /usr/local/Cellar/net-snmp/5.7.3/lib/libnetsnmptrapd.30.dylib (net-snmp)
  /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (openssl)

So concluding that boringssl in not actively linked against any component
involved It looks like httpd seems to be dynamically linking against the
incorrect library which therefore causes the system to crash.

The workaround that has been found so far is to statically link the Apache
httpd SSL module using  --enable-mods-static=ssl  which make the system not
crash but it doesn’t solve the incorrect dynamic loading.

The httpd binary is built with the following options

./configure
--enable-layout=Slackware-FHS
--prefix=/usr/local/Cellar/httpd/2.4.29_1
--sbindir=/usr/local/Cellar/httpd/2.4.29_1/bin
--mandir=/usr/local/Cellar/httpd/2.4.29_1/share/man
--sysconfdir=/usr/local/etc/httpd
--datadir=/usr/local/var/www
--localstatedir=/usr/local/var
--enable-mpms-shared=all
--enable-mods-shared=all
--enable-cgi
--enable-pie
--enable-suexec
--with-suexec-bin=/usr/local/opt/httpd/bin/suexec
--with-suexec-caller=_www
--with-port=8080
--with-sslport=8443
--with-apr=/usr/local/opt/apr
--with-apr-util=/usr/local/opt/apr-util
--with-mpm=prefork
--with-nghttp2=/usr/local/opt/nghttp2
--with-ssl=/usr/local/opt/openssl
--with-pcre=/usr/local/opt/pcre

The Erlang project seems to have run into a similar issue and have a
solution for their software which can be found here
https://bugs.erlang.org/browse/ERL-439

Full stack trace report can be found here
https://gist.github.com/javian/1c530a2daec748cafe32a47ea2512e94

So to reiterate my question - do you think this is this a bug in Apache
httpd that I should file in the bug tracker or can you point me in another
directions to find the root cause of this issue ? I’m happy to provide
further information and test whatever I can to help troubleshoot this.

Best regards,
Jan

Mime
View raw message