httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Eissing <stefan.eiss...@greenbytes.de>
Subject http/2 and ssl and crashes
Date Fri, 10 Mar 2017 15:06:25 GMT
Just saw two crashes during repeated runs of the http2 tests:

One during a SSL_free:
httpd(46062,0x700010634000) malloc: *** error for object 0x7fa576000088: incorrect checksum
for freed object - object was probably modified after being freed.
0   libsystem_kernel.dylib        	0x00007fff916e6dd6 __pthread_kill + 10
1   libsystem_pthread.dylib       	0x00007fff917d2787 pthread_kill + 90
2   libsystem_c.dylib             	0x00007fff9164c4bb __abort + 140
3   libsystem_c.dylib             	0x00007fff9164c42f abort + 144
4   libsystem_malloc.dylib        	0x00007fff91746f21 szone_error + 626
5   libsystem_malloc.dylib        	0x00007fff9173cf65 tiny_free_list_remove_ptr + 292
6   libsystem_malloc.dylib        	0x00007fff91751925 tiny_free_no_lock + 1533
7   libsystem_malloc.dylib        	0x00007fff917520b5 free_tiny + 671
8   libcrypto.1.0.0.dylib         	0x000000010e1180d5 CRYPTO_free + 37
9   libcrypto.1.0.0.dylib         	0x000000010e1bd4ed BIO_free_all + 157
10  libssl.1.0.0.dylib            	0x000000010e0dc738 SSL_free + 152
11  mod_ssl.so                    	0x000000010e082ca9 ssl_filter_io_shutdown + 441 (ssl_engine_io.c:1144)
12  mod_ssl.so                    	0x000000010e080d37 ssl_io_filter_output + 1207 (ssl_engine_io.c:1826)
13  mod_ssl.so                    	0x000000010e080826 ssl_io_filter_coalesce + 822 (ssl_engine_io.c:1763)
14  httpd                         	0x000000010dda319e ap_start_lingering_close + 222 (connection.c:89)
15  mod_mpm_event.so              	0x000000010e3ca955 worker_thread + 2005 (event.c:799)
16  libsystem_pthread.dylib       	0x00007fff917cfaab _pthread_body + 180
17  libsystem_pthread.dylib       	0x00007fff917cf9f7 _pthread_start + 286
18  libsystem_pthread.dylib       	0x00007fff917cf1fd thread_start + 13

and one during http/2 request processing:
0   libsystem_c.dylib             	0x00007fff915eeb52 strlen + 18
1   libapr-1.0.dylib              	0x000000010646c211 apr_pstrdup + 33 (apr_strings.c:77)
2   libapr-1.0.dylib              	0x000000010646f8c6 apr_table_set + 502 (apr_tables.c:529)
3   mod_ssl.so                    	0x00000001065c50f2 ssl_hook_Fixup + 242 (ssl_engine_kernel.c:1374)
4   httpd                         	0x00000001062ff6da ap_process_request_internal + 1658 (request.c:83)
5   httpd                         	0x00000001063024ea ap_sub_req_method_uri + 202 (request.c:2276)
6   mod_dir.so                    	0x0000000106873a5c dir_fixups + 1036 (mod_dir.c:323)
7   httpd                         	0x00000001062ff6da ap_process_request_internal + 1658 (request.c:83)
8   httpd                         	0x0000000106316a18 ap_process_async_request + 344 (http_request.c:442)
9   httpd                         	0x0000000106316ac9 ap_process_request + 25 (http_request.c:481)
10  mod_http2.so                  	0x00000001068a43df h2_task_process_conn + 399 (h2_task.c:623)
11  httpd                         	0x00000001062e4c75 ap_run_process_connection + 53 (connection.c:42)
12  mod_http2.so                  	0x00000001068a5501 h2_task_do + 337 (h2_task.c:581)
13  mod_http2.so                  	0x00000001068a84d6 execute + 118 (h2_worker.c:50)
14  libsystem_pthread.dylib       	0x00007fff917cfaab _pthread_body + 180
15  libsystem_pthread.dylib       	0x00007fff917cf9f7 _pthread_start + 286
16  libsystem_pthread.dylib       	0x00007fff917cf1fd thread_start + 13

The last one led me to look at the hook more closely:

ssl_engine_kernel.c: line 1373
#ifdef HAVE_TLSEXT
    /* add content of SNI TLS extension (if supplied with ClientHello) */
    if ((servername = SSL_get_servername(ssl, TLSEXT_NAMETYPE_host_name))) {
        apr_table_set(env, "SSL_TLS_SNI", servername);
    }
#endif

This is called unconditionally on every request from several threads on the same "SSL*" concurrently.
The method is innocent enough, so that should not be a problem.

However, it seems to me, with the changes I did in v1.9.0 of orderly cleaning up http2 on
a pre-clean of the conn_rec->pool that this is too late. Because, I assume, when the conn_rec->pool
is cleared, mod_ssl has already said goodbye to the connection and freed the SSL*.

But background processing of http2 requests is still going on. And those requests access SSL*
memory (access to SSLConnRec* from conn_rec->master is unguarded), which might have been
freed and then all sort of things can happen.

If someone has the time and energy to look at those things, I'd appreciate it. I am also looking
for a good idea to fix this and the damn +StdEnvVars as well, while we are at it. All conncurrent
access to SSL* needs to go, one way or the other.

And yes, this could explain the curious crashes by Stefan Priebe during connection shutdown.

Stefan Eissing

<green/>bytes GmbH
Hafenstrasse 16
48155 M√ľnster
www.greenbytes.de


Mime
View raw message