httpd-modules-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Bligh <a...@alex.org.uk>
Subject Re: Mutex protection of output bucket brigade
Date Wed, 12 Jun 2013 12:16:20 GMT
Sorin,

> "Normally" the output brigade is only written during the ap_rprintf/ap_fwrite and the
like.

OK, so in my case, I run with an SSL filter and the request timeout stuff which also operates
as a filter (let's ignore the latter for simplicity)

I think what you are saying is:

1. that the read from the socket on port 443 and the SSL decode of the client input is handled
by the main thread, as is my processing of the decoded information.

2. that the apwrite from my second thread calls the SSL encode of the client output, and this
writes to port 443.

I'm puzzled as to how this works without threading (perhaps the answer is 'it doesn't') as
if both sets of I/O are blocking, there is a deadlock opportunity.

But that aside, is it safe to call apwrite() from one thread whilst there's a read in the
other?

> There is no output brigade when the request_rec structure is created.

Sure. My second thread, dealing with data sent back to the client, is creating the output
bucket brigade, with its own mutex and allocator.

    if (
        ( apr_thread_mutex_create(&oallocatormutex, APR_THREAD_MUTEX_UNNESTED, r->pool)
== APR_SUCCESS) &&
        ( apr_allocator_create(&oallocator) == APR_SUCCESS) &&
        ( apr_allocator_mutex_set(oallocator, oallocatormutex), 1 ) &&
        ( apr_pool_create_ex(&opool, NULL, NULL, oallocator) == APR_SUCCESS) &&
/* WARNING: pool has no parent */
        ( NULL != (obucketallocator = apr_bucket_alloc_create(opool))) &&
        ( NULL != (obb = apr_brigade_create(opool, obucketallocator)))
        ) {

> The way to write something to the client is via ap_pass_brigade(r->output_filter,
brigade). Typically the function that calls ap_pass_brigade creates the brigade first. So
you write something to the brigade and you pass it downstream. The last filter writes its
contents to the network. Be aware that there are filters that buffer the brigades and do not
send them further down the chain unless the buffers are full.

I'm not calling ap_pass_brigade (at least not directly). I'm doing (roughly)

  ap_filter_t *of = state->r->connection->output_filters;
  ap_fwrite(of, state->obb, (const char *)header, pos); /* Header */

IE I'm doing an fwrite to output filter list, using the bucket brigade I've just created.
Is that OK?

> If you created a handler that would just return OK without ever calling an ap_rprintf/ap_fwrite,
the module would create no output brigade and the client would get no response.

Sure, because nothing would ever be sent to the filters.

> I think your hypothesis is unlikely. For it to hold, you'd need an input filter that
is triggered when ap_get_brigade is called, that that input filter creates a brigade and stores
it somewhere and that the spawned thread somehow writes to this stored brigade. So the ap_fwrite
in the spawned thread would need to write to a brigade created in an input filter triggered
by ap_get_brigade in the main thread.

Now I think I understand what happens with ap_fwrite (if that calls ap_pass_brigade), I'm
wondering whether (say) the input filter of mod_ssl ever talks to its output filter.

> To verify this hypothesis, check which is the brigade whose pointers are corrupted. I
mean, where is it referenced, in which apache module, in which apache filter. Then you can
inspect the sources of that module to see if the brigade is shared between input and output.


It's always the output bucket brigade's pointers that are corrupted. Typically this causes
a problem on destroy (as per the stack trace below). apr_brigade_cleanup in this instance
never finishes (i.e. typing 'finish' in gdb on apr_bucket_destroy_noop works fine, but apr_brigade_cleanup
will not 'finish' as it repeatedly calls apr_bucket_destroy_noop) as it goes around in an
infinite loop. Sometimes we see SEGVs too. Typically the stack trace looks the same (i.e.
it's ap_core_output_filter that is confused).

I suspect I may need to compile with DEBUG_BRIGADES or whatever it's called and see if I can
get it to fail more often than once every 5 days on a torture test :-/

-- 
Alex Bligh

(gdb) bt
#0  0x00007f4f2bf082b0 in apr_bucket_destroy_noop () from /usr/lib/libaprutil-1.so.0
#1  0x00007f4f2bf072ac in apr_brigade_cleanup () from /usr/lib/libaprutil-1.so.0
#2  0x00007f4f2c5c38a5 in ap_core_output_filter ()
#3  0x00007f4f25210a3e in ?? () from /usr/lib/apache2/modules/mod_ssl.so
#4  0x00007f4f25210d4e in ?? () from /usr/lib/apache2/modules/mod_ssl.so
#5  0x00007f4f265802a9 in BIO_read () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#6  0x00007f4f2689bc54 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#7  0x00007f4f2689cd7d in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#8  0x00007f4f2689a050 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#9  0x00007f4f2520faf8 in ?? () from /usr/lib/apache2/modules/mod_ssl.so
#10 0x00007f4f25210810 in ?? () from /usr/lib/apache2/modules/mod_ssl.so
#11 0x00007f4f24df528a in ?? () from /usr/lib/apache2/modules/mod_websocket.so
#12 0x00007f4f24df6173 in ?? () from /usr/lib/apache2/modules/mod_websocket.so
#13 0x00007f4f24df688f in ?? () from /usr/lib/apache2/modules/mod_websocket.so
#14 0x00007f4f2c5c5508 in ap_run_handler ()
#15 0x00007f4f2c5c597e in ap_invoke_handler ()
#16 0x00007f4f2c5d5570 in ap_process_request ()
#17 0x00007f4f2c5d2398 in ?? ()
#18 0x00007f4f2c5cbfa8 in ap_run_process_connection ()
#19 0x00007f4f2c5da1d0 in ?? ()
#20 0x00007f4f2c5da93a in ?? ()
#21 0x00007f4f2c5db4e7 in ap_mpm_run ()
#22 0x00007f4f2c5b04a4 in main ()
(gdb) finish



Mime
View raw message