httpd-modules-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Noordhuis <i...@bnoordhuis.nl>
Subject Re: Bucket brigade & filter thread safety
Date Tue, 11 Sep 2012 15:37:18 GMT
On Mon, Sep 10, 2012 at 3:53 PM, Alex Bligh <alex@alex.org.uk> wrote:
> Ben,
>
>
>> No, but the documentation omits some crucial details.
>> apr_pool_create() is thread-safe only if:
>>
>> 1. libapr is compiled with APR_HAS_THREADS
>> 2. APR_POOL_DEBUG is turned off
>> 3. the parent pool has a thread-safe allocator (which is true for the
>> global allocator that is used when parent=NULL, provided conditions #1
>> and #2 are satisfied)
>>
>> The pools you get from httpd core satisfy #3 but a module may replace
>> e.g. r->pool with another pool that doesn't. Ergo, don't rely on a
>> pool being thread safe unless you explicitly make it so.
>
>
> Ah, OK. I had thought that was only an issue as when the pool create
> was running (at which point I am single threaded). But I can fix
> that - thanks.
>
>
>>>> That won't solve all your problems though. Bucket brigades are not
>>>> thread safe, you will need something to synchronize on.
>>>
>>>
>>>
>>> So what I was trying to do was to use
>>> a) the input bucket brigade in thread #1 (main thread)
>>> b) the output bucket brigade in thread #2
>>> in an attempt to avoid synchronization
>>>
>>> But what I don't understand is whether thread #2, in writing
>>> to the output filters (which presumably have a reference
>>> to r->pool) will need synchronisation.
>>
>>
>> Yes. It's not just because r->pool may or may not be synchronized, the
>> internal structure of the bucket brigade is not protected by any locks
>> either.
>
>
> Oh I understand that, but I thought in the example above only
> thread #1 would be accessing the input bucket brigade and
> only thread #2 would be accessing the output bucket brigade,
> so there would be no need for synchronisation as they were
> thread local.
>
>
>>> And if I have to synchronize, how do I do that in practice?
>>> Thread #2 does and ap_fwrite/ap_flush so I can hold a mutex
>>> there. But what do I do in thread #1, which calls ap_brigade_get
>>> and blocks? I can't hold a mutex during that. I can make it
>>> a non-blocking ap_brigade_get (if I understood how to do it)
>>
>>
>> Non-blocking reads are pretty straightforward:
>>
>>   apr_thread_mutex_lock(&mutex);
>>   rv = ap_get_brigade(f->next, bb, AP_MODE_READBYTES, APR_NONBLOCK_READ,
>> len);   if (APR_STATUS_IS_EAGAIN(rv)) apr_thread_cond_wait(&cond, &mutex);
>>   rv = ap_get_brigade(f->next, bb, AP_MODE_READBYTES, APR_NONBLOCK_READ,
>> len);   apr_thread_mutex_unlock(&mutex);
>>
>> The other thread wakes up this thread with apr_thread_cond_signal(&cond).
>
>
> I think I may not have explained what I am doing clearly. One thread
> is doing input (from the apache client), and the other output (to
> the apache client). The ap_brigade_get is in the input thread (the
> main thread) and is blocking on the client sending more data. So
> the thing that would need to wake the thread up is more data becoming
> ready from the client - nothing to do with the other
> thread. I don't know how to detect that.
>
>
>>> but what I really need is the equivalent of a select() which
>>> I can do with the mutex not held (or some way to drop the mutex
>>> during the raw reads). Any ideas?
>>
>>
>> You could set up a pollset in the main thread and funnel incoming data
>> into your bucket brigade. Not terribly efficient (lots of context
>> switches) but the real world impact may very well be negligible and
>> you can support multi-process setups with zero changes to your code.
>
>
> Hmm, well if I could use a pollset in my main thread I wouldn't
> need bucket brigades at all. But as this is data coming from the
> client, surely it's going to be in a bucket brigade already as
> it will have passed through all the input filters etc., having
> been read by apache itself?
>
> Diagramatically:
>
>
>             | APACHE  |   ........... MODULE ........... |
>
>  Client <==> Apache ----> ap_get_brigade ----> do_something  [thread1]
>                   ^
>                   |------ ap_fwrite <--- do_something_else   [thread2]
>
> Each thread has a different bucket brigade with a different allocator.
>
> ap_get_brigade either needs to block, or if it's non-blocking it
> needs to wait on a condwait or something for /apache/ to produce
> more data from the client, not on the other thread.

Right, I think I see what you mean.

Apache may not be a perfect fit. I once had to solve a similar issue
and I eventually settled on sending the socket to another process.
Managing mostly dormant connections turned out to be too much of a
headache to do from within httpd.

Mime
View raw message