httpd-modules-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Bligh <a...@alex.org.uk>
Subject Re: Bucket brigade & filter thread safety
Date Mon, 10 Sep 2012 13:53:05 GMT
Ben,

> No, but the documentation omits some crucial details.
> apr_pool_create() is thread-safe only if:
>
> 1. libapr is compiled with APR_HAS_THREADS
> 2. APR_POOL_DEBUG is turned off
> 3. the parent pool has a thread-safe allocator (which is true for the
> global allocator that is used when parent=NULL, provided conditions #1
> and #2 are satisfied)
>
> The pools you get from httpd core satisfy #3 but a module may replace
> e.g. r->pool with another pool that doesn't. Ergo, don't rely on a
> pool being thread safe unless you explicitly make it so.

Ah, OK. I had thought that was only an issue as when the pool create
was running (at which point I am single threaded). But I can fix
that - thanks.

>>> That won't solve all your problems though. Bucket brigades are not
>>> thread safe, you will need something to synchronize on.
>>
>>
>> So what I was trying to do was to use
>> a) the input bucket brigade in thread #1 (main thread)
>> b) the output bucket brigade in thread #2
>> in an attempt to avoid synchronization
>>
>> But what I don't understand is whether thread #2, in writing
>> to the output filters (which presumably have a reference
>> to r->pool) will need synchronisation.
>
> Yes. It's not just because r->pool may or may not be synchronized, the
> internal structure of the bucket brigade is not protected by any locks
> either.

Oh I understand that, but I thought in the example above only
thread #1 would be accessing the input bucket brigade and
only thread #2 would be accessing the output bucket brigade,
so there would be no need for synchronisation as they were
thread local.

>> And if I have to synchronize, how do I do that in practice?
>> Thread #2 does and ap_fwrite/ap_flush so I can hold a mutex
>> there. But what do I do in thread #1, which calls ap_brigade_get
>> and blocks? I can't hold a mutex during that. I can make it
>> a non-blocking ap_brigade_get (if I understood how to do it)
>
> Non-blocking reads are pretty straightforward:
>
>   apr_thread_mutex_lock(&mutex);
>   rv = ap_get_brigade(f->next, bb, AP_MODE_READBYTES, APR_NONBLOCK_READ,
> len);   if (APR_STATUS_IS_EAGAIN(rv)) apr_thread_cond_wait(&cond, &mutex);
>   rv = ap_get_brigade(f->next, bb, AP_MODE_READBYTES, APR_NONBLOCK_READ,
> len);   apr_thread_mutex_unlock(&mutex);
>
> The other thread wakes up this thread with apr_thread_cond_signal(&cond).

I think I may not have explained what I am doing clearly. One thread
is doing input (from the apache client), and the other output (to
the apache client). The ap_brigade_get is in the input thread (the
main thread) and is blocking on the client sending more data. So
the thing that would need to wake the thread up is more data becoming
ready from the client - nothing to do with the other
thread. I don't know how to detect that.

>> but what I really need is the equivalent of a select() which
>> I can do with the mutex not held (or some way to drop the mutex
>> during the raw reads). Any ideas?
>
> You could set up a pollset in the main thread and funnel incoming data
> into your bucket brigade. Not terribly efficient (lots of context
> switches) but the real world impact may very well be negligible and
> you can support multi-process setups with zero changes to your code.

Hmm, well if I could use a pollset in my main thread I wouldn't
need bucket brigades at all. But as this is data coming from the
client, surely it's going to be in a bucket brigade already as
it will have passed through all the input filters etc., having
been read by apache itself?

Diagramatically:


             | APACHE  |   ........... MODULE ........... |

  Client <==> Apache ----> ap_get_brigade ----> do_something  [thread1]
                   ^
                   |------ ap_fwrite <--- do_something_else   [thread2]

Each thread has a different bucket brigade with a different allocator.

ap_get_brigade either needs to block, or if it's non-blocking it
needs to wait on a condwait or something for /apache/ to produce
more data from the client, not on the other thread.

-- 
Alex Bligh

Mime
View raw message