httpd-modules-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Noordhuis <i...@bnoordhuis.nl>
Subject Re: Bucket brigade & filter thread safety
Date Sun, 09 Sep 2012 22:55:44 GMT
On Sun, Sep 9, 2012 at 2:31 PM, Alex Bligh <alex@alex.org.uk> wrote:
> I am trying to work out how to develop a thread-safe module with two
> threads, one thread reading and one thread writing. I'm using mpm-prefork
> on apache 2.2.14-5ubuntu8.9 in case that matters. The module is a websocket
> proxy.
>
> My main thread (#1) is doing (in essence - error handling removed)
>
> apr_pool_create(&opool, r->pool);
> apr_bucket_alloc_t *oallocator = apr_bucket_alloc_create(opool);
> apr_bucket_brigade *obb = apr_brigade_create(opool, oallocator);
> apr_thread_create (... otherthread ...);
> while (1) {
>    bb = apr_brigade_create(r->pool, r->connection->bucket_alloc);
>    ap_get_brigade(r->input_filters, bb, AP_MODE_READBYTES,
>                   APR_BLOCK_READ, bufsiz));
>    apr_brigade_flatten(bb, buffer, &bufsiz);
>    apr_brigade_destroy(bb);
>    do_something_with(buffer);
> }
>
> The other thread (#2) is doing (in essence)
>
> while (1) {
>   apr_pollset_poll(recvpollset, timeout, ... );
>   apr_socket_recv(socket, buf, ...)
>   do_something_else_with(buf);
>   ap_fwrite(r->connection->output_filters, obb, ...);
>   ap_fflush(r->connection->output_filters, obb, ...);
> }
>
> I am suffering from very occasional corruption of the bucket brigade which
> normally shows up as a corrupted pool pointer or a bogus bucket entry in
> thread #1 (for instance a SEGV in apr_brigade_length). Interestingly this
> is the relatively quiet input brigade which is only ever touched by the
> main apache thread. It's almost as if an allocator is not thread safe.

That's because it isn't unless you explicitly make it so (which no MPM does).

> However, I'm using a separate bucket allocator (see code above) and (at
> least in my code) a separate pool.

They're not really separate, the sub pool is created off r->pool. You
should probably use apr_pool_create_ex() here with parent=NULL and an
allocator that you created with apr_allocator_create() +
apr_allocator_mutex_set().

That won't solve all your problems though. Bucket brigades are not
thread safe, you will need something to synchronize on.

> Could the output filter chain somehow be using the allocator attached to
> the request (i.e. thread #1)? If so, how can I stop this? I can't run the
> ap_fwrite/ap_flush under a mutex and have the same mutex held during
> ap_get_brigade, as the latter blocks (and I can't see how to use the
> non-blocking version without spinning).
>
> [apologies for the partial dupe on the apache-users mailing list - I think
> I've got nearer the problem since then and this list seems more
> appropriate]
>
> --
> Alex Bligh

Mime
View raw message