httpd-modules-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Bligh <a...@alex.org.uk>
Subject Mutex protection of output bucket brigade
Date Wed, 12 Jun 2013 08:46:59 GMT
I think I've finally figured out what's going wrong in my module but am unsure what to do about
it.

The module runs on apache 2.2.22 with mpm prefork. Occasionally I am seeing corruption of
the output bucket brigade, primarily the ring pointers (link->next and link->prev) ending
up with strange values.

Having spent some time reading the source, I believe that apache provides no protection to
these pointers, and it's inherently unsafe for a bucket brigade to be used by more than one
thread (even if you are careful with allocators), unless all callers provide their own mutex
protection. As apache itself uses the output bucket brigade without mutex protection, the
output bucket brigade can never be written to by other threads, and therefore ap_fwrite (to
this brigade) can never be safely by any thread other than the main thread. First question:
is this correct?

My module is currently structured as follows.

The main thread creates another thread for each request (the requests are long running websocket
connections).

The main thread does the following:

  while (!done)
  {
    /* Blocking read */
    apr_brigade_create;
    ap_get_brigade;
    apr_brigade_flatten;

    /* Do stuff with the data */
    blocking_socket_write;
  }

The spawned thread does the following

  while (!done)
  {
     blocking_socket_read;
   
     /* do stuff with the data */
     ap_fwrite(output_bucket_brigade);

   }
    
Now, what I believe is happening is as follows. The blocking read in the main thread at some
point calls select(), and does not only do a read, but also also a write of the data in the
output bucket brigade. This removes a bucket from the ring. If this is happens at the same
time as the ap_fwrite in the spawned thread adds something to the output ring, two threads
will be accessing the ring pointers at once.

What I can't figure out is how to fix this.

I can't put in a mutex to protect the ring pointers, because the access to the ring pointers
by apache is outside of my module.

I can't hold a mutex across the blocking read in the main thread, because otherwise my module
won't be able to write data to the output bucket brigade whilst there is no input from the
apache client; as the apache client may be waiting for data to be sent to it, this could cause
deadlock.

And I can't obviously see how to do the read in a non-blocking way.

Any ideas?

-- 
Alex Bligh


Mime
View raw message