Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\))
Subject: Re: buckets and connections (long post)
From: Graham Leggett <minfrin@sharp.fm>
In-Reply-To: <8642583F-79A4-4A8D-A7CF-43CD79029673@greenbytes.de>
Date: Thu, 22 Oct 2015 18:44:19 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <1088C2C9-EC73-4043-934E-B6B14F5DBBAE@sharp.fm>
References: 
 <CAKQ1sVNchFx=egp=Q0tgT+1GcBXiten8mh9STetcZXOfJxhsjQ@mail.gmail.com>
 <CAKQ1sVPjcA9EYNpjcmM9=AU_LCgAsFntOcZnmGoHymds3S5wmA@mail.gmail.com>
 <AB1691BE05AE7F4992697F2A0835627AC6348972@VOEXM10W.internal.vodafone.com>
 <7B24F4D1-6904-4C00-A337-88F712026BE9@greenbytes.de>
 <0D5B13DA-8642-4662-9606-6B5DF94EB87B@sharp.fm>
 <D70FBC43-2E52-4BDD-ADA5-88623657CBEC@jaguNET.com>
 <4D37F1D6-E2FB-408B-9D82-9F7ABABFFE1E@greenbytes.de>
 <8C00B305-BECC-4A48-9A03-8A7CA99EADE4@sharp.fm>
 <8642583F-79A4-4A8D-A7CF-43CD79029673@greenbytes.de>
To: dev@httpd.apache.org

On 22 Oct 2015, at 6:03 PM, Stefan Eissing =
<stefan.eissing@greenbytes.de> wrote:

> This is all true and correct - as long as all this happens in a single =
thread. If you have multiple threads and create sub pools for each from =
a main pool, each and every create and destroy of these sub-pools, plus =
any action on the main pool must be mutex protected. I found out.=20

Normally if you=E2=80=99ve created a thread from a main pool, you need =
to create a pool cleanup for that thread off the main pool that is =
registered with apr_pool_pre_cleanup_register(). In this cleanup, you =
signal the thread to shut down gracefully and then apr_thread_join to =
wait for the thread to shut down, then the rest of the pool can be =
cleaned up.

The =E2=80=9Cpre=E2=80=9D is key to this - the cleanup must run before =
the subpool is cleared.

> Similar with buckets. When you create a bucket in one thread, you may =
not destroy it in another - *while* the bucket_allocator is being used. =
bucket_allocators are not thread-safe, which means bucket_brigades are =
not, which means that all buckets from the same brigade must only be =
used inside a single thread.

=E2=80=9C=E2=80=A6inside a single thread at a time=E2=80=9D.

The event MPM is an example of this in action.

A connection is handled by an arbitrary thread until that connection =
must poll. At that point it goes back into the pool of connections, and =
when ready is given to another arbitrary thread. In this case the =
threads are handled =E2=80=9Cabove=E2=80=9D the connections, so the =
destruction of a connection doesn=E2=80=99t impact a thread.

> This means for example that, even though mod_http2 manages the pool =
lifetime correctly, it cannot pass a response bucket from a request pool =
in thread A for writing onto the  main connection in thread B, *as long =
as* the response is not complete and thread A is still producing more =
buckets with the same allocator. etc. etc.
>=20
> That is what I mean with not-thread-safe.

In this case you have different allocators, and so must pass the buckets =
over.

Remember that being lock free is a feature, not a bug. As soon as you =
add mutexes you add delay and slow everything down, because the world =
must stop until the condition is fulfilled.

A more efficient way of handling this is to use some kind of IPC so that =
the requests signal the master connection and go =E2=80=9CI=E2=80=99ve =
got data for you=E2=80=9D, after which the requests don=E2=80=99t touch =
that data until the master has said =E2=80=9CI;ve got it, feel free to =
send more=E2=80=9D. That IPC could be a series of mutexes, or a socket =
of some kind. Anything that gets rid of a global lock.

That doesn=E2=80=99t mean request processing must stop dead, that =
request just gets put aside and that thread is free to work on another =
request.

I=E2=80=99m basically describing the event MPM.

Regards,
Graham
=E2=80=94