Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\))
Subject: Re: buckets and connections (long post)
From: Stefan Eissing <stefan.eissing@greenbytes.de>
In-Reply-To: <8C00B305-BECC-4A48-9A03-8A7CA99EADE4@sharp.fm>
Date: Thu, 22 Oct 2015 18:03:26 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <8642583F-79A4-4A8D-A7CF-43CD79029673@greenbytes.de>
References: 
 <CAKQ1sVNchFx=egp=Q0tgT+1GcBXiten8mh9STetcZXOfJxhsjQ@mail.gmail.com>
 <CAKQ1sVPjcA9EYNpjcmM9=AU_LCgAsFntOcZnmGoHymds3S5wmA@mail.gmail.com>
 <AB1691BE05AE7F4992697F2A0835627AC6348972@VOEXM10W.internal.vodafone.com>
 <7B24F4D1-6904-4C00-A337-88F712026BE9@greenbytes.de>
 <0D5B13DA-8642-4662-9606-6B5DF94EB87B@sharp.fm>
 <D70FBC43-2E52-4BDD-ADA5-88623657CBEC@jaguNET.com>
 <4D37F1D6-E2FB-408B-9D82-9F7ABABFFE1E@greenbytes.de>
 <8C00B305-BECC-4A48-9A03-8A7CA99EADE4@sharp.fm>
To: dev@httpd.apache.org


> Am 21.10.2015 um 16:48 schrieb Graham Leggett <minfrin@sharp.fm>:
>=20
> On 21 Oct 2015, at 4:18 PM, Stefan Eissing =
<stefan.eissing@greenbytes.de> wrote:
>> 6. pool buckets are very tricky to optimize, as pool creation/destroy =
is not thread-safe in general
>>  and it depends on how the parent pools and their allocators are set =
up.=20
>>  Early hopes get easily crushed under load.
>=20
> As soon as I see =E2=80=9Cbuckets aren=E2=80=99t thread safe=E2=80=9D =
I read it as =E2=80=9Cbuckets are being misused=E2=80=9D or =E2=80=9Cpool =
lifetimes are being mixed up".
>=20
> Buckets arise from allocators, and you must never try add a bucket =
from one allocator into a brigade sourced from another allocator. For =
example, if you have a bucket allocated from the slave connection, you =
need to copy it into a different bucket allocated from the master =
connection before trying to add it to a master brigade.
>=20
> Buckets are also allocated from pools, and pools have different =
lifetimes depending on what they were created for. If you allocate a =
bucket from the request pool, that bucket will vanish when the request =
pool is destroyed. Buckets can be passed from one pool to another, that =
is what =E2=80=9Csetaside=E2=80=9D means.
>=20
> It is really important to get the pool lifetimes right. Allocate =
something accidentally from the master connection pool on a slave =
connection and it appears to work, because generally the master outlives =
the slave. Until the master is cleaned up first, and suddenly memory =
vanishes unexpectedly in the slave connections - and you crash.
>=20
> There were a number of subtle bugs in the proxy where buckets had been =
allocated from the wrong pool, and all sorts of weirdness ensued. Make =
sure your pool lifetimes are allocated correctly and it will work.

This is all true and correct - as long as all this happens in a single =
thread. If you have multiple threads and create sub pools for each from =
a main pool, each and every create and destroy of these sub-pools, plus =
any action on the main pool must be mutex protected. I found out.=20

Similar with buckets. When you create a bucket in one thread, you may =
not destroy it in another - *while* the bucket_allocator is being used. =
bucket_allocators are not thread-safe, which means bucket_brigades are =
not, which means that all buckets from the same brigade must only be =
used inside a single thread.

This means for example that, even though mod_http2 manages the pool =
lifetime correctly, it cannot pass a response bucket from a request pool =
in thread A for writing onto the  main connection in thread B, *as long =
as* the response is not complete and thread A is still producing more =
buckets with the same allocator. etc. etc.

That is what I mean with not-thread-safe.

//Stefan=