Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
From: =?iso-8859-1?Q?Pl=FCm=2C_R=FCdiger=2C_Vodafone_Group?=
	<ruediger.pluem@vodafone.com>
To: "dev@httpd.apache.org" <dev@httpd.apache.org>
Subject: AW: [Patch] Async write completion for the full connection filter
 stack
Thread-Topic: [Patch] Async write completion for the full connection filter
 stack
Thread-Index: 
 AQHPywTEYoJdm0/EIEupsHVqM77tu5v3IDYAgAAbqQCAAAZmAIAAMo4AgAFdlwCAAFp7kA==
Date: Tue, 9 Sep 2014 19:19:48 +0000
Message-ID: 
 <AB1691BE05AE7F4992697F2A0835627AC5D859AB@VOEXM10W.internal.vodafone.com>
References: <A57B894B-50DA-4252-8FFF-C3DEF8473A61@sharp.fm>
 <D4B68B53-592F-4EEC-B39F-EFC9887F60D5@jaguNET.com>
 <715A3AE8-8015-4537-943F-A34E0E2A76F7@sharp.fm>
 <D7EA895A-9C37-4387-AE89-C1B7DF9B0706@jaguNET.com>
 <540DFB30.6050700@apache.org> <2F3B060C-E383-407F-A24C-7F026AD3CA81@sharp.fm>
In-Reply-To: <2F3B060C-E383-407F-A24C-7F026AD3CA81@sharp.fm>
Accept-Language: en-US
Content-Language: de-DE
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


> -----Urspr=FCngliche Nachricht-----
> Von: Graham Leggett [mailto:minfrin@sharp.fm]
> Gesendet: Dienstag, 9. September 2014 17:45
> An: dev@httpd.apache.org
> Betreff: Re: [Patch] Async write completion for the full connection
> filter stack
>=20
> On 08 Sep 2014, at 8:53 PM, Ruediger Pluem <rpluem@apache.org> wrote:
>=20
> > Wouldn't it make more sense instead of using an empty brigade to
> create yet another metabucket that signals write
> > completion? It could also contain information how much data to send
> down the chain for single filters if they e.g. send
> > heap or transient buckets. Otherwise how should they know?
> > If you have a filter that has a large file bucket set aside and it
> does transform it e.g. to a heap bucket during it's
> > processing because it changes data on it I guess it doesn't make sense
> if it does send all stuff once it gets triggered
> > for write completion as we would end up in a blocking write then in
> the core filter. But if it knows how much is left in
> > the core filter buffer it could try to just sent this and avoid thus
> blocking writes. And if there is no room left in
> > the buffer or if what is left is too small for the filter to operate
> on it, the filter could just pass the bucket down
> > the chain and if it would end up in the core output filter, the core
> output filter would just try to write what it has
> > buffered.
>=20
> I spent a lot of time going down this path of having a dedicated
> metabucket, and quickly got bogged down in complexity. The key problem
> was "what does a filter actually do when you get one", it was unclear

Don't we have the same problem with an empty brigade? Some filters are not =
going
to handle it as we expect. Hence the additional logic in ap_pass_brigade.
I guess the minimum behavior we need to get from every filter is to ignore =
and pass on.

> and it made my head bleed. That makes life hard for module authors and
> that is bad. As I recall there were also broken filters out there that
> only knew about FLUSH and EOS buckets (eg ap_http_chunk_filter()).

We already have additional metabuckets like error buckets or EOR.
So I don't see an issue creating a new one. Any filter not passing a meta b=
ucket
that is does not understand or even try to process it is simply broken.=20

>=20
> The problem we're trying to solve is one of starvation - no filters can
> set aside data for later (except core via the NULL hack), because there
> is no guarantee that they'll ever be called again later. You have to
> write it now, or potentially write it never. The start of the solution
> is ensure filters aren't starved: if you have data in the output filters
> - and obviously you have no idea which filters have setaside data - you
> need a way to wake them all up. The simplest and least disruptive way is
> to pass them all an empty brigade, job done. We've got precedent for
> this - we've been sending NULL to the core filter to achieve the same

But this is *our* filter and it will not hit any custom filters. So we can
do this kind of hacky game here.

> thing, we want something that works with any filter.

Yes, and this is the reason why I still believe a meta bucket is better.

>=20
> The second part of the problem is filters biting off more than they can
> chew. Example: give mod_ssl a 1GB file bucket and mod_ssl won't yield
> until that entire 1GB file has been sent for the reason (now solved)
> above. The next step to enable write completion is to teach filters like
> mod_ssl to yield when handling large quantities of data.
>=20
> The core filter has an algorithm to yield, including various checks for
> flow control and sanity with respect to file handles. If a variant of
> this algorithm could be exposed generically and made available to
> critical filters like mod_ssl, we'll crack write completion.

See my other post. I proposed some kind of advisor API that tells a filter
how much it should write to avoid buffering too much and consuming too much
memory and how much it could write to likely avoid a blocking write.
As this will not be always accurate I call it advisor API.

Regards

R=FCdiger