httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Plüm, Rüdiger, Vodafone Group <ruediger.pl...@vodafone.com>
Subject AW: [Patch] Async write completion for the full connection filter stack
Date Tue, 09 Sep 2014 19:19:48 GMT


> -----Ursprüngliche Nachricht-----
> Von: Graham Leggett [mailto:minfrin@sharp.fm]
> Gesendet: Dienstag, 9. September 2014 17:45
> An: dev@httpd.apache.org
> Betreff: Re: [Patch] Async write completion for the full connection
> filter stack
> 
> On 08 Sep 2014, at 8:53 PM, Ruediger Pluem <rpluem@apache.org> wrote:
> 
> > Wouldn't it make more sense instead of using an empty brigade to
> create yet another metabucket that signals write
> > completion? It could also contain information how much data to send
> down the chain for single filters if they e.g. send
> > heap or transient buckets. Otherwise how should they know?
> > If you have a filter that has a large file bucket set aside and it
> does transform it e.g. to a heap bucket during it's
> > processing because it changes data on it I guess it doesn't make sense
> if it does send all stuff once it gets triggered
> > for write completion as we would end up in a blocking write then in
> the core filter. But if it knows how much is left in
> > the core filter buffer it could try to just sent this and avoid thus
> blocking writes. And if there is no room left in
> > the buffer or if what is left is too small for the filter to operate
> on it, the filter could just pass the bucket down
> > the chain and if it would end up in the core output filter, the core
> output filter would just try to write what it has
> > buffered.
> 
> I spent a lot of time going down this path of having a dedicated
> metabucket, and quickly got bogged down in complexity. The key problem
> was "what does a filter actually do when you get one", it was unclear

Don't we have the same problem with an empty brigade? Some filters are not going
to handle it as we expect. Hence the additional logic in ap_pass_brigade.
I guess the minimum behavior we need to get from every filter is to ignore and pass on.

> and it made my head bleed. That makes life hard for module authors and
> that is bad. As I recall there were also broken filters out there that
> only knew about FLUSH and EOS buckets (eg ap_http_chunk_filter()).

We already have additional metabuckets like error buckets or EOR.
So I don't see an issue creating a new one. Any filter not passing a meta bucket
that is does not understand or even try to process it is simply broken. 

> 
> The problem we're trying to solve is one of starvation - no filters can
> set aside data for later (except core via the NULL hack), because there
> is no guarantee that they'll ever be called again later. You have to
> write it now, or potentially write it never. The start of the solution
> is ensure filters aren't starved: if you have data in the output filters
> - and obviously you have no idea which filters have setaside data - you
> need a way to wake them all up. The simplest and least disruptive way is
> to pass them all an empty brigade, job done. We've got precedent for
> this - we've been sending NULL to the core filter to achieve the same

But this is *our* filter and it will not hit any custom filters. So we can
do this kind of hacky game here.

> thing, we want something that works with any filter.

Yes, and this is the reason why I still believe a meta bucket is better.

> 
> The second part of the problem is filters biting off more than they can
> chew. Example: give mod_ssl a 1GB file bucket and mod_ssl won't yield
> until that entire 1GB file has been sent for the reason (now solved)
> above. The next step to enable write completion is to teach filters like
> mod_ssl to yield when handling large quantities of data.
> 
> The core filter has an algorithm to yield, including various checks for
> flow control and sanity with respect to file handles. If a variant of
> this algorithm could be exposed generically and made available to
> critical filters like mod_ssl, we'll crack write completion.

See my other post. I proposed some kind of advisor API that tells a filter
how much it should write to avoid buffering too much and consuming too much
memory and how much it could write to likely avoid a blocking write.
As this will not be always accurate I call it advisor API.

Regards

Rüdiger

Mime
View raw message