Mailing-List: contact new-httpd-help@apache.org; run by ezmlm
Precedence: bulk
Reply-To: new-httpd@apache.org
Date: Sun, 4 Jun 2000 20:03:55 -0700
From: "Life is hard, and then you die" <ronald@innovation.ch>
To: new-httpd@apache.org
Subject: Re: recap of filtered I/O
Message-ID: <20000604200354.A2219@innovation.ch>
Mail-Followup-To: new-httpd@apache.org
References: <20000604153810.D1800@innovation.ch>
 <Pine.LNX.4.10.10006041541430.29036-100000@nebula.lyra.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
In-Reply-To: <Pine.LNX.4.10.10006041541430.29036-100000@nebula.lyra.org>;
 from gstein@lyra.org on Sun, Jun 04, 2000 at 04:16:21PM -0700

On Sun, Jun 04, 2000 at 04:16:21PM -0700, Greg Stein wrote:
> On Sun, 4 Jun 2000, Life is hard, and then you die wrote:
> >...
> > On Sat, Jun 03, 2000 at 02:26:16PM -0700, Greg Stein wrote:
> >...
> > > *) how would my insert-100Mb-filter be written?
> > 
> > I Ryan addressed this point with the filter being able to signal that
> > it hasn't sent all yet.
> 
> In the hook-based scheme, the hook is only called when the
> content-generator writes some output (e.g. via ap_rwrite()). If the filter
> isn't going to write all the data at that point, then when will it?
> 
> content-generator:
>     ap_rwrite("foobar<!--#giant-data -->bleek");
>     ap_rwrite("more stuff");
> 
> That 100Meg needs to go out before the "bleek" can be written. Either, it
> gets completely dumped into an iovec[], or the hook calling will look
> like:
> 
> ap_rwrite:
>     iovec[0] = (buf, len)
>     run_hook(iovec, &more_to_write)
>     ap_bwrite(iovec)
>     while more_to_write:
>         iovec = empty
>         run_hook(iovec, &more_to_write)
>         ap_bwrite(iovec)
> 
> That just seems a bit complicated :-)
> 
> I'm not sure, but you might be confusing the "I have more to write" with
> the concept of "I haven't processed all the input yet."
[snip]

Hmm, yes, I guess I was confusing the two. And yes, the above was about
what I had thought would be done. Pretty? No.

> [ both schemes can handle the "wait for the rest of the input" scenario; I
>   believe that the link-based approach is a bit easier, it that it
>   provides an explicit context pointer for maintaining this data; the
>   hook-based approach requires the filter to attach "user data" to the
>   request (which then prevents "SetFilters SSI PHP SSI" because there is
>   only one "SSI user data slot" ]

Agreed - using the stack to hold the context is definitely easier.
However, even the layer approach needs to attach user data to the
request to keep state accross function calls (e.g. mod_include needs to
remember the "<!--#inc").

> > > *) if the filter wants to insert a file, then how do we get the FD
> > >    returned to Apache so that it can use sendfile/TransmitFile?
> > 
> > This is something that's not clear to me in *either* scheme. Can you
> > elaborate how this should be done using layers?
> 
> Quite easily :-)
> 
>   mod_include::filter_callback(this_layer, buf, len):
>      ...
>      ap_lwrite(this_layer->next, plain_text)
>      ... oh! found a #include ...
>      ap_lsend_fd(this_layer->next, fd)
>      ...
>      ap_lwrite(this_layer->next, plain_text)
> 
> Under the covers, ap_lsend_fd() looks like:
> 
>   ap_lsend_fd(next_layer, fd):
>      if next_layer == NULL:
>         ap_send_fd(fd)
>      else:
>         mmap_thingy map = mmap_file(fd)
>         invoke_callback(next_layer, map.buf, map.len)

Eeek! The problem with this scheme is that ap_send_fd will *never* be
used for any HTTP/1.1 responses, assuming chunking will be a layer.
Also, somebody just wants to add a header and trailer - bam, mmap
instead of sendfile. No, there needs to be more intelligence here,
along the lines of what Roy suggested: typed buffers (char*, fd, sd,
etc), with ways to read from those if necessary.

> > > *) flow control
> > 
> > I don't see this as really any different from the layer approach. But
> > I'm assuming that a filter/content-generator that creates large output
> > will do so in parts (third point above), and only smaller stuff will
> > slurp all up and send it in one write.
> 
> A hook-based filter never generates a block-on-the-network event. It
> stuffs everything into the iovec[], regardless of what is happening with
> the network.
[snip]

Like I said, I was assuming partial writes from above. If those don't
exist, then yes, I don't see the flow-control either.

> > > *) how would "SetFilter SSI PHP" be implemented in the hook scheme?
> > >    (note this implies a table mapping names to functions; also, I believe
> > >     that solving this directive for the hook-based scheme will essentially
> > >     look just like the link-based scheme)
> > 
> > I think this is a minor point. The hook scheme allows you to order a specific
> > filter after another one. Yes, it essentially degrades to a linked-list.
> 
> Actually, I don't believe it is all that minor. If the typical behavior is
> to order them, and this degrades to a linked-list, then why are we
> bothering with the hook-based scheme and its iovec complexity? (among my
> other concerns raised in the recap)

I see two issues here which are getting confused: one is how to determine and
setup the list of filters/layers, the other is whether to use filters or
layers. These are independent AKAICT. Even the hook scheme eventually produces
an ordered list, so question is just how to produce that list. Hmm, I guess
I've also been assuming that the hooks stuff was just to get a list of filters,
and ap_run_hook wouldn't actually be used,

> One other item here:
> 
> In both schemes, we call the (new) "install_filters" hook on the modules.
> In the hook-based scheme, this inserts per-request hooks. In the
> link-based scheme, this adds to a linked-list of layers.
> 
> One of the things about the hook-based scheme is the notion of "we don't
> care who registered in the per-request chain... we'll just call them; they
> might have ordered themselves." Well, this "opaqueness" of the callbacks
> totally disappears when you add support for the "SetFilter" directive.
> Why? Because modules are no longer adding private functions into the
> per-request chain. An external mechanism must find the function pointers
> and insert them appropriately.

Ok, I see your point. However, we need both things: "external" configuration
by the core (the "SetFilter"), and "internal" configuration by each module
(mod_auth_digest parses the request headers and figures out from that
whether it needs to add a layer, similarly for content-encoders,
transfer-encoders, content-md5, etc).

I guess the way I see it is that during the post_read_request phase the
modules register which filters/layers to use (along with the filter
type), and the core then orders the filters first according to type and
then according to additional directives such as "SetFilter". So in some
sense the opaqueness is still there - the filter modules don't care
about the order, as that's determined by something external to them (the
core).

> > OTOH, I've been thinking how this would integrate with Dean's proposal to
> > have a dedicated process/thread which serves slow clients (via some
> > select/poll based pseudo-asynch-io) so as to free up the processes/threads
> > for doing the content-generation/filtering work as quickly as possible,
> > which I think is a really cool idea.  With the iovec's it seems more
> > straightforward at first glance, because it would get all the data it
> > needs in one go. However, as soon as the filters can produce partial
> > output this reverts to what I'm guessing will be needed for the layer
> > scheme: a largish buffer into which the stuff is assembled (i.e copied),
> > which the slow-client-process can then dribble to the client at will. So
> > I'm not sure I see any advantage here using iovec's either. However, I
> > suspect I may be missing something, because I also don't see how sendfile
> > integrates into these schemes - I'm starting to think Roy's
> > bucket-brigades may be the way to go after all.
> 
> I do not believe you're missing anything either. In both schemes, they can
> take the (filtered) network output and pass that off to the delivery
> thread. If the response is fully generated and passed off, then the worker
> thread can start on another request/response. Both schemes could do this
> invisibly w.r.t. the filters.
> 
> There is no advantage to either scheme in terms of the delivery thread,
> presuming the "iovec" in the hook-based scheme is made a bit more complex.
> 
> I do believe the link-based approach has some simplicity advantages. The
> ability to pass a file descriptor to the network layer, for example. The
> network layer can pass that fd off to the delivery thread, which can then
> use sendfile() on it. In the hook-based system, when we say "iovec", we
> really mean something like Roy's bucket brigades -- each element would be
> typed as a buffer, as a file descriptor, or whatever. IMO, introducing the
> buckets into the system is more complex than different types of writes
> (e.g. ap_lwrite, ap_lsend_fd, ap_lwritev)

IMHO, the different types of write isn't sufficient - see above. Btw.,
viz assembling the data into a buffer for transmission, I suppose for
efficiency that could be assembled into an iovec instead, so the layer
approach need not be worse off in this respect (this implies that
modules aren't allowed to reuse their send buffers until after the
request has been sent, though).


  Cheers,

  Ronald