httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Stein <>
Subject Re: recap of filtered I/O
Date Mon, 05 Jun 2000 06:11:27 GMT
On Sun, 4 Jun 2000, Life is hard, and then you die wrote:
> On Sun, Jun 04, 2000 at 04:16:21PM -0700, Greg Stein wrote:
> > [ both schemes can handle the "wait for the rest of the input" scenario; I
> >   believe that the link-based approach is a bit easier, it that it
> >   provides an explicit context pointer for maintaining this data; the
> >   hook-based approach requires the filter to attach "user data" to the
> >   request (which then prevents "SetFilters SSI PHP SSI" because there is
> >   only one "SSI user data slot" ]
> Agreed - using the stack to hold the context is definitely easier.
> However, even the layer approach needs to attach user data to the
> request to keep state accross function calls (e.g. mod_include needs to
> remember the "<!--#inc").

The layer structure looks like:

  struct ap_layer_t {
      ap_layer_func_t *callback;
      void *context;
      struct ap_layer_t *next;
      request_rec *r;

Each "installed" layer thus has its own context for retaining state across
invocations. Thus, if you have a particular layer installed more than
once, then you have matching context pointers for each one.

In the "store it in the request 'user data'" case, you can only store it
once. If you will: it is a global, with the associated single-use

> > > > *) if the filter wants to insert a file, then how do we get the FD
> > > >    returned to Apache so that it can use sendfile/TransmitFile?
> > > 
> > > This is something that's not clear to me in *either* scheme. Can you
> > > elaborate how this should be done using layers?
> > 
> > Quite easily :-)
> > 
> >   mod_include::filter_callback(this_layer, buf, len):
> >      ...
> >      ap_lwrite(this_layer->next, plain_text)
> >      ... oh! found a #include ...
> >      ap_lsend_fd(this_layer->next, fd)
> >      ...
> >      ap_lwrite(this_layer->next, plain_text)
> > 
> > Under the covers, ap_lsend_fd() looks like:
> > 
> >   ap_lsend_fd(next_layer, fd):
> >      if next_layer == NULL:
> >         ap_send_fd(fd)
> >      else:
> >         mmap_thingy map = mmap_file(fd)
> >         invoke_callback(next_layer, map.buf, map.len)
> Eeek! The problem with this scheme is that ap_send_fd will *never* be
> used for any HTTP/1.1 responses, assuming chunking will be a layer.
> Also, somebody just wants to add a header and trailer - bam, mmap
> instead of sendfile. No, there needs to be more intelligence here,
> along the lines of what Roy suggested: typed buffers (char*, fd, sd,
> etc), with ways to read from those if necessary.


Typed buffers would offer more possibilities of optimization here. This
does start to look a bit like the complex "iovec" thingy from the
hook-based scheme. However, [compared to the hook scheme] the control flow
makes it a bit simpler -- you accept an iovec and deal with it accordingly
(wrap some chunking around it, prepend/append text, etc). I don't like
this as much due to the complexity, but for serious perf issues it may be
necessary. In the hook-based scheme, these iovecs are in/out parameters,
which make them a bit harder to work with.

Why harder? Well, consider the following:

  my_layer_callback(layer, items[]):
    ap_lwrite(layer, header_text)
    ap_lwritev(layer, items)
    ap_lwrite(layer, trailer_text)


    new_items = alloc(len(items) + 2)
    new_items[0] = ap_pstrdup(header_text)
    memcpy(&new_items[1], items, some_size)
    new_items[len(items)-1] = ap_pstrdup(trailer_text)
    *items = new_items

eesh... :-)

(and note the allocs in there)

> > > > *) how would "SetFilter SSI PHP" be implemented in the hook scheme?
> > > >    (note this implies a table mapping names to functions; also, I believe
> > > >     that solving this directive for the hook-based scheme will essentially
> > > >     look just like the link-based scheme)
> > > 
> > > I think this is a minor point. The hook scheme allows you to order a specific
> > > filter after another one. Yes, it essentially degrades to a linked-list.
> > 
> > Actually, I don't believe it is all that minor. If the typical behavior is
> > to order them, and this degrades to a linked-list, then why are we
> > bothering with the hook-based scheme and its iovec complexity? (among my
> > other concerns raised in the recap)
> I see two issues here which are getting confused: one is how to determine and
> setup the list of filters/layers,

I think we're pretty well in agreement that both schemes can handle the
determination and setup. IMO, it is a bit simpler for the link-based
mechanism, but I do agree that it is possible to write covers (and etc)
for the hook scheme to keep things reasonably simple for the user and for
most module writers.

> the other is whether to use filters or
> layers. These are independent AKAICT. Even the hook scheme eventually produces
> an ordered list, so question is just how to produce that list. Hmm, I guess
> I've also been assuming that the hooks stuff was just to get a list of filters,
> and ap_run_hook wouldn't actually be used,

As far as I know, the design has been to use ap_run_hook. If you don't use
that, then you really *do* have something like the linked-list scheme.

> Ok, I see your point. However, we need both things: "external" configuration
> by the core (the "SetFilter"), and "internal" configuration by each module
> (mod_auth_digest parses the request headers and figures out from that
> whether it needs to add a layer, similarly for content-encoders,
> transfer-encoders, content-md5, etc).

Agreed. Modules can always insert themselves manually by virtue of the
(new) "install_filters" hook. External direcives can also insert and order

> I guess the way I see it is that during the post_read_request phase the
> modules register which filters/layers to use (along with the filter

In both schemes, we plan to introduce a "install_filters" hook that runs
just before the "handler" phase. This hook allows the modules to examine

the request/response and base a filter-insertion on the values


Greg Stein,

View raw message