httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Stein <>
Subject Re: Filtered I/O ... again. :-)
Date Thu, 01 Jun 2000 01:10:28 GMT
On Wed, 31 May 2000 wrote:
> Okay, I am going to sum up this message because it is just easier.  At the
> top and bottom, you argue that it isn't possible to add ordering to the
> hook-based design.

Sorry, I meant that it was not as *convenient* or as *intuitive*. The
hooks take great pains to avoid ordering issues.

On the contrary, the linked-list design is quite explicit about ordering.

> In the middle you argue that the list based design isn't indirectly
> tail-recursive, and even if it is, that won't hurt our performance.  The

Discussed in another thread (pls respond there). I'm unclear on how this
creates a performance problem.

> There is also the argument about writable/non-writable, yes this will be
> harder to implement, but it is relatively easy to implement some
> copy-on-write functions that make this relatively easy for the module
> author.

I agree that this can be done, and even detailed a possible expansion of
your initial proposal to do just this (the filter_vec_item thing).
However, I believe this introduces more complexity into the system than a
simple "write it to the next filter" model.

> > > Your PHP script is not a filter, at least I wouldn't expect it to
> > > be.  Your PHP script should be a generator.
> > 
> > The PHP script *should* be a filter. The generator is the filesystem or
> > the database where I fetch my script. That script is then fed through the
> > PHP filter.
> Fine, but the PHP filter still gets a shot at the re-written data.

I'm unclear on what "re-written data" you are referring to in this
context. Seems like something has been lost.

Are you referring to the header/footer thing?

> > In mod_dav, I can fetch content from arbitrary repositories. However, they
> > cannot be processed by script systems such as PHP unless I drop them into
> > the filesystem first (ugh). If PHP were a filter, then I could pipe the
> > (fetched) script through it. That would be Goodness.
> And that will work with my scheme.

Never said it wouldn't. This was a supporting argument for why I believe
PHP and friends should be filters rather than content-generators.

> > The above code is broken, as I understand PHP's operation. The header text
> > gets shoved out the network, then the SetCookie() is seen. Of course, the
> > SetCookie() fails because we are in the body now and cannot "back up" to
> > send new headers.
> We are arguing about stupid things in this case.  I have already replied
> to Jeff that there are definately ways to add ordering to the hook-based
> scheme.  I am simply not worried about this case because it can easily be
> solved.

No problem. I'm simply trying to explain why I believe that ordering is
very important.

Next time you use a pipe ("|") at the Unix command line, flip the two
commands. :-)  I believe we have the same issue here -- the user demands
some determinism in how things are ordered.

> > > But you imply that the link-based method doesn't have this allocation, and
> > > it does.
> > 
> > Implicitly on the stack. And certainly, it does not have any copying onto
> > the heap.
> I disagree.  I think that while you are actually generating the data to
> replace the tag with you will have to allocate some memory for it, but
> that is an implementation detail in mod_include that can easily be worked
> around for both schemes.

The context was lost here. I meant that the "iovec" is allocated on the
heap in your scheme, but it is implicit in the stack/control of the
link-based model. For example:

  my_filter_callback(...) {
    ap_lputs(next, "hi there", r);
    write_more_stuff(next, r);
    ap_lputs(next, "goodbye", r);

In the above, there is (logically) three iovec elements. The hook-based
approach alloc's them. The link-based implies them.

The other part of my argument was that the link-based scheme can avoid
allocations much more easily. The ap_send_fd() thing was one. Another
would be the following:

  php_print_hex(int v, ap_layer_t *next, request_rec *r)
      char buf[20];

      sprintf(buf, "%08x", v);
      ap_lputs(next, buf, r);

This avoids an alloc for the integer string. I also believe it is easier
to code than trying to insert the resulting allocation into an iovec-like

Note that we could certainly write additional helpers:

      ap_lprintf(next, r, "%08x", v);

> > > This is just not true.  Why does it have to be writable?
> > 
> > Because that is your initial design. You pass a *writable* buffer into the
> > filter hook. That is why you always alloc/copy before calling the filter.
> The key being *initial* not writable.  With the iovecs, this can actually
> be done with some copy-on-write logic, and nothing has to be writable
> until a filter actually wants to modify it.

I complete agree. The above text specifically referred to your "initial
design" as you first posted it. That design required the buffer to be
always-writable (and fixed length, for that matter).

Later on, I described the extensions necessary to make your design more
flexible in terms of readonly vs. read/write.

> > When Filter A creates a new iovec, it cannot know whether Filter B is
> > going to modify the values in the iovec. Therefore, it must presume that
> > all entries are writeable.
> Nope, filter A presumes all are not writable, and filter B is responsible
> for making them writable when it needs them.

In the modified design, yes.

> > Who determines "need to be writeable". That is determined by Filter B.
> > Filter A does not have that knowledge, so it must presume the worst.
> > 
> > The saving grace is to expand the definition/complexity to include tags
> > with each region specified in the iovec. Each filter must then test the
> > tag before it modifies the region in question.

Here is my point where I agree that your basic design can accomodate
readonly vs read/write regions.

After the quick prototype of the filter_vec_item, I'm not sure that I like
it all that much. That structure didn't even account for the *list* of
filter_vec_items that would have to be carried around. Utilities for
appending, inserting, transforming to writable, etc, would be required to
ease the use of the mechanism.

> > Each of those functions is going to need to tear apart and put together
> > new iovec arrays to deal with the output they generate. When you consider
> > that util_func might actually be "php_invoke_script", then it becomes a
> > bit clearer just how much funkiness might be needed w.r.t. that iovec.
> > 
> > In the link-based design:
> > 
> >   my_filter(next, buf, len)
> >      ...
> >      util_func(next, params)
> >      ...
> >      other_func(next, more_params)
> > 
> > Here, each of the utility functions calls ap_lwrite(next, ...) with their
> > information. No need to bubble changes back into an iovec.
> So, you are arguing here that it is easier to do foo(my_changes) than
> iovec[n] = my_changes.  Bah.  I don't buy it.

Yes, that is exactly what I am arguing.

Consider the include_cgi() function in mod_include.c. I will write out the
pseudo-code for the link-based design:

static int include_cgi(ap_layer_t *next, char *s, request_rec *r)
    rr = ap_sub_req_lookup_uri();

    /* scans to the end of the existing layers and appends our next layer
       as the next output stage */
    ap_append_layer(rr, next);


That is about it. When the subrequest does an ap_rwrite(), it will go
through N filters installed by the sub-request itself, then it will go
into the "next" filter, as defined by <this> request.

(where "next" was passed into mod_include's filter callback)

I am not really sure how ap_run_sub_req() is going to get an "iovec"
updated, but would love to see an example :-)

>... tail recursion, performance, unrolling, etc. ...

Discussed in another thread.

> > I'm not saying it is a big step. But it is not automatically understood
> > either. They cannot install their hook at register_hooks() time. They have
> > to figure out when/where and how to do the per-request hook insertion.
> > Even more: they need to figure out the prev/next hook thingy.
> I don't buy this at all.  Module writers already have to understand the
> prev/next hook thingy.  If it is too complex for filters, then it is too
> complex for regular hooks.  If you are worried about having a chance to
> insert the hooks, add another regular hook called register_filter
> hook.  This would be a chance for filters to be registered.

I proposed that "register_filters" hooks in my original proposal.

See my earlier-stated concerns regarding the hook system's prev/next.


Greg Stein,

View raw message