httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manoj Kasichainula <>
Subject Re: My proposal for buckets/filtering in 3.0.
Date Tue, 15 Aug 2000 08:36:33 GMT
On Mon, Aug 14, 2000 at 01:52:05PM -0700, Roy T. Fielding wrote:
> >Most everything else is a bucket in this case. Actually, they are
> >probably better described as matrushka dolls or those little nested
> >plastic barrels. In fact, I'll call them barrels to increase
> >confusion.
> Hah, cute

That's what I was trying for!

> but how is it different than our current subrequests?

Just extends and abuses the concept, I guess. Originally, the whole
concept of ap_rwrite was taken out (though see below, maybe it gets
put back), which was a difference.

I'll admit that I've never had a chance to get intimately familiar
with how subrequests work, so I've undoubtedly missed some nice
features and reinvented some as well.

> >When anything creates a barrel, it passes in:
> >
> >- a pool to allocate memory from. The stuff read from the barrel must
> >  be in the scope of that pool.
> >- a list of capabilities that are required, and a list of preferred
> >  capabilities. capabilities include: "send-from-file-descriptor",
> >  "send from memory", "write-to-content", and so on.
> >
> >  This feels like content negotation, which scares me. But, very
> >  little of it is actually necessary in a first pass. A single
> >  "memory-block" capability is all that's mandatory, really.
> >  Everything else is extra features and optimization.
> Hmmm, sorry, I think I've heard that phrase one too many times this
> past week.  Figure out what the architectural context is -- all of the
> forces that will impact this design in terms of the application needs.
> When you have covered all of that, everything else is extra features
> and optimization.  Things like single-copy IO and sendfile support are
> not optimizations -- they are the requirements that motivate our next
> generation architecture.

Oh, definitely. My point was that the very first implementation
wouldn't need to actually implement non-blocking I/O and sendfile
support. But, it's important that the infrastructure should be in
place from the very beginning. Also, if a third-party wants to
implement a module, it would be good if they aren't forced to deal
with sendfile if they don't feel like it.

I think necessary preconditions for including this, or any new
filtering scheme in 3.0 are that it deals with non-blocking I/O and
sendfile, and supports near-zero-copy, while hopefully not requiring
every possible module to support these features. Async I/O would be
nice as well, though I don't see this being possible with fancy
content handlers.

> >The barrel bring created will then attempt to meet those wishes if
> >possible, or return an error.
> Hmmm, that sounds like magic to me.  The whole point of bucket brigades
> was to specify that magic in a way that can be standard for all modules.

Well, it's a simple attempt at negotiation. If a consumer says that it
requires sendfile support from a barrel, and the barrel can only
return blocks of memory, then there has to be an error.

I think a simple way to avoid the error case is to specify a minimum
set of standards that all barrels must provide and that all consumers
can deal with.  For example, there must be support for a
thread-per-request mode passing around copy-on-write memory buffers,

I see the negotation looking something like:

actual_features = ssi->create_barrels(WANT_SENDFILE, WANT_NON_BLOCKING);

Tony Finch suggested that the feature request isn't really useful, and
that these barrels should just list the things that they are capable
of doing. Then there's no negotiation at all. I'm starting to agree
with him, and that makes me much happier. But if this behavior ends up
being necessary, it is available. And I think it can be useful to
avoid annoying conversions from one bucket-type to another. Hmm, but
maybe that conversion will always take place and there's no point
trying to pass along lots of hints to make sure that it happens early.

> >- non-blocking I/O can be a capability of a barrel. One API of the
> >  barrel would be to return the file descriptor (or "event") it is
> >  waiting for, so that it can be selected on, and a full event-based
> >  server should be possible. When some barrel three levels deep
> >  doesn't support non-blocking I/O, the request handler can decide to
> >  punt to a seperate thread. This way, different programming models
> >  for modules can be supported. 
> It does mean that something has to read from the barrel and write to
> the network, right?  Or is this a model where we give the network to
> the barrel and it writes?  Kind of hard top manage the latter.

I thought about this after writing the message. You've hit possible
answers spot on:

- barrels, when asked for data, basically return buckets :) Then the
  protocol handler writes them out. Hmmm, maybe this doesn't replace
  bucket brigades after all. But there are basic ways of getting data
  to the network, and maybe a set of simple buckets (that, because of
  the rest of the infrastructure outlined above, still avoid lifetime
  and other hairy issues) is the best way to do this.

- barrels know how to talk to filters, using sendfile or write()

These alternatives are not that different from each other, and I think
they won't look too dissimilar from each other. I was leaning towards
the latter before, but I'm starting to change my mind. 

So, if we eliminate negotitation, bring back buckets (w/ brigades as
an optimization), does this proposal sound more reasonable?

> >- Writability would be another capability. If a barrel is writable,
> >  that URI is available to DAV.
> Yes, all source resources should be available to DAV.  The way to do that
> is to asign them URI and pass their identifiers as metadata.  Let the
> protocol filter decide what to do with that information.

Yeah, I think you're right.

> >- This scheme allows not just chains of barrels, but trees. So half of
> >  a document can come from PHP and half can come from SSI. You just
> >  need a container barrel that knows what parts are interpreted by
> >  what modules. There could be an MS-Word barrel that munches on
> >  multiple HTTP requests for HTML + images. This scares me.
> It should.  Keep in mind that all of the sources would have to pass
> through the access control steps.

Yes. There is a file-access barrel.  That barrel is essentially the
filesystem abstraction that Greg (is|wants to be|has been) working
on, and it would implement all access controls for file resources.

The URI barrel would implement Location-based access control.

> >- Because of the content-negotiation features, a barrel can always
> >  figure out what the optimal format for sending content back should
> >  be.
> You mean every barrel will have to know everything about the request,
> including things like HTTP negotiation?  Yikes.

OK, I was extremely unclear here. I was referring to the negotiation
that takes place between a barrel and its consumer as to how the data
is passed.  It's kinda sorta like content negotiation, but is very
different, so I was just wrong. :) I should stop writing these things
at 5am.

> >- Should allow "magic cache" like was discussed back at the June/July
> >  '98 meeting. In fact, it should be really easy.
> Easy to identify cacheable items, yes, but how easy is it for the
> cache manager to manage overall allocations and reap old entries?

Well, everything managed by the cache is in the cache's scope and the
cache's pool. Now, a pool in this proposal isn't quite the same as it
is in 2.0. It provides the same APIs as current pools, but this pool
can also provide a free() call so that the cache manager can free an
object long before the cache's pool is cleaned up. How the pool is
implemented underneath probably would look more like malloc/free.

View raw message