apr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Stein <gst...@lyra.org>
Subject Re: [RFC] Network Abstraction Layer
Date Fri, 02 Mar 2001 11:41:38 GMT
On Fri, Mar 02, 2001 at 09:16:27PM +1100, Luke Kenneth Casson Leighton wrote:
> On Fri, 2 Mar 2001, Greg Stein wrote:
> >   BRIGADE = { FILE, EOS }    # FILE bucket and EOS (End Of Stream) bucket
> > 
> > becomes
> > 
> >   BRIGADE = { "packet header bytes", FILE, EOS }
> > 
> > We inserted a header without touching the file. The output filter generates
> > a sendfile which contains an iovec for those header bytes. Blam! Out the
> > network it goes... :-)
> so, one bucket can deal with the NetBIOS header.

Careful. We may be getting some terminology mixed up here. I think we're
definitely on the same page :-), but I'd like to clarify...

*) in the above example, the brigade has a HEAP or POOL or some other kind
   of simple bucket inserted as the first element. It refers to some memory
   with the appropriate data.

*) it also has a FILE bucket and an EOS bucket

*) a filter inserted the bucket into the brigade

*) in the Samba case, I suspect your NetBIOS header is just another HEAP
   bucket, inserted by a NetBIOS *filter*

> one with the SMB header.
> one with the IPC$ layer.
> one with the SMBtrans layer.
> one with the DCE/RPC pipe layer.
> one with the DCE/RPC header layer.
> one with the DCE/RPC data.

Again, I would think *filters* are inserting standard bucket types into the
brigade. You wouldn't necessarily need to create new bucket types for each
of the above concepts. You could definitely create filters for each one,

But I do see a great correlation in the models!

> ... we still need an apr_bucket_NAL which can "bounce" in-and-out of
> buckets, _even though it may actually be implemented as buckets itself,
> underneath!_ and handle the possibility that on one particular OS - e.g.
> NT - is in fact a call to a kernel-level function, e.g. CreateNamedPipe().

Hmm. I'm thinking that a filter would just choose the right kind of bucket
to insert into the brigade. It could insert a SOCKET bucket, a PIPE bucket,
or it could read data from the OS over an OS-specific API and then insert a
HEAP bucket pointing to that memory.

Another approach is a custom bucket that does the OS-specific read. The
filter would then simply insert the custom bucket rather than do the
read-into-HEAP-bucket stuff.

> or even, later on, let's say that someone decides to provide a linux
> kernel-level SMB layer or even a NetBIOS kernel API.

Yup. When this happens, I'd say there are a couple choices:

1) write a new bucket type. your "read from network" filter could be
   configured to insert the new bucket type, rather than a standard type.

2) write a new filter which is inserted as the "read from network" filter.
   it does the job of talking to the new kernel API, and populating the
   brigade with standard bucket types containing the data.

3) a combination of the above two: rather than a configurable filter, you
   simply insert a new filter which always uses the new custom bucket type.

> as you can see, seven levels of buckets like that - all of which a

Shouldn't this be seven levels of filters?

Hmm. I can also see a case where you have a bucket that implements its
read() function by reading from another bucket, performing something on that
data, and returning it. In this sense, the chaining of buckets would be
similar to a chain of filters.

However, I'd think the filters are a slightly better approach, as they
operate on a brigade level rather than a single bucket level. Let's say that
you have a low-level bucket that reads from the network and you get a
partial packet. What should the upper level bucket do? Where should it put
that data while it asks for more data? In the brigade approach, the various
bits of data are accumulated into a brigade until "enough" is available.

> programmer will need to be "aware" of - is exactly what makes me very,
> very wary of introducing "newbie" programmers - no matter _how_ skilled or
> advanced or experienced they are - into this kind of environment.

Agreed. This is why we all try to encapsulate processing between tightly
controlled APIs. Nice little black boxes :-)

In the Apache case, each little black box is a filter. It gets a brigade,
it operates on that brigade, then it passes it. It doesn't have to know
where the brigade came from, or who is next in line. It usually doesn't have
to know anything about the bucket types (just that it can read()).

Then, some "higher level magic" organizes the filters into a useful
sequence, and puts the right data into the filter stack.

> if we can split things out so that an individual programmer only has to
> deal with *at most* two layers - the one they're coming from and the one
> they're going to - i'll be well-happy.

Get a brigade. Muck with it. Send it.   ... nothing more :-)

> and to do _that_, in TNG, i decided to split at the SMBtrans / DCE/RPC
> pipe layer with a unix-domain-socket.  and on NT, that is going to be
> CreateNamedPipe() that will do that for us, bypassing the
> SMBtrans-and-below layers altogether.
> and still get good code reuse / abstraction.
> what you think?

It sounds great! I'm encouraged by your interest in the filters, buckets,
and brigades. It seems to be a matter of some details at the endpoints of
the filter stack, to determine whether the custom socket is a bucket vs a
filter, but that should be straightforward.

Something that I just thought of. Apache operates its I/O in two modes:

1) it "pulls" data from an input filter stack

[ processes it ]

2) it "pushes" data into the output filter stack

I could see a case where the following could be set up:

              /    \

The driver sits in a loop, pulling content (a brigade) from (replaceable)
input function, and then simply shoves that brigade into a processing
filter. Those filters do all the needed work and just keep moving stuff down
the line. Eventually, it drops back to the (same) network in the

IOW, maybe there isn't an "input filter" so much as a replaceable function
to get brigades of data. All processing is just a matter of shoving the
brigades at an "output" filter stack. But those filters would do things like
decrypt the request, grab a file from the filesystem based on the request,

Not really sure. Just blue-skying a thought here. It would get complicated a
bit by certain authentication protocols that need to speak to the output,
but not run through the filter stack. Ah well...
[ in the Apache case, the filters in the stack are "instantiated" with
  references to the current connection and request objects. conceivably,
  Samba would do a similar thing, and it could use the connection object to
  speak back to the client. ]

Oh. Silly me. You could set it up such that the processing filters simply
wouldn't be added until after the authentication occurs. The auth filter
would simply pass bits-for-the-client to the next filter. More bits will
arrive at the auth filter, it would verify the auth, and trigger the
insertion of some processing filters.

Hoo boy. This is where I step out and let you guys deal with it :-). Just
ask about Apache's filters, the buckets and brigades, and I (and others
here!) can handle that... :-)


Greg Stein, http://www.lyra.org/

View raw message