directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Berin Loritsch <blorit...@d-haven.org>
Subject [RT] SEDA Package: Rework Proposal (long, sorry)
Date Wed, 24 Nov 2004 19:08:38 GMT
My initial reaction to what is currently the SEDA package is that in 
order to
do what I want would result in a hack.  I'm not opposed to doing that 
for the
short term, but I do want to look at a sustainable future.  While the 
networking
part isn't the bread and butter of Eve and Kerberos, it does play an 
important
part.  Things like firewalling (dynamic and static), load shedding (for 
protocols
that allow that sort of thing), and load balancing are all network layer 
functions.

After talking to Alex a bit, it seems that he, Trustin, and I are all on 
the same
page that SEDA needs a bit of rework.  The question of course is to do it in
a sane fashion.  That obviously means that we deal with a branch and 
keep the
one interface that everything else is defining and using.  Beyond that 
is a question
of what would make the best approach.

The purpose of this email is to get some dialog going to come up with 
the best
approach overall.  I am stating my thoughts as a starting point, and am not
married to these thoughts.

Alex put together what we have now with the only intention that it be quick
and dirty and be something that could get the project going.  The SEDA 
project
fills that need right now.

I understand Trustin comes to the table with the Netty2 library, of 
which I must
admit ignorance.  He also has been doing most of the work getting SEDA to
work the way it does now.

Now, I come to the table with an existing, well tested library that is 
used by
several projects including Excalibur, some more stuff at D-Haven, and 
several
commercial projects.  That is the D-Haven Event library.

What I would like to propose is that, as much as possible, we avoid NIH 
syndrome
(Not Invented Here) and use stuff that we know works.  I will be the 
first to admit
that the D-Haven Event library as it is now is not a drop in replacement 
for SEDA.
It provides the plumbing work which can be leveraged for the final 
product.  If it
makes sense to use Event and Netty2 together under one umbrelly then we 
should
pursue that.

Nevertheless, here is my understanding of SEDA in a nutshell.  As the 
current project
does right now, each stage listens for events, processes them, and then 
sends new or
processed events on to the next stage.  Everything is decoupled through 
Queues or
Pipes.  Typically each stage will have at least two output pipes: the 
loopback pipe and
the output pipe.  The loopback pipe is used to push events we can't deal 
with right now
out of the way until we can.  The output pipe of course is used to push 
events on to
the next stage.  With this basic structure, we can really do wonders 
pushing events through
the system using enqueue predicates, multicast pipes, load balancing 
pipes, etc.

D-Haven Event handles the core part of moving things through a pipeline 
quite well,
and is pretty well tested.  It comes to the table with the following 
features:

* An event pipeline (pre-matched pairs of a set of sources and an event 
handler)
* A thread manager (using whatever policy you want for pushing events 
through each pipeline)
* A rate limiting enqueue predicate
* A multicast pipe
* Asynchronous command management (including periodically recurring 
commands)

Now, all this is just the skeleton of what makes a SEDA system go.  The 
real power is
in what the stages do and how the pipelines and stages are configured.  
That part is not
done in the D-Haven Event library.  The core set of stages that I see we 
need are as follows:

1. ConnectionManager (this includes firewalling by dropping unallowable 
connections)
2. Reader (start reading bytes from the stream)
   * Router (1 pipe per protocol)
3. Decoder (use the decoder from the protocol handler)
4. RequestHandler (use the request handler from the protocol handler)
5. Encoder (use the encoder from the protocol handler)

6. Writer (start writing bytes to the stream)

The reader will route the ByteBuffers to the proper protocol (based on a 
port mapping
or something like that) which will then do all its necessary steps for 
dealing with its
own type of information, and write the response ByteBuffers to the 
output stream.

Later on we can look at the pipelines to use load-balancing pipelines 
for each protocol
as necessary.  That will allow us the flexibility to merely forward 
requests to other
machines in a DMZ if we need it.

I haven't looked at the details of the rest of the stuff in the SEDA 
package to see how
that would best be served in this architectural view, but I am 
definitely open to suggestions.

-- 

"Programming today is a race between software engineers striving to build bigger and better
idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far,
the Universe is winning."
                - Rich Cook


Mime
View raw message