directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Karasulu <aok...@bellsouth.net>
Subject Re: [seda] A preview on redesigned SEDA (or Netty?)
Date Thu, 09 Dec 2004 05:34:01 GMT
Richard Wallace wrote:

> Alex Karasulu wrote:
>
>> Richard Wallace wrote:
>>
>>> Alex Karasulu wrote:
>>
>>
<snip/>

>>
>> Actually the parse part is the "decode" stage.  The stateful 
>> (chunking) encoder decoder pair will do this for you.  This is why 
>> the ProtocolProvider has a getEncoderFacotry() and a 
>> getDecoderFactory() method defined.  Regulating these operations with 
>> a stage makes sense under load.  Sometimes you'll have encodings that 
>> are CPU intensive and so you need a higher ratio of threads to load.  
>> SEDA is excellent for that.  Somethimes the encoding is a joke.  In 
>> this case I'd like to make it so the provider could somehow tell SEDA 
>> to skip encoding or decoding stages to not suffer synchronization 
>> penalties and latency costs.
>>
>
> Oh I see. I hadn't seen the getEncoderFactory() and 
> getDecoderFactory() before.  I agree that in the decoding stage you 
> could do the parsing part and in most scenarios that is probably what 
> you want to do.  But I can see cases where you may want to have more 
> stages for doing things like (en|de)cryption, (un)compression, etc.  
> But then, for those that want/need something like this, they can have 
> their encoder/decoder factories build little sub-pipelines that do that.
>
Right exactly.  The Kerberos server may have to do this because it 
encodes PDUs inside encrypted ASN.1 PDUs.  Hairy little protocol ain't 
it.  I'm sure there are several exceptions like this that make you point 
valid.

<snip/>

>>
>> Funny you say this Trustin and I were talking last night about this.  
>> I think he rights some bullet proof code and his abstractions for 
>> handing non-blocking IO are really really nice.  But this is not 
>> SEDA.  Berin touched on this too and talked about the possibility of 
>> a hybrid.  Perhaps we just need more than one choice or just that a 
>> hybrid that uses one over the other until certail thresholds are 
>> reached.
>
>
> Now that really would be awesome!  The tough thing is figuring out 
> those thresholds.  But that's a bit OT right now.
>
That's fun testing and sysadmin tuning stuff.  Love to get there some day.

<snip/>

>> Exactly! Well said.  Take a look at the CoR stuff and the 
>> commons-pipeline effort over at Jakarta.  They have been doing 
>> similar things perhaps better for network specific and general 
>> applications that are processing anything like images in the example 
>> you gave above.  Please let us know what you think.  Your grip on 
>> SEDA and its merits is pretty solid and well I have been trying to 
>> understand the SEDA commons-pipline connection to no avail.  Looking 
>> forward to your impressions and  comments....
>
<snip/>

> I'm not sure what you mean by a connection between commons-pipeline 
> and SEDA.  The concepts are fundamentally the same.  There is are 
> pipelines and within pipelines are stages and the stages processing is 
> controlled by what in c-p is a "StageDriver" and in SEDA is a 
> Controller with a thread pool.  Right now it looks like c-p only has a 
> SingleThreadStageDriver, but it wouldn't be difficult to add a 
> StageDriver that uses a thread pool, I don't think.

Sounds very similar to what we have.

> The biggest feature that they are currently missing that is in the 
> directory seda implementation is event routing.  

Funny you and Trustin both see that as a strength.  Trustin sees that as 
a weakness.  I can go both ways.

> From what I've seen in the directory seda code an Event can be routed 
> to one or more (? it looks like right now DefaultEventRouter throws an 
> UnsupportedOperationException, on line 148, if this situation occurs, 
> but the framework is there to support such a thing) different stages. 
> Right now the c-p pipeline just passes events from one stage to the 
> next in the order that the stages were added.  There is support for 
> doing a branch to another pipeline, though it's not clear when this 
> branch is taken.
>
> I think the framework in directory is much more complete and usable.  

Yeah I think its really easy to use because the complex rarly used 
concepts are left out for you to implement.  You only need a little bit 
of SEDA concepts to wreak 90% of its power.  I've always wanted to keep 
it simple:

  Stage = ThreadPool + Queue + EnquePredicates + (Work) Driver

It's that simple to me.  Also the event router / bus pattern which I 
stole from Berin I might add decouples stages.  This is excellent for 
dyamically assembing stages or creating stages that process different 
aspects of an event that need not synchronize by routing them to more 
than one stage.  There's lots of fancy things we could do but until we 
get a chance to test out the basic with real protocols we'll never know 
for sure.

One of my main goals is to get something really simple writen to test.  
Then take that and build a real protocol server I can bang against.  
Until then everything sounds good in theory.  So I built this simple 
SEDA thing and it works but may not be anything but an experiment.  
That's why I was not against having multiple implementations going on at 
the same time in the seda project who knows what we're going to discover.

> If it turns out that I can't use it for the sub-pipelines I need that 
> we talked about before then I'll probably just use Berin's d-haven 
> event package to build my own pipeline for processing.  Might not be 
> pretty but I think I could make something that would work for my 
> purposes.  I don't think I'll need to do that tho, not from what I've 
> seen so far.

Yeah I think Berin's stuff is going to be closest to what we have in the 
trunk of SEDA.  I think MINA was more a thing that should have gone into 
a sandbox.  There is no reason to have it in a branch when it is not the 
same thing that cannot merge back.  Anyway let's see what happens when 
we bang against these servers.   If you go with SEDA in the trunk today 
I don't think switching to Berin's new rendition of SEDA (which usess 
the d-haven event packages) will be hard when it is done.

I'd like to see what the performance curves are for these 
implementations using Eve with search operations as we increase the 
number of concurrent connections and requests. 

Alex





Mime
View raw message