Mailing-List: contact directory-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Apache Directory Developers List"
 <directory-dev@incubator.apache.org>
Received-SPF: pass (hermes.apache.org: domain of aok123@bellsouth.net
 designates 205.152.59.69 as permitted sender)
Message-ID: <41B9D9E0.1080209@bellsouth.net>
Date: Fri, 10 Dec 2004 12:16:16 -0500
From: Alex Karasulu <aok123@bellsouth.net>
User-Agent: Mozilla Thunderbird 1.0RC1 (X11/20041201)
MIME-Version: 1.0
To: Apache Directory Developers List <directory-dev@incubator.apache.org>
Subject: Re: Proposing some changes to the ProtocolProvider API
References: <41B8D8B9.8060600@d-haven.org> <41B93A89.4040204@bellsouth.net>
 <41B9AEFC.206@d-haven.org>
In-Reply-To: <41B9AEFC.206@d-haven.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Berin Loritsch wrote:

> Alex Karasulu wrote:
>
> <snip>
>
>> Sorry but from a first look that's a negative.  All your response 
>> objects are expected to be created in advance.  Namely I'm referring 
>> to he Object[] returned from the processRequest() method.  Basically 
>> think about Search operations in Eve....
>>
>> Search requests can return 1 or more responses that equals zero or 
>> more ExtendedResponses or SearchResultEntryResponses followed by as 
>> SearchResultDone to terminate the request.  Eve is designed to stream 
>> out search results rather than keep an entire result set in memory.  
>> By having an Object[] return all responses at once I have to cache 
>> all the results before even starting to return any entries.  This 
>> will make Eve extremely inefficient.  Hence the reason why I return 
>> an Iterator.  Every time you ask for the next response its pulling an 
>> entry from the database which matches the filter.
>> Eve does not fetch an entry until you're ready to return it to a 
>> client.  This keeps the perclient resources used to a minimum. 
>> Without this performance would degrade considerably.  So look at 
>> using a Cursor friendly (one that can wrap a cursor) construct like 
>> Iterators, Enumerations, et, cetera.
>
>
> I can understand where you are coming from.  However, the current 
> protocol provider solution is far too complex to really digest quickly--

Understood! I guess its easier for me since I came up with it :(.  
However let me break down my thoughts a little on what I see as protocol 
concerns.  It's really simple in my head.  You got two big yet general 
aspects to a protocol:

encoding/decoding
request handling

That's all we need to expose in these interfaces but with the caveats I 
mention above for performance.

> hense making a simple Echo server or something of that nature is too 
> complicated.  I boiled everything down as much as possible to start 
> the conversation going.

Yeah but who's really in it to just build an Echo server.  Really its 
easy to think this is way to complex when you look at the simplest of 
cases.  But still I'm going to take your word for it that the present 
interfaces are too complex.  However try to see that Echo servers are 
not a good example to use for guiding the overall design.  They're just 
too trivial.  But we should handle the trivial case without making it 
appear way to complex.

>
> I believe that something needs to change--and I am not necessarily 
> looking for change *before* release.  

Cool I'm comfortable with that.  Hope I have not dismayed you by saying 
somethings are not good to do. 

> I think that both of the proposed network layers are slated more for 
> the next release of Eve.  However, the details of how things change 
> are always up for discussion.

Absolutely! Please let me know if I ever give you another impression.

>
> Based on this information, I would like to propose a modification to 
> what I originally put up.  More below.
>
Cool.

>>> This will allow the smallest footprint for really small and easy 
>>> systems, yet allows
>>> more complex protocol providers to delegate to helper objects as 
>>> need be.  I have
>>> no problems with using the current factory approach either--as long 
>>> as the encode/
>>> decode process returns something that can be passed on.
>>
>>
>> I like the factory approach because you can delay when the 
>> encoder/decoder pairs are created and by which module.  There is lots 
>> of flexibility here.   Otherwise you're basically making the provider 
>> the codec as well as the request processor.  These are three separate 
>> functions that need not be coupled.  Let's explore why the 
>> ProtocolProvider interface is so inhibitive.  What problems are you 
>> having with it?
>
>
>
> The problems I am having with it are as follows:
>
> 1) Part of the API is in Snickers while the other part is in 
> Seda--tying two implementations together

Yep I see what you mean.  So you want to just split the two and we can 
use adapters to adapt snickers codecs to SEDA constructs like I think 
Trustin has done? I'm cool with doing that then we have no dep on snickers.

> 2) More complex than it needs to be--leading to more time trying to 
> figure the thing out

I think this is objective but probably true.  I just have a hard time 
seeing it but I admit my biases :).

> 3) It is too wed to the current API to be retrofit easily into what 
> either Trustin or I are doing

Don't know how this is the case perhaps I can have an example of this 
just to enlighten me - I'm drawing a blank on this.

> 4) Encoders do not encode and respond... they encode and it goes into 
> an abyss

:) this is how the callbacks work!  You keep feeding the decoder for 
example and you get a callback to tell you when it is done.  You can 
make such constructs synchronous or asynchronous this way if you wanted 
to.  But I do understand how these callbacks are not the most intuitive 
things for people to get a comfort level with.

> 5) Same with Decoders...
>
Yep.

> The thing is that in order to implement a Protocol you need no less 
> than 6 objects--even if your needs are very minimal.

6 objects being?

- EncoderFactory impl
- DecoderFactory impl
- ProtocolProvider impl
- Encoder impl
- Decoder impl
- Handlers ...

>   I boiled everything down to one interface--recognizing that internal 
> to that interface some protocols might want to use separate 
> incoders/decoders/ etc.  However, since that is shifting the bulk of 
> the work on the protocol writer, I can temper the request a bit.
>
I don't think shifting work to the protocol writer is that much of an 
issue.  My concerns are just making sure we separate out codecs from 
providers.  This drive comes from the fact that some codecs may be 
shared by multiple protocols.  However then again this is at the lowest 
levels.  I'm referring to BER/DER/PER specifically here.  But we are at 
higher levels dealing with beans representing PDUs in the protocol.  So 
the codec in question here is a high level one specific to the protocol. 

Thinking out loud. ...

So separation of codec from protocol req handling is just good form 
then. I don't think reuse is going to be the drive behind it.

>>> Currently the encoder explicitly publishes the "encode event" 
>>> directly to the
>>> event publisher from within client code.  In this sense it is 
>>> explicitly tied to the
>>> current SEDA implementation.
>>>
>> There is an EncoderManager which does the publishing since as a 
>> component it holds a dep on the event router.  No encoder will 
>> publish to an event queue.  Encoders simply trigger the manager to 
>> enqueue.
>
>
> Yes and this manager you speak of is, again, an extra step that might 
> not be needed.  It is an added complexity that we can most likely do 
> without.

Ok let me see some of the examples as you come up with them.

>
>>
>>> How does this sound?
>>
>>
>> Sorry Berin I cannot go with this for the reasons stated inline.  
>> However I'm open to working with you.  I just need to know why you're 
>> having problems.  I also want to make sure you're familiar enough 
>> with the current SEDA and Eve code to understand why I made most of 
>> my design choices and why I cannot veer away from certain mechanisms.
>> Alex
>
>
> Currently, the ProtocolProvider has three methods: 
> getDecoderFactory(), getEncoderFactory(), and getRequestHandler()
>
> That is fairly workable; however, there are two levels of 
> abstraction--the ProtocolProvider is a factory for the request 
> handler, but only returns the factory for the encoders and decoders.  
> Can we simply make the ProtocolProvider a Factory?

I made the provider return factories for codec halves rather than then 
the codecs themselves because of the fact that the factory is needed to 
create a encoder/decoder whenever a new client connects.  This needs to 
be managed by a session manager concerned with setting up encoding 
decoding infrastructure for the newly connected client (in TCP land).  
But if your architecture does not need this then do not bother keeping it.

> That would simplify it in this way:
>
> // Responsibility of ProtocolProvider to write
> interface ProtocolProvider
> {
>    StatefulDecoder getDecoder( Socket client, DecoderCallBack callback );
>
>    RequestHandler getRequestHandler( Socket client, RequestCallBack 
> callback );
>
>    StatefulEncoder getEncoder( Socket client, EncoderCallBack callback );
> }
>
> // Responsibility of ProtocolProvider to write
> interface StatefulDecoder
> {
>    // internally calls the DecoderCallBack when something is there to 
> send on
>    void decode(ByteBuffer buffer);
> }
>
> // Responsibility of ProtocolProvider to write
> interface RequestHandler
> {
>    // internally calls the RequestCallBack when any response is ready
>    void handleRequest(Object fromDecoder);
> }
>
> // Responsibility of ProtocolProvider to write
> interface StatefulEncoder
> {
>    // internally calls the EncoderCallBack when buffers are ready.
>    void encode(Object fromRequestHandler);
> }
>
> // Responsibility of network system to write
> interface DecoderCallBack
> {
>    void decoded(Socket client, Object requestObject);
> }
>
> // Responsibility of network system to write
> interface RequestCallBack
> {
>    void response(Socket client, Object responseObject);
> }
>
> // Responsibility of network system to write
> interface EncoderCallBack
> {
>    void encoded(Socket client, ByteBuffer buffer);
> }
>
> This takes things like the "XXXManager" out of the picture--the 
> responsibilties are much smaller.

Ok that's cool - I always thought the XXXManager thing was weak and I 
agree with you!

> It also makes it easier to forward events asynchronously when they 
> happen.  As far as the network
> is concerned, only the Socket identifies a client--session objects and 
> other more complex things are
> part of whatever protocol is being serviced by the network layer.
>
> And more importantly, this allows us to identify a specific 
> ProtocolProvider API.  This API can
> (and should) be a dependency of the network layers and the individual 
> providers.  However the
> protocol providers should not have to have a dependency on the network 
> layer.

+1

>
> Currently we have part of the provider API provided by snickers, and 
> yet another part of it provided
> by the network API.  It makes the project hierarchy more difficult to 
> manage as well.  By separating
> out the protocol provider API into its own _clearly_marked_ 
> subproject, we can clean up the
> dependency structure in a way that truly decouples the network layer 
> from the protocol layer--which
> also gives us more of an opportunity to switch things up and optimize 
> things without fearing breaking
> the different protocol providers.
>
> It also makes it tremendously easier to add new protocol providers as 
> necessary.
>
Yes I agree this is excellent!

=== Alex's Big Point ===

I have taken the wrong approach to guarding these ProtoPro interfaces.  
We need to use a common one but perhaps not impose that immediately.

Just to conclude don't fear demolishing existing structures for new 
ones.  I recommend we don't make these branches branches but new 
projects that do the same thing in different ways.  The current setup 
should not hold back your new ideas and simpler ways of doing things.  
We can afford this now if we're not going to push it into this next 
release.

I know this is a bit contradictory right now to me saying we gotta agree 
on ProtoPro interfaces to allow for interop.  However let's take a cycle 
where we just hack away at it to make it work the way each individuals 
thinks it should.  Then let's come back, compare and look at interop via 
agreeing upon the common interfaces.  Now this does not mean that we 
don't discuss these interfaces openly in advance cuz doing so will help 
make sure we get it right the first time.

Alex