geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jules Gosnell <ju...@coredevelopers.net>
Subject Re: [Geronimo] Clustering
Date Mon, 19 Dec 2005 11:53:42 GMT
Rajith Attapattu wrote:

>Jules,
>
>Sorry for the late reply and thanks for all the info!!! It's awesome.
>I have more questions if you don't mind. Not very organized so bear with me :(
>  
>
no problem.

>Answers are greatly appreciated.
>
>JNDI implementation
>--------------------
>I guess what you are talking about is our own version of (at the least)JNDI Context implementation
that is cluster aware and the lookup is based on the various strategies outlined in your email.
(This is where I guess Apache directory can be leveraged to build our impl on top)
>  
>
I think so.

ActiveSpace may be useful here.

The distributed caching problem space is large and varied. wadi-core is 
designed very much with a particular subset of this space in mind. This 
is the area concerned with numbers of distributed objects that are too 
great to be held on a single node and that are frequently written and 
read and require pessimistic locking policies to be used in their 
manipulation. JNDI has slightly different requirements. Objects placed 
in JNDI are generally few, rarely written (only on [un]deployment) and 
frequently read. Optimistic locking policies may be sufficient.

Ultimately, wadi-core and ActiveSpace are complimentary technologies, 
both providing solutions to different areas of this problem space. WADI 
also contains lots of integration code to various other containers, so I 
am imagining a situation where a layer of WADI code integrates Geronimo, 
Apache Directory and ActiveSpace, all sitting on top of ActiveCluster 
and ActiveMQ.

>Plus you are taked about passing in membership information to the client via a proprietary
protocol or the client taking on itself to obtain membership info via configuration or an
auto discovery handle.
>  
>
Correct. We may well settle on ActiveCluster as the API for membership. 
I am not sure whether we can reuse autodiscovery code from ActiveMQ 
(which uses it to connect peers when running on its peer:// protocol 
stack), whether there is other suitable Geronimo or ASF-licensed code 
available, or whether we will need to write our own WADI-autodiscovery 
classes. The important thing is to impose as few dependencies on the 
client as possible. The client side code should literally be a few 
lines. Clients using clusters should not suddenly find themselves 
sucking down e.g. the whole of activemq, just to do a once off 
autodiscovery. Early versions of WADI had its own autodiscovery code. If 
we need them, they could be resuscitated.

>Instead can we have a proxy (which sits between the stub and skeleton) which will sit
on our JNDI server and handle any membership issues. Sort of a proxy to a proxy :)
>
>Client --> stub --> proxy (which handles membership issues) --> skeleton -->
EJB
>
>This will shield the client from having to have any knowledge whatsoever about the cluster
membership.
>  
>
Sounds fine. We can solve this problem by aggregation and delegation 
(your model, we delegate all cluster code to an additional proxy and 
aggregate an instance to our client-side stub) or  by inheritance and 
specialisation (we subclass the stub and add cluster awareness - uses 
one less object). If by the Aggregation/Delegation model, we can somehow 
delay the snapshotting of cluster membership to the point when the stub 
is passed to the client, rather than the point when the stub is 
registered with the directory, then this would give significant advantage.

>
>Assumptions
>-------------
>So the basic assumption is that it's a homogenous cluster. Does it mean that only the
EJB's are present in every cluster or is it EJB's plus + other resources are available???
>  
>
Assumptions are dangerous things :-) - lets just say that our initial 
focus should probably be on homogeneous deployments, whilst bearing in 
mind that heterogeneous deployments are bound to eventually impose 
further requirements upon us. I am hoping that we can implement 
heterogeneity in terms of intersecting sets of nodes. So, we specifiy a 
set of nodes that provide 'service-a', a set that provide 'service-b' 
etc. Initially, we will just talk in terms of one universal set that 
provides all services, for the sake of simplicity.

For JNDI, we shouldn't just be thinking in terms of just EJBs, but of 
any distributed component that wishes to register with JNDI.

>Does it mean that they all have access to the same data?? Like they could be pointing
to the same database or a cluster of databases.
>  
>
Hmmm... - I've lost you here - how about this ?

It is entirely up to the EJBs as to what they may map to and not 
something that JNDI should concern itself with.

If two containers both need to publicise EJBs that map to the same 
resource, then I would expect them to both register them with the same 
JNDI name - in this way they are 'unified' and become different 
incarnations of the same 'service'. If they map to different resources, 
then they should be registered with different names, otherwise two 
clients who believe that they are manipulating the same shared resource 
may in fact have their own private copies...

In a homogeous deployment, you can make the assumption (that word again) 
that if a service is available on one node it is available on all nodes. 
In a heterogeneous deployment, we will have to calculate the set of 
nodes on which each service is available and somehow include this info 
in the stub.

This all needs much more thought, but I think that we are going in the 
right direction - does it make sense to you ?

>
>SLSB
>----
>I agree that in general we have a use case for non-sticky clustering. Again as you pointed
out the SLSB is usually a fa├žade for other more complex operations that may use system resources
heavily.
>
>So in such a case are we going to collocate the resources and use a per-Bean or per-User
(or per-Server?) sticky behavior here? If so that means we have to make sure that particular
SLSB is sticky in terms of clustering.
>  
>
Well, I think we need to provide a variety of pluggable policy objects 
at this point, so that the person deploying the EJB can decide which 
policy best suits its intended use. The type of stickiness used at the 
front end will be linked to the data representation at the back end, so 
that they coordinate properly.

>So it boils down to the fact that we are going to define a clustering strategy per bean
(not bean type for ex SLSB or SFSB). Is this correct???
>  
>
I think that this is the only way to do it, We can provide sensible 
defaults based on EJB type, but there will always be counter-examples 
that we have to deal with and it is likely that whatever policies we 
come up with initially will not be a closed set. Someone is bound to 
come along with another form of e.g. 'randomness', or 'stickiness' that 
they want to plug in for their bean.

>SFSB
>-----
>As u mentioned most SFSB are going to be sticky, but if we do cluster the data associated
with the session then we can use a non sticky algo. But u mentioned this is sub optimal? So
is this really expensive?? Is it really worth looking at this option?
>  
>
If the policy is pluggable, then we can avoid having to make tricky 
decisions like this and delegate it to the person doing the deployment.

In a stateful situation, stickiness is usually preferable as 
serialisation and deserialisation are cpu intensive operations. By 
sending an invocation to the wrong node, you may cause the relevant 
session to be serialised on one node, sent over the network to the 
invocation's location and be deserialised here underneath it, ready for 
use. This is going to be much more expensive than sending the invocation 
to the correct place in the first place, or even redirecting or 
forwarding a badly placed invocation (as WADI does).

By making the policy pluggable you also allow people to test exceptional 
cases - e.g. a non-sticky SFSB. This is something that you probably do 
not want in production, but may want to stress test so that you can be 
confident that if affinity is lost (e.g. a node is shutdown and a 
session emigrates to a new location, unbeknownst to a number of 
clients...) you know that your cluster is going to stand up to the 
exceptional stress and not descend into deadlock and chaos.

>Again I guess it depends on the amount of information that should be replicated which
bring us back to the strategy of choosing a clustering type per Bean not by bean type.
>  
>
exactly. We are on the same page :-)

>
>Entity
>--------
>I guess if the underlying data sources are clustered then we can use a non-sticky algo
if not we have to use sticky. Is that correct for all scenarios??
>  
>
By delegating the decision to the deployer, we don't even have to ask 
this question... but, in 99% of cases this is probably the case, 
although, once again, I think that a non-sticky policy here would 
generally be sub-optimal as Entity performance is all about caching, and 
a non-sticky algorithm, whilst possibly working fine, will give you a 
much reduced cache-hit rate.

>More Questions/Answers and discussions will help us to flesh this out more.
>  
>
Good - I hope my answers help - please keep the questions coming - this 
is a very useful thread.

Jules

>Regards,
>Rajith Attapattu.
>
>
>-----Original Message-----
>From: Jules Gosnell [mailto:jules@coredevelopers.net] 
>Sent: Tuesday, December 06, 2005 1:51 PM
>To: dev@geronimo.apache.org
>Cc: dev@wadi.codehaus.org
>Subject: Re: [Geronimo] Clustering
>
>Rajith,
>
>I'll have a crack at these and then if Jeff wants to add anything I'm 
>sure he will :-)
>
>please see inline...
>
>Rajith Attapattu wrote:
>
>  
>
>>Jeff,
>>
>>I am currently involved with JavaMail implementations so haven't had
>>much time to look at the clustering side. Will do soon after the mail
>>thingy is done.
>>
>>However can please comment on the points stated below so when you are
>>done with the 1.0 release and me with the  JavaMail thing, we can
>>continue with the clustering conversation.
>>
>>Thanks,
>>
>>Rajith
>>
>>-----Original Message-----
>>From: Rajith Attapattu [mailto:rajith.attapattu@redknee.com] 
>>Sent: Wednesday, November 30, 2005 7:29 PM
>>To: dev@geronimo.apache.org; jgenender@apache.org
>>Subject: RE: [Geronimo] Clustering
>>
>>
>>Jeff,
>>
>>We maybe able to leverage the Apache Directory for the underlying JNDI
>>aspect of it (I will look in to this, but might need help)
>>
>>
>>How about the policy management portion of clustering service??
>>For ex 
>>Clustering strategy
>>======================
>>Whether to use Sticky vs Random vs other load balancing mechanisms or
>>are we allowing the user to choose a strategy from above.
>> 
>>
>>    
>>
>OK - lets get down to brass tacks.
>
>Ultimately we will need a number of different policies including those 
>that you have outlined above.
>
>Different policies will suit different types of bean and different use 
>cases and it would be good to put together some sort of list of common 
>combinations.
>
>Here is a start...
>
>This is all straight off the top of my head - so be gentle :-)
>
>I think we probably can assume homogeneous JNDI population - i.e. all 
>JNDI services carry exactly the same information - otherwise a client 
>would have to visit all of them to be sure that a service it required 
>did not exist. This means that deploying a new JNDI entry is expensive 
>(since it has to update all JNDI service replicas), but the deployment 
>of new entries is an exceptional event, whereas client lookup is a 
>common event - so we keep the common event cheap and the exceptional one 
>pays the price.
>
>So, somehow (probably initially via some sort of autodiscovery 
>mechanism) the client hooks up with a jndi service replica and asks it 
>for a EJBHome stub.
>
>This stub is returned from JNDI to the Client.
>
>I guess the JNDI service could be cluster-aware and package recent 
>information about cluster membership with the stub back to the client 
>(proprietary protocol required?), or it could be dumb, in which case the 
>newly demarshalled stub may have to obtain this information for itself - 
>by having had some information about the cluster's membership and 
>auto-discovery handle (e.g. multicast address) serialised with it 
>(possibly a long time before it is finally used - so the membership data 
>may be stale and need refreshing via autodiscovery).
>
>I guess we will at least want to have the opportunity of colocating all 
>serverside resources associated with a single user - since this will 
>greatly improve our response to said user. This also creates a handy way 
>to begin partitioning the problem that we might end up with if we just 
>created every ejb on a random box then hooked them all up together. So, 
>there is a good usecase here for requiring some sort of 'stickiness' 
>immediately.
>
>So, it looks like we need two types of Home stub - a sticky one, which 
>might look at a unique userid held within the client (can we do this?) 
>and use this to map this user to a particular server and a non-sticky 
>type, which might use a variety of pluggable algorithms (or subtypes) 
>including random (but also e.g. trying to find the least loaded server, 
>the nearest server etc...). (there is also another type of stickiness 
>where we use the non-sticky algorithm to choose an initial server, then 
>go back to the same one thenceforth - but this does not give us such 
>finegrained control over the colocation of resources belonging to the 
>same user - perhaps we call this per-server stickiness).
>
>So - when I talk about sticky here - I am talking about per-user 
>stickiness (all invocations from the same user go to the same target) 
>not just per-bean stickiness (all invocations for the same bean go to 
>the same server).
>
>Lets say the client uses the Home stub to create an EJB instance for 
>which it receives a client-side stub. Lets examine the different 
>possibilities.
>
>SLSB -
>
>Logically it does not matter (in a homogeneous deployment) where 
>invocations for a SLSB (I'm open to correction on all of this - my EJB 
>is very rusty) might land. A SLSB should be lightweight and be able to 
>be transparently brought into existance underneath the incoming 
>invocation, service it and then be trashed (of course containers pool 
>etc. to reduce overheads associated with this sort of thing). So, we 
>have a good usecase for non-sticky behaviour in the SLSB stub here.
>
>In reality, it is likely that this SLSB will access further resources 
>associated with the client that may be more expensive to pop in and out 
>of existance (e.g. Entities), so there is a good case for a per-Bean or 
>per-User (or per-Server?) sticky behaviour here.
>
>SFSB -
>
>The reverse of the SLSB.
>
>Logically it does matter where invocations fall. If the state held in 
>the SFSB is tied to a single JVM then we would expect a 
>per-User/Bean/Server sticky algorithm in this sort of stub.
>
>In reality, hopefully, the associated SFSB will be WADI-enabled and able 
>to travel to wherever it is needed. So, a non-sticky algorithm would 
>also work, but would probably be rather sub-optimal :-)
>
>Entity -
>
>More like an SLSB, I guess. Logically an Entity should be able to spring 
>into existance anywhere it is needed. In reality there is cost 
>associated with this. Cache hits are good, Cache misses are bad - So, 
>again, all algorithms should work, but a sticky variant is probably the 
>most common deployment option.
>
>On the server-side, these different policies should tie up with some 
>ideas that I have discussed on the list about the way that a users 
>remote resources may be colocated in some sort of SuperSession (maybe a 
>UserSession or an ApplicationSession (maybe a per-Application stickiness 
>is also required) (colocates resources from various applications and 
>tiers that are all associated with the same application-and-user or just 
>user). So choosing a particular ServSide impl would constrain the 
>possible client-side load-balancing algorithms available to the 
>application i.e. they walk hand in hand. Unless we want to be really 
>outrageous (Geronimo-4.0?), these choices should be made at deployment 
>time, by humans with knowledge of the layout of the application 
>components within the cluster and their expected behaviour.....
>
>I think that is it for the moment !
>
>Apologies for the rather rambling nature of my thoughts on this - I 
>haven't really thought about this stuff rigorously yet, but this should 
>be enough to share where I am with the list.
>
>
>Jules
>
>P.S.
>
>We need to figure out how the various forms of stickiness are maintained 
>when a node dies or voluntarily leaves a cluster. There is lots to talk 
>about.
>
>
>  
>
>>We can represent each clustering strategy as a GBean which the user can
>>pick from (under the Clustering services GBean, I assume you have the
>>whole clustering feature represented as a GBean ).
>>
>>So if somebody is not happy about the clustering strategy then simply
>>write there own and add that as a GBean.
>>
>>Of course we will have to come up with a neat API for exposing the
>>aspects that should be open for improvement.
>>
>>This will also help us to come up with better clustering strategies
>>later on in the future without a major impact on the code base.
>>
>>What are your thoughts on this??? Everybody please help with ideas :)
>>
>>Regards,
>>
>>Rajith Attapattu.
>>
>>-----Original Message-----
>>From: Jeff Genender [mailto:jgenender@apache.org] 
>>Sent: Wednesday, November 30, 2005 1:13 PM
>>To: dev@geronimo.apache.org
>>Subject: Re: [Geronimo] Clustering
>>
>>
>>
>>Rajith Attapattu wrote:
>> 
>>
>>    
>>
>>>Jeff,
>>>
>>>Apologize for late reply, down with flu.
>>>
>>>Is high availability JNDI (or JNDI clustering) a concept brought up by
>>>JBoss??
>>>   
>>>
>>>      
>>>
>>I don't know the answer to this question.
>>
>> 
>>
>>    
>>
>>>Frankly I am no expert on this, so any pointers will be very helpful.
>>>   
>>>
>>>      
>>>
>>I
>> 
>>
>>    
>>
>>>see that WADI is yet to implement this. So do u have any documentation
>>>on this?
>>>   
>>>
>>>      
>>>
>>This is an area we are all starting to look at.  One area I would 
>>recommend looking at is seeing if we can leverage the Apache Directory 
>>to handle the HA component of JNDI.  If so, this may be a much simpler
>>job.
>>
>>Thanks for helping out...this is great to have more folks chipping in on
>>
>>this.
>>
>>
>> 
>>
>>    
>>
>>>I assume we will follow the jboss concept closely, but hopefully to
>>>improve on it.
>>>
>>>Any help is greatly appreciated. 
>>>
>>>Rajith.
>>>
>>>-----Original Message-----
>>>From: Jeff Genender [mailto:jgenender@apache.org] 
>>>Sent: Monday, November 28, 2005 1:00 PM
>>>To: dev@geronimo.apache.org
>>>Subject: Re: [Geronimo] Clustering
>>>
>>>
>>>
>>>Rajith Attapattu wrote:
>>>   
>>>
>>>      
>>>
>>>>As per Jeff's request I am currently ramping up on WADI. I guess jeff
>>>>     
>>>>
>>>>        
>>>>
>>>will shortly announce the integration or any other intermediate tasks
>>>that needs to be done before WADI can be integrated.
>>>   
>>>
>>>      
>>>
>>>>I guess we will have some discussion on what areas we will work on
>>>>     
>>>>
>>>>        
>>>>
>>>when the plan is announced??
>>>   
>>>
>>>      
>>>
>>>>Jeff can you pls comment on this?
>>>>     
>>>>
>>>>        
>>>>
>>>I think we just need a couple of Gbeans to get it initially integrated
>>>   
>>>
>>>      
>>>
>> 
>>
>>    
>>
>>>in the web tier...I will tackle that.  It currently works under Tomcat
>>>   
>>>
>>>      
>>>
>> 
>>
>>    
>>
>>>and Jetty in their standalone configurations.  Gianni is currently 
>>>working on the OpenEJB session integration with WADI...and we look 
>>>forward to getting that.
>>>
>>>We are interested in the HA JNDI...so lets definitely get some 
>>>discussion going on that.
>>>
>>>Jeff
>>>
>>>   
>>>
>>>      
>>>
>>>>With Kind Regards,
>>>>
>>>>Rajith Attapattu.
>>>>________________________________________
>>>>From: Panda, Akshaya Kumar (Cognizant)
>>>>     
>>>>
>>>>        
>>>>
>>>[mailto:akshaya.panda@cognizant.com] 
>>>   
>>>
>>>      
>>>
>>>>Sent: Sunday, November 27, 2005 11:42 PM
>>>>To: dev@geronimo.apache.org
>>>>Subject: RE: [Geronimo] Road Map and TODO
>>>>
>>>>Jeff/Rajith,
>>>>Yes i will be intersted to work on this. But before jumping in on, i
>>>>     
>>>>
>>>>        
>>>>
>>>would like to know who all are involved in     what areas so that it
>>>   
>>>
>>>      
>>>
>>can
>> 
>>
>>    
>>
>>>help me to avoid any duplicate effort.
>>>   
>>>
>>>      
>>>
>>>>Rajith: can you pl outline what you have planned to do?
>>>>
>>>>thanks
>>>>akshay
>>>>
>>>>
>>>>-----Original Message-----
>>>>From: Jeff Genender [mailto:jgenender@apache.org]
>>>>Sent: Sat 11/26/2005 9:17 PM
>>>>To: dev@geronimo.apache.org
>>>>Subject: Re: [Geronimo] Road Map and TODO
>>>>
>>>>
>>>>
>>>>Panda, Akshaya Kumar (Cognizant) wrote:
>>>>
>>>>     
>>>>
>>>>        
>>>>
>>>>>I think WADI is going to provide web tier clustering, any initiative
>>>>>       
>>>>>
>>>>>          
>>>>>
>>>for HA-JNDI?
>>>   
>>>
>>>      
>>>
>>>>>thanks
>>>>>Akshay
>>>>>       
>>>>>
>>>>>          
>>>>>
>>>>No, WADI (and probably should be renamed remove the web-only
>>>>connotations) will be providing all HA facets to Geronimo.  If you
>>>>     
>>>>
>>>>        
>>>>
>>are
>> 
>>
>>    
>>
>>>>interested in this area, this is where we could use a helping hand,
>>>>     
>>>>
>>>>        
>>>>
>>so
>> 
>>
>>    
>>
>>>>feel free to jump in on this.
>>>>
>>>>Jeff
>>>>
>>>>     
>>>>
>>>>        
>>>>
>
>
>  
>


-- 
"Open Source is a self-assembling organism. You dangle a piece of
string into a super-saturated solution and a whole operating-system
crystallises out around it."

/**********************************
 * Jules Gosnell
 * Partner
 * Core Developers Network (Europe)
 *
 *    www.coredevelopers.net
 *
 * Open Source Training & Support.
 **********************************/


Mime
View raw message