geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jules Gosnell <>
Subject Re: Clustering (long)
Date Tue, 05 Jul 2005 03:29:38 GMT
Dain Sundstrom wrote:

> This is cool stuff and an excellent intro to your code (you should  
> put this summary on your website :)
> I suggest you start chatting with OpenEJB and ActiveMQ about a shared  
> session key sooner rather then later as it could take a wile to get  
> it into their code bases.

I'm just hammering out the portlet specs requirements with David, Jeff, 
Greg and Jan, then I will get back to the other teams.


> I can't wait to see this stuff integrated,
> -dain
> On Jun 30, 2005, at 2:31 PM, Jules Gosnell wrote:
>> Guys,
>> I thought that it was high time that I brought you up to date with my
>> efforts in building a clustering layer for Geronimo.
>> The project,, started as an effort to build a
>> scalable clustered HttpSession implementation, but in doing this, I
>> have built components that should be useful in clustering the state
>> held in any tier of Geronimo e.g. OpenEJB SFSBs etc.
>> WADI (Web Application Distribution Infrastructure) has two main
>> elements - the vertical and the horizontal.
>> Vertically, WADI comprises a stack of pluggable stores. Each store has
>> a pluggable Evicter responsible for demoting aging Sessions
>> downwards. Requests arriving at the container are fed into the top of
>> the stack and progress downwards, until their corresponding Session is
>> found and promoted to the top, where the request is correctly rendered
>> in its presence.
>> Typically the top-level store is in Memory. Aging Sessions are demoted
>> downwards onto exclusively owned LocalDisc. The bottom-most store is a
>> database shared between all nodes in the Cluster. The first node
>> joining the Cluster promotes all Sessions from the database into
>> exclusively-owned store - e.g. LocalDisc. The last node to leave the
>> Cluster demotes all Sessions down back into the database.
>> Horizontally, all nodes in a WADI Cluster are connected (p2p) via a
>> Clustered Store component within this stack. This typically sits at
>> the boundary between exclusive and shared Stores. As requests fall
>> through the stack, looking for their corresponding Session they arrive
>> at the Clustered store, where, if the Session is present anywhere in
>> the Cluster, its location may be learnt. At this point, the Session
>> may be migrated in, underneath the incoming request, or, if its
>> current location is considered advantageous, the request may be
>> proxied or redirected to its remote location. As a node leaves the
>> Cluster, all its Sessions are evacuated to other nodes via this store,
>> so that they may continue to be actively maintained.
>> The space in which Session ids are allocated is divided into a fixed
>> number of Buckets. This number should be large enough such that
>> management of the Buckets may be divided between all nodes in the
>> Cluster roughly evenly. As nodes leave and join the Cluster, a single
>> node, the Coordinator, is responsible for re-Bucketing the Cluster -
>> i.e. reorganising who manages which Buckets and ensuring the safe
>> transfer of the minimum number of Buckets to implement the new
>> layout. The Coordinator is elected via a Pluggable policy. If the
>> Coordinator leaves or fails, a new one is elected. If a node leaves or
>> joins, buckets emigrate from it or immigrate into it, under the
>> control of the Coordinator, to/from the rest of the Cluster.
>> A Session may be efficiently mapped to a Bucket by simply %-ing its
>> ID's hashcode() by the number of Buckets in the Cluster.
>> A Bucket is a map of SessionID:Location, kept up to date with the
>> Location of every Session in the Cluster, of which the id falls into
>> its space. i.e. as Sessions are created, destroyed or moved around the
>> Cluster notifications are sent to the node managing the relevant
>> Bucket, informing it of the change.
>> In this way, if a node receives a request for a Session which it does
>> not own locally, it may pass a message to it, in a maximum of
>> typically two hops, by sending the message to the Bucket owner, who
>> then does a local lookup of the Sessions actual location and forwards
>> the message to it. If Session and Bucket can be colocated, this can
>> reduced to a single hop.
>> Thus, WADI provides a fixed and scalable substrate over the more fluid
>> arrangement that Cluster membership comprises, on top of which further
>> Clustered services may be built.
>> The above functionality exists in WADI CVS and I am currently working
>> on hardening it to the point that I would consider it production
>> strength. I will then consider the addition of some form of state
>> replication, so that, even with the catastrophic failure of a member
>> node, no state is lost from the Cluster.
>> I plan to begin integrating WADI with Geronimo as soon as a certified
>> 1.0-based release starts to settle down. Certification is the most
>> immediate goal and clustering is not really part of the spec, so I
>> think it best to satisfy the first before starting on subsequent
>> goals.
>> Further down the road we need to consider the unification of session
>> id spaces used by e.g. the web and ejb tiers and introduction of an
>> ApplicationSession abstraction - an object which encapsulates all
>> e.g. web and ejb sessions associated with a particular client talking
>> to a particular application. This will allow WADI to maintain the
>> colocation of associated state, whilst moving and replicating it
>> around the Cluster.
>> If anyone would like to know more about WADI, please feel free to ask
>> me questions here on geronimo-dev or on wadi-dev.
>> Thanks for listening,
>> Jules

"Open Source is a self-assembling organism. You dangle a piece of
string into a super-saturated solution and a whole operating-system
crystallises out around it."

 * Jules Gosnell
 * Partner
 * Core Developers Network (Europe)
 * Open Source Training & Support.

View raw message