Mailing-List: contact dev-help@geronimo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@geronimo.apache.org
Received-SPF: neutral (asf.osuosl.org: local policy)
Message-ID: <4408E0E3.3010004@hanik.com>
Date: Fri, 03 Mar 2006 18:35:47 -0600
From: Filip Hanik - Dev Lists <devlists@hanik.com>
User-Agent: Thunderbird 1.5 (Windows/20051201)
MIME-Version: 1.0
To: dev@geronimo.apache.org
Subject: Re: Session Policy was: heads up: initial contribution of a client
 API to session state management for OpenEJB, ServiceMix, Lingo and Tuscany
References: <1F5F68F5-2E35-4AAF-B90B-072081D34FE7@gmail.com>
 <44042DE9.2080203@mortbay.com>
 <F10E5C28-9354-4C62-84B8-2FA718F27720@iq80.com>
 <4406A4DE.2070603@mortbay.com>
 <E4F739BA-93E1-4C51-AF9A-6286CFCB2A4B@iq80.com> <44085A1C.80708@hanik.com>
 <080FB77D-659D-489E-AE83-9E1797DA4F86@iq80.com> <4408D77F.2020908@hanik.com>
In-Reply-To: <4408D77F.2020908@hanik.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

btw, for very large clusters, you use the same mechanism, except, 
instead of distributing the entire session map, the backup node info is 
stored in a cookie.

and by doing this, you don't need to remember the backup location 
throughout the cluster. you still broadcast cancellations of primary 
node to account for false positive.

the only scenario that is not accounted for is when you have a wacky lb 
that sends two parallel requests to two different servers. this would 
require distributed locking, and that is a path that is too much 
overhead to walk down.


Filip


Filip Hanik - Dev Lists wrote:
> Hi Dain,
> let me address the location, and show you how the location is 
> completely transparent.
>
> The way the LazyReplicatedMap works is as follows:
> 1. Backup node fails -> primary node chooses a new backup node
> 2. Primary node fails -> since Tomcat doesn't know which node the user 
> will come to their
>   next http request, nothing is done.
>   When the user makes a request, and the session manager says 
> LazyMap.getSession(id) and that session is not yet on the server,
>   the lazymap will request the session from the backup server, load it 
> up, set this node as primary.
>   that is why it is called lazy, cause it wont load the session until 
> it is actually needed, and because it doesn't know what node will 
> become primary, this is decided by the load balancer. remember, that 
> each node knows where the session with Id=XXXX is located. they all 
> carry the same map, but only two carry the data (primary secondary).
>
> on a false positive, the new primary node will cancel out the old one. 
> so you can have as many false positives as you want, but the more you 
> have the worse your performance will get :). that is why sticky lb is 
> important, but false positive is handled the same way as a crash 
> except that the old primary gets cancelled out.
>
> the rest is inlined
>>>
>>> 1. Requirements to be implemented by the Session.java API
>>>   bool isDirty - (has the session changed in this request)
>>>   bool isDiffable - is the session able provide a diff
>>>   byte[] getSessionData() - returns the whole session
>>>   byte[] getSessionDiff() - optional, see isDiffable, resets the 
>>> diff data
>>>   void setSessionDiff(byte[] diff) - optional, see isDiffable, apply 
>>> changes from another node
>>
>> To throw you arguments back on you, why should my code be exposed to 
>> this level of detail :)   From my perspective, I get a session and it 
>> is the Session API implementation's problem to figure out how to diff 
>> it, back it up, and migrate it.
>>
> exactly. the methods above is what is required from the servlet 
> container, not the webapp developer.
> so if you are a jetty developer, you would implement the above 
> methods. This way, the jetty developer can optimize the serialization 
> algorithm, and locking (during diff creation), and your session will 
> never be out of date. in tomcat, we are making the getSessionDiff() a 
> pluggable algorithm, but it is implemented in the container, 
> otherwise, just serialization is too slow.
>
>>> 2. Requirements to be implemented by the SessionManager.java API
>>>   void setSessionMap(HashMap map) - makes the map implementation 
>>> pluggable
>>>
>>> 3. And the key to this, is that we will have an implementation of a 
>>> LazyReplicatedHashMap
>>>   The key object in this map is the session Id.
>>>   The map entry object is an object that looks like this
>>>   ReplicatedEntry {
>>>      string id;//sessionid
>>>      bool isPrimary; //does this node hold the data
>>>      bool isBackup; //does this node hold backup data
>>>      Session session; //not null values for primary and backup nodes
>>>      Member primary; //information about the primary node
>>>      Member backup; //information about the backup node
>>>   }
>>>
>>>   The LazyReplicatedHashMap overrides get(key) and put(id,session)
>>
>> Why would anyone need to know this level of detail?
> you don't and you will not, I just giving you some architectural 
> insight on how it works under the hood :)
>
>>
>>> So all the nodes will have the a sessionId,ReplicatedEntry 
>>> combinations in their session map. But only two nodes will have the 
>>> actual data.
>>> This solution is for sticky LB only, but when failover happens, the 
>>> LB can pick any node as each node knows where to get the data.
>>> The newly selected node, will keep the backup node or select a new 
>>> one, and do a publish to the entire cluster of the locations.
>>
>> I don't see anyway to deal with locking or the fact that servlet 
>> sessions are multi threaded (overlaping requests).  How do you know 
>> when the session is not being used by anyone so you have a stable 
>> state for replication.
> in tomcat we have an access counter, gets incremented when the request 
> comes in, and decremented when the request leaves. if the counter is 
> 0, lock the session and suck out the diff. or just lock it at the end 
> of each request on a periodic basis, regardless of what the counter is.
>
>>
>>> As you can see, all-to-all communications only happens when a 
>>> Session is (created|destroyed|failover). Other than that it is 
>>> primary-to-backup communication only, and this can be in terms of 
>>> diffs or entire sessions using the isDirty or getDiff. This is 
>>> triggered either by an interceptor at the end of each request or by 
>>> a batch process for less network jitter but less accuracy (but 
>>> adequate) for fail over.
>>>
>>> As you can see, access time is not relevant here, nor does the 
>>> Session API even know about clustering.
>>
>> How do you deal with access-time?  I agree that your API doesn't know 
>> about clustering, but you also can't do a client side or server side 
>> redirect to the correct node; you must always migrate the session to 
>> your request.
> it doesn't, there is no reason to. only the primary node can expire 
> it, and when the primary manager, without knowing it is primary, does 
> a sessionmap.remove() the LazyReplicatedMap removes it across the 
> cluster.
> remember, when the session manager does, 
> sessionmap.entrySet().iterator() it only gets session from this node, 
> not the other nodes.
> so the implementation is completely transparent to the jetty programmer.
>
>>
>>> In tomcat we have separated out group communication into a separate 
>>> module, we are implementing the LazyReplicatedHashMap right now just 
>>> for this purpose.
>>
>> Cool.  I'm interested to see what you come up with.
> I will keep you posted, maybe we could share the code/experience.
>
> Filip
>