geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jules Gosnell <ju...@coredevelopers.net>
Subject Re: WADI clustering
Date Tue, 24 Jan 2006 10:33:17 GMT
Rajith Attapattu wrote:

> Jules,
>  
> More questions :)
>  
> If we assume the single-active-model (with n backups)
>  
> 1. Do we still need a locking mechanism?? Bcos if we have only one 
> live session where updates can happen and the rest are serialized 
> backup copies there is no need for a locking mechanism. Is this 
> correct?? or have I missed something here?

This is correct IFF you have 100% affinity, or, to be absolutely 
specific, you can guarantee that concurrent requests qill ALWAYS be 
routed to the same node. There are a number of cases where this may not 
happen (e.g. mod_jk[12] when a node leaves).

High-end HTTP lbs and our own EJB proxies will guarantee this, so you 
would think that there is room for serious optimisation here - However...

WADI is a partitioned cache - This allows the sum of the state held in a 
tier to be greater than the capacity of a single node - scalability. 
Every node needs to be able to locate every session. Session locations 
are held in indeces in Partition components, which are also responsible 
for holding a session's distributed lock. So, during a migration,  
although you can avoid having to take a distributed lock, you can't 
avoid having to still make a round trip to the Partition to find and 
then update the session in question's location... So the savings that we 
can make are not as great as you might imagine :-(

>  
> 2. Assuming a change happed within the live session (for ex an 
> attribute was added or modified)
>   a) will it be immedialely broadcasted or are they batched (lets say 
> two or 3 changes to gether) before being broadcasted
>  
>  b) Is it the delta that is sent or is it the whole session thats 
> moved across. (I assume it's the delta with some meta-info like 
> session_id, version, date_time..etc)
>
>  

Currently (unless Gianny Damour is racing away with this?) we only have 
one point at which replication occurs (although I expect that we will 
support several e.g. immediate, after so many seconds of inactivity 
etc...). This point is end-of-request-group. (EORG). EORG is the only 
provably safe place to replicate, since it is the only point at which we 
know for sure that all requests/invocations have left the container and 
so cannot be reading/writing the session. We cannot assume the 
end-of-request is safe (for web), because requests may overlap (in the 
web container - in the ejb container EORG==end-of-request since 
invocations are serialised).

Each request takes a shared-lock on the session. As the last of these is 
released the lock calls back into container code and replication is done.

As to whether the replication message will contain a complete copy, or a 
delta - this will be pluggable.

> 3. when these changes are recived by the other copies how do they use it?
>    Is it stored seperately in a queue in serialized format and applied 
> once the session is de-srialized??
>  
>   I guess it will be an expensive operation if we keep serializing the 
> session, then apply the change and then de-serialize it.

It depends on the granularity of the replication.

With per-session, the new copy will just replace the old copy.
With per-attribute (delta), I expect that the replicant node will hold a 
map of key:serialised-attribute. The new serialised attr will replace 
the old one...
There are other ways to generate a delta - we could actually 'diff' an 
old serialised copy against a new one on one side and 'patch' the old 
copy on the other side - but this is probably too much work...
If this stuff REALLY interests you, I did have an idea for a marshalling 
stream that does per-object delta-ing of components passed onto it, but 
whilst great to have, this would be a considerable amount of work....

>  
>  
> 4. How are these backup copies stored within WADI??
>     a) Can they be stored in a central place like DB or a file

yes - there is the beginnings of a pluggable API here and a basic 
DB-based impl to test it.

>  
>     b) Can they be in memory or a file within the node?

yes - these would just be other impls of the same API.

I expect the in-vm version to be the recommended solution.

>  
>  
> >ActiveCluster allows each node to maintain a small amount of published
> >data about itself. It can write this data. All other nodes can read it.
> >I use it, currently, to advertise human-readable-node-name, http-port,
> >shutting-down?, etc... - I figure that if if e.g. the number of sessions
> >that you are carrying fluctuates by more than 10% since the last
> >published figure, you publish the new number etc. We want to throttle
> >publishing a little because each publishing causes a 1->all message
>  
> 1. Ok, where is this information about each node is stored??
>    If a nide publish it's chnages I assume it will go to each node and 
> essentialy everynode will have copy of each nodes meta-info like 
> node_name, ip, port, no_of_sessions ..etc

yes

>  
> 2. But as the nodes increase then the chatter among each nodes are 
> going to increase a lot. If there are N nodes then each node will have 
> N no of copies to maintain.
>  

yes - which is why you throttle the amount of notification that you do 
and only provide the minimum amount of info needed to get the job done.

>    Does logical partitioning slove this problem? Like u divide the 10 
> nodes into groups of 3,3 & 4, where they interact only between the group.
>    Does WADI (or Active Cluster) provide this logical partitioning yet 
> allow to view the cluster as a whole.

WADI uses automatic partitioning to resolve this problem for data 
(sessions), but metadata (data about the nodes and sessions) is 
considered to be sufficiently small and static as to not require this 
approach.

>  
>    If this is the case I guess the load balancer should be aware of 
> this configuration and keep requests within the respective groups. Is 
> this possible with current load balances ??

Various strategies are available with various load-balancers, but if you 
go this route, you might as well just keep your clusters down to three 
nodes, replicate your sessions 1->all etc - you are back at square 1. 
WADI allows you to forget about these scaling issues, partitioning etc 
and just add and remove nodes to/from as large a cluster as you like 
without further complexity. I see no scaling problem with the freq/size 
of metadata messages.

>  
> 3. So basically each node makes the decesions for themselves instead 
> of a central command and control center??? I guess distributed 
> decesion making is good bcos otherwise there will be election 
> stratergies involved in electing a leader if the leader_node goes down.

Essentially - yes. I prefer the distributed to the centralised metaphor 
as I think that it is generally going to be more scalable and resilient. 
It is, however, much harder to work out.

>  
> 4. Lets say there is need for a system admin to monitor the cluster. 
> And u did mention that u can plug in listerners to the underlying AC code.
>    Basically what I like is a  management API where I can monitor the 
> node via listners or by registering my self as a dummy node and 
> provide that infomation in  human readable format.
>    You mentioned this is under construction at the moment. (I really 
> like that feature and see a lot of value in it for an end user)

Monitoring is an area that I delineate in the clustering doc (I am still 
waiting for the 'which wiki' thread to culminate in a winner before I 
make it available). Monitoring is one of the few areas where I think a 
centralised model may be necessary, since you need to aggregate all the 
data in one place to present an decent overview of the cluster and you 
certainly do not want to aggregate it in every place.... When the doc 
gets out, I will kick off a thread about this, from a Geronimo rather 
than WADI perspective.

>  
> By the way when are u guys moving WADI into incubation so we can 
> implement some of the ideas we talked about??

It is on my list, close to the top, but why wait ? Just jump in at 
wadi.codehaus.org and take a look.

>  
> Sorry about the long list, but more questions I ask the better 
> understanding I get (of course at the expense of your time :) )

No problem - I am happy to spend the time, because this thread will be 
available to anyone with an interest in what we are talking about and 
because you are asking questions that they would ask, but that I might 
not think to include in doc.


Jules

>  
> Regards,
>  
> Rajith.
>  
> On 1/22/06, *Jules Gosnell* <jules@coredevelopers.net 
> <mailto:jules@coredevelopers.net>> wrote:
>
>     Rajith Attapattu wrote:
>
>     > >>However if we use 100% session affinity then the chance
>     > Sorry it should be asume not use. :)
>     >
>     > Don't think of it as a battle, it's a disscussion for me (and
>     perhaps
>     > for you) to understand both sides of the coin. Unless u think my
>     > questions are stupid :)
>
>     Good - I'm glad that we see eye to eye on this one. It is indeed an
>     interesting technical debate. I'm just a little paranoid about
>     upsetting
>     people at the moment, because I seem to be doing a lot of it :-)
>
>     >
>     > >>Sorry to put you down again - I don't want to, but when I
>     think it
>     > all through, I just can't see a good reason to go this route.....
>     >
>     > >>I am planning to refactor this area in WADI to be pluggable,
>     so you
>     > wil have the option of writing something like this if you really
>     want it
>     >
>     > Well u haven't put me down, u have agreed to refactor this area in
>     > WADI to be pluggable (Thats what I wanted).
>     > All I am saying is we don't have to go ahead and implement the
>     > scenario, but we shouldn't close the door either !!!
>     > So what your doing will allow a somebody the provision to use it
>     they
>     > really want :)
>
>     OK - so, we understand each others position much more clearly and have
>     what we want. Cool. I have found this thread particularly beneficial
>     because you have forced me to crystallise what I felt into hard
>     reason,
>     which is a very useful thing to be forced to do. As a result, I now
>     understand my position much better and have have noticed a couple of
>     further optimisations that I can make :-)
>
>     >
>     > Ok, so finnaly I have achived what I wanted, that is to open up the
>     > API :) and thanks for that.
>     >
>     > State-Balancing
>     > ----------------------------
>     > Ok thats what I was looking for. My concern was that if we use
>     session
>     > affinity can a server get overloaded. Sorry if never asked about
>     this
>     > directly and all credit to you for identifying what my real concern
>     > was wrt session affinity.
>     >
>     > It's cool if we have a mechanism if the ClusterManager is aware
>     of no
>     > of sessions in each node and make sure new requests are
>     redirected to
>     > another node if the current server has already reached the
>     threshold.
>
>     I was already intending for each node to publish a rough indication of
>     the number of session that it carries, because I figured that it would
>     come in useful - we now have a good usecase for it.
>
>     >
>     > Does WADI provide a management API for this kind of thing??
>
>     Not yet - look at the LoadBalancer strategy stuff - it will be
>     extended
>     to hang off here somewhere.
>
>     >
>     > also doe it have some sort of mechanism to get info about a
>     particular
>     > node, no of nodes, no of nodes in a session at given time...etc ????
>
>     ActiveCluster allows each node to maintain a small amount of published
>     data about itself. It can write this data. All other nodes can
>     read it.
>     I use it, currently, to advertise human-readable-node-name, http-port,
>     shutting-down?, etc... - I figure that if if e.g. the number of
>     sessions
>     that you are carrying fluctuates by more than 10% since the last
>     published figure, you publish the new number etc. We want to throttle
>     publishing a little because each publishing causes a 1->all message...
>
>     >
>     > I assume u still expose the listerners in ActiveCluster for
>     interested
>     > parties to get events like node joined, left ...etc
>
>     This area of the code is still undergoing a lot of change, but it
>     should
>     be possible to get the ActiveCluster Cluster object from it and
>     attach a
>     ClusterListener.
>
>     Jules
>
>     >
>     > Regards,
>     >
>     > Rajith.
>     >
>     > On 1/19/06, *Jules Gosnell* <jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>
>     > <mailto: jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>>> wrote:
>     >
>     >     Rajith Attapattu wrote:
>     >
>     >     >
>     >     > Ok, I am not fixed on multiple-active-sessions.
>     >     > But my concern is high availability with
>     single-active-session
>     >     model
>     >     > under high load conditions.
>     >     >
>     >     > As u pointed out,
>     >     > >In the web world, clients commonly throw multiple concurrent
>     >     requests at
>     >     > >clusters, however, if we could assure total affinity, these
>     >     would always
>     >     > >arrive at the same copy, avoiding the chance of a collision
>     >     >
>     >     > However if we use 100% session affinity then the chance of
>     one
>     >     server
>     >     > getting too many hits is possible (due to that being the
>     primary
>     >     and
>     >     > the LB indiscrimantely maintaining session affinity
>     without due
>     >     > consideration for load).
>     >     > Thus a compromise of service quality is inevitable. The
>     service will
>     >     > have to drop requests or degrade the service (provide only
>     some
>     >     of the
>     >     > services which are not expensive).
>     >     >
>     >     > So sometimes the cost of maintaining
>     multiple-active-sessions may be
>     >     > less compared to the exceptional cost that has to be paid
>     with a
>     >     > server crash thus increasing the load within the remaining
>     nodes of
>     >     > the cluster.
>     >
>     >     I haven't heard this argument before - lets take a look at it...
>     >
>     >     I'm not aware of loadbalancers that allow you to ask for
>     e.g. 50%
>     >     affinity - do they exist ? So, lets look at the extremes
>     (100% and
>     >     0%),
>     >     this is often the best way of seeing how something will
>     actually work.
>     >
>     >     100%
>     >     - all requests are routed to the correct node, regardless of
>     >     cluster size
>     >     - no replicants ever need to be deserialised because they are
>     >     never hit
>     >     - if a node collects too many sessions, it may get overloaded
>     >
>     >     0%
>     >     - the amount of extra deserialisation that you will have to
>     >     perform will
>     >     increase in line with the number of nodes in your cluster as the
>     >     chance
>     >     of hitting the one deserialised copy decreases.
>     >     - this extra deserialisation cost will mean that total work
>     done
>     >     for the
>     >     same load will be higher and may overload nodes.
>     >
>     >     So, I guess we want to choose the solution that leads to the
>     least
>     >     work
>     >     being done in the cluster - 100% affinity - and figure out
>     how to
>     >     avoid
>     >     so many clients getting stuck to one node that they overload
>     it -
>     >     solution - state-balancing.
>     >
>     >     Each node needs to be aware of roughly how many sessions the
>     other
>     >     nodes
>     >     are carrying. If it feels that it has more than everyone
>     else, it
>     >     could
>     >     (load-balancer  integration permitting) offload sessions to
>     its peers
>     >     and relocate subsequent requests for them to their new
>     location, or
>     >     simply relocate requests that look like they might create a new
>     >     session
>     >     to peers that are carrying fewer sessions - or use both of these
>     >     algorithms to keep state equally balanced across the cluster.
>     >
>     >     how does that sound ? It shouldn't be too hard in WADI, because
>     >     most of
>     >     the stuff to do this sort of thing is already there...
>     >
>     >     >
>     >     > The cost in terms of money value of loosing revenue due to
>     service
>     >     > un-availability could be higher than providing more memory,
>     >     high-speed
>     >     > network infra which could handle the cost of
>     >     > serialization/desirialization of replicas and the overhead
>     of a
>     >     > distributed locking mechanism without compromising
>     performance.
>     >     >
>     >     > Thats why I said that we should provide both stratergies
>     and the
>     >     > end-user can make an informed decesion based on there
>     business
>     >     > requirments and load conditions within there cluster.
>     >     >
>     >     > We should avoid making those decesions before hand.
>     >     >
>     >
>     >     > Also allowing the idea of configurable active replicas will
>     >     allow the
>     >     > end-user the flexibility of trying out both
>     multiple-active-session
>     >     > and single-active-session models and see what works best
>     for them.
>     >     >
>     >     > I would strongly advocate the idea of a Replication mgt
>     abstraction
>     >     > API especially with some of the ideas Gianny provided on
>     the thread.
>     >     >
>     >     > What do u think about that?? Have I made a case??
>     >
>     >     Hmmm...
>     >
>     >     I don't want to make a battle out of this, but I really
>     cannot see
>     >     any
>     >     advantage to multiple live copies. All you will do is
>     increase the
>     >     work
>     >     that your cluster is having to perform. If you choose 1->few
>     >     replication, you will also miss your session and its copies
>     with
>     >     increasing frequency as you add nodes, creating even more
>     work. If
>     >     you
>     >     choose 1->all replication you will suffer from increasing
>     workload and
>     >     memory requirements on all nodes as you add new ones, and
>     you will
>     >     quite
>     >     quickly hit a ceiling in terms of scale... Affinity is the
>     silver
>     >     bullet
>     >     as far as clustering sessions in concerned. I can't
>     understand why
>     >     anyone would want to run without affinity turned up as high
>     as it
>     >     will go.
>     >
>     >     I am planning to refactor this area in WADI to be pluggable, so
>     >     you will
>     >     have the option of writing something like this if you really
>     want
>     >     it...but I am not convinced that I should provide it...
>     >
>     >     Sorry to put you down again - I don't want to, but when I
>     think it all
>     >     through, I just can't see a good reason to go this route.....
>     >
>     >     Jules
>     >
>     >     >
>     >     > Regards,
>     >     >
>     >     > Rajith.
>     >     >
>     >     >
>     >     > On 1/18/06, *Jules Gosnell* <jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>
>     >     <mailto: jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>>
>     >     > <mailto: jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>
>     >     <mailto:jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>>>> wrote:
>     >     >
>     >     >     Jules Gosnell wrote:
>     >     >
>     >     >     > Oh Rajith - you've got me thinking :-(
>     >     >     >
>     >     >     > I'm not happy with the last answer - lets try again....
>     >     >     >
>     >     >     > lets agree some points :
>     >     >     >
>     >     >     > 1) since changes made to sessions are made in
>     app-space, apps
>     >     >     are not
>     >     >     > written with the expectation that a change collision may
>     >     occur
>     >     >     and the
>     >     >     > container would not be able to avoid such a
>     collision, it must
>     >     >     never
>     >     >     > happen.
>     >     >     >
>     >     >     > 2) in order for a change-collision to occur multiple
>     >     concurrent
>     >     >     > requests/invocations must hit multiple copies of the
>     session
>     >     >     >
>     >     >     > In the web world, clients commonly throw multiple
>     concurrent
>     >     >     requests
>     >     >     > at clusters, however, if we could assure total
>     affinity,
>     >     these would
>     >     >     > always arrive at the same copy, avoiding the chance of a
>     >     collision.
>     >     >     > There are various situations within the web tier
>     that may
>     >     cause the
>     >     >     > breakdown of affinity. Different loadbalancers
>     handle these
>     >     >     situations
>     >     >     > with varying degrees of correctness. I have decided that
>     >     it is safer
>     >     >     > to assume that, whilst affinity is a substantial
>     >     optimisation, it
>     >     >     > cannot be relied on 100%.
>     >     >     >
>     >     >     > So, in the web tier, it is possible for concurrent
>     >     requests for the
>     >     >     > same session to arrive at different session copies.
>     So we
>     >     need a
>     >     >     > pessimistic distributed locking strategy to ensure that
>     >     >     collisions do
>     >     >     > not occur.
>     >     >     >
>     >     >     > In the EJB world, we have more control over the
>     load-balancer,
>     >     >     because
>     >     >     > it is effectively built into the proxy  that we
>     supplied,
>     >     and we
>     >     >     could
>     >     >     > enforce the serial nature of invocations at this
>     point. So it
>     >     >     might be
>     >     >     > possible to move forward on the assumption that we
>     don't need
>     >     >     > pessimistic locking (provided that no-one ever passes a
>     >     session
>     >     >     handle
>     >     >     > to another client).
>     >     >     >
>     >     >     > I'm going to give this a little more thought...
>     >     >     >
>     >     >     > I think the outcome will be that I can avoid some
>     locking
>     >     in the EJB
>     >     >     > world, but need to send the same messages anyway... but
>     >     we'll see.
>     >     >     >
>     >     >     > Thanks for getting me to revisit this,
>     >     >     >
>     >     >     BTW - if we do assume that we can rely on affinity 100% in
>     >     the EJB
>     >     >     tier
>     >     >     then I am still not sure that I see any real advantage in
>     >     holding
>     >     >     multiple active copies of a session. I guess you will
>     have to
>     >     >     explain to
>     >     >     me exactly why you would want to do this.
>     >     >
>     >     >     Finally, the locking system that WADI currently uses will
>     >     only incur
>     >     >     extra work, taking distributed locks, if affinity breaks
>     >     down, so the
>     >     >     cost of applying it to the SFSBs where, we hope for 100%
>     >     affinity,
>     >     >     should be 0.
>     >     >
>     >     >     Jules
>     >     >
>     >     >     >
>     >     >     > Jules
>     >     >     >
>     >     >     >
>     >     >     >
>     >     >     > Jules Gosnell wrote:
>     >     >     >
>     >     >     >> Rajith Attapattu wrote:
>     >     >     >>
>     >     >     >>> More question if you don't mind.
>     >     >     >>>
>     >     >     >>> > 2.) Assuming sombody wants to do session
>     replication (All
>     >     >     >>> > Active) instead of (one Active and "n" backups)
>     is there
>     >     >     provision
>     >     >     >>> > within the WADI api to plug in this stratergy?
>     >     >     >>>
>     >     >     >>> >I'm giving this some thought in terms of SFSB
>     support,
>     >     I'm not
>     >     >     >>> aware of
>     >     >     >>> >similar constraints in the EJB world...
>     >     >     >>>
>     >     >     >>> >I guess we could relax this constraint in the web
>     >     world, but
>     >     >     I am not
>     >     >     >>> >sure that I think that this is a good idea. Can
>     you see
>     >     a way
>     >     >     to do
>     >     >     >>> this
>     >     >     >>> >and maintain spec compliance and performance ?
>     >     >     >>> Is WADI designed primarily for Web?? (bcos u talked
>     >     about being
>     >     >     >>> servlet spec compliant) and u also mention about SFSB
>     >     support.
>     >     >     >>
>     >     >     >>
>     >     >     >>
>     >     >     >> WADI was initially designed for the web - because I saw
>     >     the issues
>     >     >     >> surrounding HttpSession distribution, particularly the
>     >     requirement
>     >     >     >> for a single 'active' session as unresolved in any open
>     >     source
>     >     >     >> offering and I thought it was about time that there
>     was a
>     >     truly
>     >     >     >> compliant solution.
>     >     >     >>
>     >     >     >> It soon became clear that many of the problems faced by
>     >     >     sessions in
>     >     >     >> the web-tier were also faced by sessions in the
>     EJB-tier...
>     >     >     >>
>     >     >     >>> Can we abstract the Replication problem to a more
>     higher
>     >     level and
>     >     >     >>> have the two (or more if there is) stratergies as
>     impls
>     >     of the
>     >     >     >>> replication API that installs as a pluggin by the
>     user.
>     >     >     >>
>     >     >     >>
>     >     >     >>
>     >     >     >> Well, we could, but you would have to convince me that
>     >     SFSBs would
>     >     >     >> benefit from a 'multiple-active-sessions' approach... I
>     >     haven't
>     >     >     given
>     >     >     >> it much thought, but I don't see any advantage - bear
>     >     with me :
>     >     >     >>
>     >     >     >> - in the EJB world, we own the client side proxy.
>     We can
>     >     impose
>     >     >     >> strict affinity. An invocation arriving at a node
>     that is not
>     >     >     >> carrying the primary session copy will be an
>     exceptional
>     >     occurance.
>     >     >     >>
>     >     >     >> - If you go with the 'single-active-session' model,
>     and an
>     >     >     invocation
>     >     >     >> does land on a secondary, you then pay an
>     exceptional cost -
>     >     >     >> deserialisation and promotion (messaging) from
>     secondary to
>     >     >     primary.
>     >     >     >> This is OK, since you are in an exceptional situation.
>     >     >     >>
>     >     >     >> - If you go with the 'multiple-active-sessions'
>     approach you
>     >     >     have two
>     >     >     >> choices regarding deserialisation of replication
>     messages.
>     >     >     >>
>     >     >     >> 1) you can deserialise them as they arrive - bad idea,
>     >     because
>     >     >     >> deserialisation is extremely expensive and most of the
>     >     time these
>     >     >     >> copies will never be used.
>     >     >     >> 2) you can deserialise them lazily - only bother to do
>     >     the work,
>     >     >     >> if/when an invocation arises.
>     >     >     >>
>     >     >     >> Regardless of which you choose (and I hope you would
>     >     choose 2),
>     >     >     you
>     >     >     >> are now in a situation where two copies of a
>     session may
>     >     >     diverge from
>     >     >     >> each other. Lets say you make a change to one, then
>     you
>     >     make a
>     >     >     change
>     >     >     >> to the other, but the replication message from the
>     first
>     >     session
>     >     >     >> arrives at the second session after your second change
>     >     and wipes it
>     >     >     >> out, and the replication message from your second
>     change
>     >     then
>     >     >     >> overwrites the first session with what is now a
>     different
>     >     value
>     >     >     than
>     >     >     >> that carried by the second.... you can detect these
>     issues by
>     >     >     >> versioning, but the best way to protect against them
>     >     occuring (see
>     >     >     >> reasons for needing a pessimistic algorithm below)
>     is by
>     >     having
>     >     >     some
>     >     >     >> form of distributed locking. In effect, the guy
>     with the lock
>     >     >     is the
>     >     >     >> primary and the guy without it the secondary.
>     >     >     >>
>     >     >     >> OK, so now we have a working 'multiple-active-sessions'
>     >     model -
>     >     >     but
>     >     >     >> hold on, it is doing lazy deserialisation and
>     distributed
>     >     locking -
>     >     >     >> it looks very like the 'single-active-session'
>     model....
>     >     >     >>
>     >     >     >> Does that help ?
>     >     >     >>
>     >     >     >>>
>     >     >     >>> We can abstract things like a ReplicationManager that
>     >     >     >>> handles/controls no of replicas etc.. and a
>     >     ReplicatedSession
>     >     >     which
>     >     >     >>> decides wether it's active or passive (backup)
>     based on the
>     >     >     >>> parameters passed to the ReplicatedSessionFactory at
>     >     create time
>     >     >     >>> from the ReplicationManager.
>     >     >     >>
>     >     >     >>
>     >     >     >>
>     >     >     >> sure - and all of these things are already
>     pluggable in WADI.
>     >     >     >>
>     >     >     >>>
>     >     >     >>> The ReplicationManager impl could be the stratergy
>     that
>     >     decides
>     >     >     >>> wether it maintains n of active replicas or 1
>     active and
>     >     n backups
>     >     >     >>> or any other stratergy.
>     >     >     >>
>     >     >     >>
>     >     >     >>
>     >     >     >> Yes it could - but I think that this is still being
>     >     driven by your
>     >     >     >> attachment to the multiple-active-sessions model
>     and I do
>     >     not see
>     >     >     >> that as viable.
>     >     >     >>
>     >     >     >>>
>     >     >     >>> Also the ReplicatedSession could impl stratergies
>     like in
>     >     >     >>> MemoryReplication or PassiveReplication (based on
>     active or
>     >     >     passive)
>     >     >     >>> or anything else. And PassiveReplication can be
>     extended
>     >     to file
>     >     >     >>> based, database backed (not recomended) or
>     anything else.
>     >     >     >>>
>     >     >     >>
>     >     >     >>
>     >     >     >>
>     >     >     >> WADI's replication strategy is already pluggable.
>     We have a
>     >     >     basic DB
>     >     >     >> replication scheme and are working on the in-vm scheme.
>     >     Other
>     >     >     schemes
>     >     >     >> could easily be added.
>     >     >     >>
>     >     >     >>> If we open up the API and let the user choose the
>     >     stratergy they
>     >     >     >>> want then we are delaying our concerns to the user
>     level
>     >     and allow
>     >     >     >>> them to make the decesion.
>     >     >     >>> I am sure we cannot address every situation, and the
>     >     user is the
>     >     >     >>> best to judge about there env.
>     >     >     >>>
>     >     >     >>> But we can always provide some sensible
>     stratergies and
>     >     >     >>> recomendations and use cases around them to make
>     an informed
>     >     >     decesion.
>     >     >     >>>
>     >     >     >>> Then We can leave the decesion to the user about
>     >     >     >>> spec-complient/performance.
>     >     >     >>>
>     >     >     >>> What do u think??
>     >     >     >>
>     >     >     >>
>     >     >     >>
>     >     >     >> Unless you can demonstrate a clear win for a strategy
>     >     that is
>     >     >     >> non-compliant, I would be very hesitant to ship one.
>     >     >     >>
>     >     >     >> WADI is designed so that pretty much everything
>     that you
>     >     might want
>     >     >     >> to plug is pluggable. But the larger the piece that you
>     >     want to
>     >     >     plug
>     >     >     >> in, the more work you would have to do writing it and
>     >     making sure
>     >     >     >> that it did not collide with any other fn-ality.
>     >     >     >>
>     >     >     >>>
>     >     >     >>> >If a request arrives at a secondary, primary and
>     secondary
>     >     >     swap roles
>     >     >     >>> >and processing happens locally.
>     >     >     >>> >If a request arrives on a node with no copy of the
>     >     relevant
>     >     >     >>> session, it
>     >     >     >>> >may be relocated to the primary, or the primary
>     to it.
>     >     >     >>>
>     >     >     >>> 1. Do u plan to have an abstraction around the above
>     >     concerns
>     >     >     as well??
>     >     >     >>>     So we can have impls of different stratergies, So
>     >     people can
>     >     >     >>> decide wether they want to relocate the primary or
>     the
>     >     request.
>     >     >     >>
>     >     >     >>
>     >     >     >>
>     >     >     >> yes - this decision is pluggable.
>     >     >     >>
>     >     >     >>>
>     >     >     >>>     In case of a relocation of either request or
>     session I
>     >     >     assume u
>     >     >     >>> have hidden the impls behind an interface/API sort of
>     >     thing so
>     >     >     ppl
>     >     >     >>> can do different impls of the same stratergies or impl
>     >     their own
>     >     >     >>> stratergy.
>     >     >     >>
>     >     >     >>
>     >     >     >>
>     >     >     >> yes
>     >     >     >>
>     >     >     >>>
>     >     >     >>> 2. In the event of a primary and secondary
>     swapping roles or
>     >     >     having
>     >     >     >>> n of active replicas don't we need some sort of
>     distributed
>     >     >     locking
>     >     >     >>> mechanism.
>     >     >     >>> I heard that in memory locking should be
>     optimistic and
>     >     storage
>     >     >     >>> backed replicas should be pessimistic locking.
>     >     >     >>
>     >     >     >>
>     >     >     >>
>     >     >     >> session locking has to be pessimistic, because changes
>     >     are made by
>     >     >     >> app, not container code. So a collision of changes
>     could
>     >     not be
>     >     >     >> resolved by the container, so it cannot be allowed
>     to happen.
>     >     >     >>
>     >     >     >> WADI contains a distributed locking mechanism
>     within its
>     >     >     Partitioning
>     >     >     >> system. When a copy is promoted, a message will
>     pass from it
>     >     >     >> (containing its version number), to its partition
>     (where it
>     >     >     will take
>     >     >     >> a lock and find the location of the primary), on to the
>     >     primary
>     >     >     >> (where it will compare version numbers), back to the
>     >     secondary
>     >     >     (with
>     >     >     >> a possible update, if its version is out of date)
>     and finally
>     >     >     back to
>     >     >     >> the partition (where the primary's new location will be
>     >     stored and
>     >     >     >> the lock released). local locking will also occur
>     around both
>     >     >     >> secondary and primary whilst they are involved in this
>     >     >     interaction.
>     >     >     >>
>     >     >     >>>
>     >     >     >>> I hope I haven't got the too mixed up :)
>     >     >     >>
>     >     >     >>
>     >     >     >>
>     >     >     >> No, I don' think so, but I do think that you need
>     to take
>     >     a careful
>     >     >     >> look at exactly how you think a
>     mutiple-active-sessions model
>     >     >     might
>     >     >     >> work and whether this would, in fact, be any different
>     >     from the
>     >     >     model
>     >     >     >> that I am proposing.
>     >     >     >>
>     >     >     >>>
>     >     >     >>> Can u please touch on this problem as my knoweldge is
>     >     limited on
>     >     >     >>> this area.
>     >     >     >>
>     >     >     >>
>     >     >     >>
>     >     >     >> is this enough detail ? :-)
>     >     >     >>
>     >     >     >>
>     >     >     >> Jules
>     >     >     >>
>     >     >     >>>
>     >     >     >>> Regards,
>     >     >     >>>
>     >     >     >>> Rajith.
>     >     >     >>>
>     >     >     >>>
>     >     >     >>> On 1/17/06, *Jules Gosnell*
>     <jules@coredevelopers.net <mailto:jules@coredevelopers.net>
>     >     <mailto:jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>>
>     >     >     <mailto: jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>
>     >     <mailto:jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>>>
>     >     >     >>> <mailto: jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>
>     >     <mailto:jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>>
>     >     >     <mailto: jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>
>     >     <mailto:jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>>>>> wrote:
>     >     >     >>>
>     >     >     >>>     Rajith Attapattu wrote:
>     >     >     >>>
>     >     >     >>>     >
>     >     >     >>>     >  Hi,
>     >     >     >>>     >
>     >     >     >>>     > Some of these questions came up after
>     reading the
>     >     thread on
>     >     >     >>> totem.
>     >     >     >>>     > However I started the new thread so that
>     searching is
>     >     >     easy and
>     >     >     >>> also
>     >     >     >>>     > want distract the intense discussions on
>     totem with
>     >     >     out-of-topic
>     >     >     >>>     > questions.
>     >     >     >>>     >
>     >     >     >>>     > Jules Gosnel wrote
>     >     >     >>>     >
>     >     >     >>>     > >This is not something that is really
>     considered a
>     >     >     significant
>     >     >     >>>     saving in
>     >     >     >>>     > >WADI (see my last posting's explanation of why
>     >     you only
>     >     >     want one
>     >     >     >>>     > >'active' copy of a session). WADI will keep
>     >     session backups
>     >     >     >>>     serialised,
>     >     >     >>>     > >to save resources being constantly expended
>     >     deserialising
>     >     >     >>> session
>     >     >     >>>     > >backups that may never be accessed. I guess
>     >     actually,
>     >     >     you could
>     >     >     >>>     consider
>     >     >     >>>     > >that WADI will do a lazy deserialisation in
>     the case
>     >     >     that you
>     >     >     >>> have
>     >     >     >>>     > >outlined, as primary and secondary copies will
>     >     actually
>     >     >     swap
>     >     >     >>>     roles with
>     >     >     >>>     > >attendant serialisation/passivation and
>     >     >     >>> deserialisation/activation
>     >     >     >>>     > >coordinated by messages.
>     >     >     >>>     >
>     >     >     >>>     > >If you are running a reasonable sized cluster (
>     >     e.g. 30
>     >     >     nodes -
>     >     >     >>>     it's all
>     >     >     >>>     > >relative) with a small number of backups
>     >     configured (
>     >     >     e.g. 1),
>     >     >     >>>     then, in
>     >     >     >>>     > >the case of a session affinity brekdown
>     (due to the
>     >     >     leaving of a
>     >     >     >>>     > >primary's node), you have a 1/30 chance
>     that the
>     >     >     request will
>     >     >     >>>     hit the
>     >     >     >>>     > >primary, a 1/30 that you will hit the
>     secondary
>     >     and a 28/30
>     >     >     >>>     that you
>     >     >     >>>     > >will miss :-) So, you are right :-)
>     >     >     >>>     >
>     >     >     >>>     > So just to figure out if I understand this
>     correctly.
>     >     >     >>>     >
>     >     >     >>>     > 1.) WADI only has one active and one-two backups
>     >     at most (I
>     >     >     >>>     assume the
>     >     >     >>>     > no of backups is configurable)
>     >     >     >>>
>     >     >     >>>     replication is under implementation at the
>     moment. Any
>     >     >     number of
>     >     >     >>>     backups
>     >     >     >>>     should be configurable, but the more you have
>     the less
>     >     >     >>> performant you
>     >     >     >>>     are. You trade off safety for speed.
>     >     >     >>>
>     >     >     >>>     >
>     >     >     >>>     > 2.) WADI is built up on the assumption of
>     session
>     >     >     affinity. So
>     >     >     >>> the
>     >     >     >>>     > probability of missing the primary and the
>     secondary
>     >     >     >>> backup(s)  goes
>     >     >     >>>     > up as the cluster grows according to your
>     example
>     >     >     >>>
>     >     >     >>>     WADI will work without session affinity,
>     however, as
>     >     you would
>     >     >     >>> expect,
>     >     >     >>>     this will not perform as well as it might. If you
>     >     switch on
>     >     >     >>> affinity,
>     >     >     >>>     you will drastically cut down the amount of
>     >     request/session
>     >     >     >>> relocation
>     >     >     >>>     and most interactions should become local.
>     >     >     >>>
>     >     >     >>>     Switch off affinity, and of course, your
>     chances of
>     >     hitting a
>     >     >     >>> copy of
>     >     >     >>>     the session will go down. There are a fixed
>     number of
>     >     >     sessions and
>     >     >     >>>     you
>     >     >     >>>     are increasing the number of nodes... If you are
>     >     intending
>     >     >     to use
>     >     >     >>>     an lb
>     >     >     >>>     without affinity, then you should really
>     reconsider. The
>     >     >     costs are
>     >     >     >>>     tiny
>     >     >     >>>     and the gains enormous. Affinity is a standard
>     >     feature on any
>     >     >     >>> serious
>     >     >     >>>     HTTP LB.
>     >     >     >>>
>     >     >     >>>     >
>     >     >     >>>     > 3.) How does WADI handle a situation where
>     there is no
>     >     >     session
>     >     >     >>>     affinity??
>     >     >     >>>
>     >     >     >>>     If a request lands on the primary, processing
>     occurs
>     >     locally.
>     >     >     >>>     If a request arrives at a secondary, primary and
>     >     secondary
>     >     >     swap
>     >     >     >>> roles
>     >     >     >>>     and processing happens locally.
>     >     >     >>>     If a request arrives on a node with no copy of the
>     >     relevant
>     >     >     >>>     session, it
>     >     >     >>>     may be relocated to the primary, or the
>     primary to it.
>     >     >     >>>
>     >     >     >>>     >
>     >     >     >>>     > 4.) Have you compared the overhead of
>     maintaining
>     >     session
>     >     >     >>>     affinity vs
>     >     >     >>>     > having R replicas (all-Active) to service
>     the client.
>     >     >     >>>
>     >     >     >>>     I have worked on impls using both approaches
>     and am
>     >     satisfied
>     >     >     >>> that my
>     >     >     >>>     most recent approach will be the most performant.
>     >     >     >>>
>     >     >     >>>     >
>     >     >     >>>     > >If, however,  you did your deserialisation of
>     >     replicants up
>     >     >     >>>     front and
>     >     >     >>>     > thus avoided further messages when a
>     secondary was
>     >     hit, by
>     >     >     >>>     maintaining
>     >     >     >>>     > >all copies 'active' (I think you would not
>     be spec
>     >     >     compliant
>     >     >     >>> if you
>     >     >     >>>     > did this),
>     >     >     >>>     >
>     >     >     >>>     > 1.) What do u mean by spec here ?? Are u talking
>     >     about
>     >     >     the WADI
>     >     >     >>>     spec?
>     >     >     >>>
>     >     >     >>>     There is no WADI spec :-) - I'm talking about the
>     >     servlet
>     >     >     spec -
>     >     >     >>>     specifically :
>     >     >     >>>
>     >     >     >>>     SRV 7.7.2 - "Within an application marked as
>     >     >     distributable, all
>     >     >     >>>     requests
>     >     >     >>>     that are part of a session must be handled by one
>     >     Java Virtual
>     >     >     >>>     Machine1
>     >     >     >>>     ( JVM ) at a time." and "Containers must
>     notify any
>     >     session
>     >     >     >>> attributes
>     >     >     >>>     implementing the HttpSessionActivationListener
>     during
>     >     >     migration
>     >     >     >>> of a
>     >     >     >>>     session. They must notify listeners of
>     passivation
>     >     prior to
>     >     >     >>>     serialization of a session, and of activation
>     after
>     >     >     >>>     deserialization of a
>     >     >     >>>     session."
>     >     >     >>>
>     >     >     >>>     These two constraints make it, IMHO, much more
>     difficult
>     >     >     to try
>     >     >     >>>     implementing any system that maintains multiple
>     >     'active', or
>     >     >     >>>     'primary'
>     >     >     >>>     copies of a session. The system needs to be
>     >     absolutely clear
>     >     >     >>> where the
>     >     >     >>>     single 'active' copy is at any one time, in
>     order to
>     >     remain
>     >     >     >>> compliant.
>     >     >     >>>     To ensure that activation/passivation
>     semantics work
>     >     OK,
>     >     >     only this
>     >     >     >>>     session may be activated, whilst the other
>     'secondary'
>     >     >     copies are
>     >     >     >>>     passivated. By leaving the secondaries in
>     serialised
>     >     form,
>     >     >     you save
>     >     >     >>>     further cycles and arrive at WADI's current
>     design.
>     >     >     >>>
>     >     >     >>>     >
>     >     >     >>>     > 2.) Assuming sombody wants to do session
>     >     replication (All
>     >     >     >>>     > Active) instead of (one Active and "n"
>     backups) is
>     >     there
>     >     >     >>> provision
>     >     >     >>>     > within the WADI api to plug in this stratergy?
>     >     >     >>>
>     >     >     >>>     I'm giving this some thought in terms of SFSB
>     >     support, I'm not
>     >     >     >>>     aware of
>     >     >     >>>     similar constraints in the EJB world...
>     >     >     >>>
>     >     >     >>>     I guess we could relax this constraint in the web
>     >     world, but I
>     >     >     >>> am not
>     >     >     >>>     sure that I think that this is a good idea.
>     Can you
>     >     see a
>     >     >     way to
>     >     >     >>>     do this
>     >     >     >>>     and maintain spec compliance and performance ?
>     >     >     >>>
>     >     >     >>>     >
>     >     >     >>>     > If u remeber we talked about extention points
>     >     within WADI.
>     >     >     >>>     >
>     >     >     >>>     > 1.) Is there a doc that describes WADI
>     architecture
>     >     >     >>>
>     >     >     >>>     Not as yet, just a website with various resources
>     >     hanging
>     >     >     of it.
>     >     >     >>>     WADI is
>     >     >     >>>     still relatively young. The best source of
>     architecture
>     >     >     info is the
>     >     >     >>>     conversations that we have been having.
>     >     >     >>>
>     >     >     >>>     >
>     >     >     >>>     > 2.) Is there a doc that describes these
>     extention
>     >     points
>     >     >     and how
>     >     >     >>>     to do
>     >     >     >>>     > it?? (Looking for a little more info than
>     the API
>     >     doc)
>     >     >     >>>
>     >     >     >>>     WADI is put together using Spring. You just check
>     >     out the
>     >     >     >>> javadoc and
>     >     >     >>>     plug the pojos together. A lot of what we have
>     been
>     >     talking
>     >     >     >>> about is
>     >     >     >>>     architectural design and not implemented
>     (although the
>     >     >     >>>     primary/secondary
>     >     >     >>>     stuff is all in and working).
>     >     >     >>>
>     >     >     >>>     regards,
>     >     >     >>>
>     >     >     >>>
>     >     >     >>>     Jules
>     >     >     >>>
>     >     >     >>>     >
>     >     >     >>>     > Thanks,
>     >     >     >>>     >
>     >     >     >>>     > Rajith.
>     >     >     >>>     >
>     >     >     >>>
>     >     >     >>>
>     >     >     >>>
>     >     >     >>>     --
>     >     >     >>>     "Open Source is a self-assembling organism.
>     You dangle a
>     >     >     piece of
>     >     >     >>>     string into a super-saturated solution and a whole
>     >     >     operating-system
>     >     >     >>>     crystallises out around it."
>     >     >     >>>
>     >     >     >>>     /**********************************
>     >     >     >>>     * Jules Gosnell
>     >     >     >>>     * Partner
>     >     >     >>>     * Core Developers Network (Europe)
>     >     >     >>>     *
>     >     >     >>>     *    www.coredevelopers.net
>     <http://www.coredevelopers.net>
>     >     <http://www.coredevelopers.net <http://www.coredevelopers.net>>
>     >     >     < http://www.coredevelopers.net> <
>     >     http://www.coredevelopers.net>
>     >     >     >>>     *
>     >     >     >>>     * Open Source Training & Support.
>     >     >     >>>     **********************************/
>     >     >     >>>
>     >     >     >>>
>     >     >     >>
>     >     >     >>
>     >     >     >
>     >     >     >
>     >     >
>     >     >
>     >     >     --
>     >     >     "Open Source is a self-assembling organism. You dangle a
>     >     piece of
>     >     >     string into a super-saturated solution and a whole
>     >     operating-system
>     >     >     crystallises out around it."
>     >     >
>     >     >     /**********************************
>     >     >     * Jules Gosnell
>     >     >     * Partner
>     >     >     * Core Developers Network (Europe)
>     >     >     *
>     >     >     *    www.coredevelopers.net
>     <http://www.coredevelopers.net> <http://www.coredevelopers.net>
>     >     < http://www.coredevelopers.net>
>     >     >     *
>     >     >     * Open Source Training & Support.
>     >     >     **********************************/
>     >     >
>     >     >
>     >
>     >
>     >     --
>     >     "Open Source is a self-assembling organism. You dangle a
>     piece of
>     >     string into a super-saturated solution and a whole
>     operating-system
>     >     crystallises out around it."
>     >
>     >     /**********************************
>     >     * Jules Gosnell
>     >     * Partner
>     >     * Core Developers Network (Europe)
>     >     *
>     >     *     www.coredevelopers.net <http://www.coredevelopers.net>
>     <http://www.coredevelopers.net>
>     >     *
>     >     * Open Source Training & Support.
>     >     **********************************/
>     >
>     >
>
>
>     --
>     "Open Source is a self-assembling organism. You dangle a piece of
>     string into a super-saturated solution and a whole operating-system
>     crystallises out around it."
>
>     /**********************************
>     * Jules Gosnell
>     * Partner
>     * Core Developers Network (Europe)
>     *
>     *    www.coredevelopers.net <http://www.coredevelopers.net>
>     *
>     * Open Source Training & Support.
>     **********************************/
>
>


-- 
"Open Source is a self-assembling organism. You dangle a piece of
string into a super-saturated solution and a whole operating-system
crystallises out around it."

/**********************************
 * Jules Gosnell
 * Partner
 * Core Developers Network (Europe)
 *
 *    www.coredevelopers.net
 *
 * Open Source Training & Support.
 **********************************/


Mime
View raw message