geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jules Gosnell <ju...@coredevelopers.net>
Subject Re: WADI clustering
Date Thu, 19 Jan 2006 19:48:49 GMT
Rajith Attapattu wrote:

>  
> Ok, I am not fixed on multiple-active-sessions.
> But my concern is high availability with single-active-session model 
> under high load conditions.
>  
> As u pointed out,
> >In the web world, clients commonly throw multiple concurrent requests at
> >clusters, however, if we could assure total affinity, these would always
> >arrive at the same copy, avoiding the chance of a collision
>  
> However if we use 100% session affinity then the chance of one server 
> getting too many hits is possible (due to that being the primary and 
> the LB indiscrimantely maintaining session affinity without due 
> consideration for load).
> Thus a compromise of service quality is inevitable. The service will 
> have to drop requests or degrade the service (provide only some of the 
> services which are not expensive).
>  
> So sometimes the cost of maintaining multiple-active-sessions may be 
> less compared to the exceptional cost that has to be paid with a 
> server crash thus increasing the load within the remaining nodes of 
> the cluster.

I haven't heard this argument before - lets take a look at it...

I'm not aware of loadbalancers that allow you to ask for e.g. 50% 
affinity - do they exist ? So, lets look at the extremes (100% and 0%), 
this is often the best way of seeing how something will actually work.

100%
- all requests are routed to the correct node, regardless of cluster size
- no replicants ever need to be deserialised because they are never hit
- if a node collects too many sessions, it may get overloaded

0%
- the amount of extra deserialisation that you will have to perform will 
increase in line with the number of nodes in your cluster as the chance 
of hitting the one deserialised copy decreases.
- this extra deserialisation cost will mean that total work done for the 
same load will be higher and may overload nodes.

So, I guess we want to choose the solution that leads to the least work 
being done in the cluster - 100% affinity - and figure out how to avoid 
so many clients getting stuck to one node that they overload it - 
solution - state-balancing.

Each node needs to be aware of roughly how many sessions the other nodes 
are carrying. If it feels that it has more than everyone else, it could 
(load-balancer  integration permitting) offload sessions to its peers 
and relocate subsequent requests for them to their new location, or 
simply relocate requests that look like they might create a new session 
to peers that are carrying fewer sessions - or use both of these 
algorithms to keep state equally balanced across the cluster.

how does that sound ? It shouldn't be too hard in WADI, because most of 
the stuff to do this sort of thing is already there...

>  
> The cost in terms of money value of loosing revenue due to service 
> un-availability could be higher than providing more memory, high-speed 
> network infra which could handle the cost of 
> serialization/desirialization of replicas and the overhead of a 
> distributed locking mechanism without compromising performance.
>  
> Thats why I said that we should provide both stratergies and the 
> end-user can make an informed decesion based on there business 
> requirments and load conditions within there cluster.
>  
> We should avoid making those decesions before hand.
>  

> Also allowing the idea of configurable active replicas will allow the 
> end-user the flexibility of trying out both multiple-active-session 
> and single-active-session models and see what works best for them.
>  
> I would strongly advocate the idea of a Replication mgt abstraction 
> API especially with some of the ideas Gianny provided on the thread.
>  
> What do u think about that?? Have I made a case??

Hmmm...

I don't want to make a battle out of this, but I really cannot see any 
advantage to multiple live copies. All you will do is increase the work 
that your cluster is having to perform. If you choose 1->few 
replication, you will also miss your session and its copies with 
increasing frequency as you add nodes, creating even more work. If you 
choose 1->all replication you will suffer from increasing workload and 
memory requirements on all nodes as you add new ones, and you will quite 
quickly hit a ceiling in terms of scale... Affinity is the silver bullet 
as far as clustering sessions in concerned. I can't understand why 
anyone would want to run without affinity turned up as high as it will go.

I am planning to refactor this area in WADI to be pluggable, so you will 
have the option of writing something like this if you really want 
it...but I am not convinced that I should provide it...

Sorry to put you down again - I don't want to, but when I think it all 
through, I just can't see a good reason to go this route.....

Jules

>  
> Regards,
>  
> Rajith.
>
>  
> On 1/18/06, *Jules Gosnell* <jules@coredevelopers.net 
> <mailto:jules@coredevelopers.net>> wrote:
>
>     Jules Gosnell wrote:
>
>     > Oh Rajith - you've got me thinking :-(
>     >
>     > I'm not happy with the last answer - lets try again....
>     >
>     > lets agree some points :
>     >
>     > 1) since changes made to sessions are made in app-space, apps
>     are not
>     > written with the expectation that a change collision may occur
>     and the
>     > container would not be able to avoid such a collision, it must
>     never
>     > happen.
>     >
>     > 2) in order for a change-collision to occur multiple concurrent
>     > requests/invocations must hit multiple copies of the session
>     >
>     > In the web world, clients commonly throw multiple concurrent
>     requests
>     > at clusters, however, if we could assure total affinity, these would
>     > always arrive at the same copy, avoiding the chance of a collision.
>     > There are various situations within the web tier that may cause the
>     > breakdown of affinity. Different loadbalancers handle these
>     situations
>     > with varying degrees of correctness. I have decided that it is safer
>     > to assume that, whilst affinity is a substantial optimisation, it
>     > cannot be relied on 100%.
>     >
>     > So, in the web tier, it is possible for concurrent requests for the
>     > same session to arrive at different session copies. So we need a
>     > pessimistic distributed locking strategy to ensure that
>     collisions do
>     > not occur.
>     >
>     > In the EJB world, we have more control over the load-balancer,
>     because
>     > it is effectively built into the proxy  that we supplied, and we
>     could
>     > enforce the serial nature of invocations at this point. So it
>     might be
>     > possible to move forward on the assumption that we don't need
>     > pessimistic locking (provided that no-one ever passes a session
>     handle
>     > to another client).
>     >
>     > I'm going to give this a little more thought...
>     >
>     > I think the outcome will be that I can avoid some locking in the EJB
>     > world, but need to send the same messages anyway... but we'll see.
>     >
>     > Thanks for getting me to revisit this,
>     >
>     BTW - if we do assume that we can rely on affinity 100% in the EJB
>     tier
>     then I am still not sure that I see any real advantage in holding
>     multiple active copies of a session. I guess you will have to
>     explain to
>     me exactly why you would want to do this.
>
>     Finally, the locking system that WADI currently uses will only incur
>     extra work, taking distributed locks, if affinity breaks down, so the
>     cost of applying it to the SFSBs where, we hope for 100% affinity,
>     should be 0.
>
>     Jules
>
>     >
>     > Jules
>     >
>     >
>     >
>     > Jules Gosnell wrote:
>     >
>     >> Rajith Attapattu wrote:
>     >>
>     >>> More question if you don't mind.
>     >>>
>     >>> > 2.) Assuming sombody wants to do session replication (All
>     >>> > Active) instead of (one Active and "n" backups) is there
>     provision
>     >>> > within the WADI api to plug in this stratergy?
>     >>>
>     >>> >I'm giving this some thought in terms of SFSB support, I'm not
>     >>> aware of
>     >>> >similar constraints in the EJB world...
>     >>>
>     >>> >I guess we could relax this constraint in the web world, but
>     I am not
>     >>> >sure that I think that this is a good idea. Can you see a way
>     to do
>     >>> this
>     >>> >and maintain spec compliance and performance ?
>     >>> Is WADI designed primarily for Web?? (bcos u talked about being
>     >>> servlet spec compliant) and u also mention about SFSB support.
>     >>
>     >>
>     >>
>     >> WADI was initially designed for the web - because I saw the issues
>     >> surrounding HttpSession distribution, particularly the requirement
>     >> for a single 'active' session as unresolved in any open source
>     >> offering and I thought it was about time that there was a truly
>     >> compliant solution.
>     >>
>     >> It soon became clear that many of the problems faced by
>     sessions in
>     >> the web-tier were also faced by sessions in the EJB-tier...
>     >>
>     >>> Can we abstract the Replication problem to a more higher level and
>     >>> have the two (or more if there is) stratergies as impls of the
>     >>> replication API that installs as a pluggin by the user.
>     >>
>     >>
>     >>
>     >> Well, we could, but you would have to convince me that SFSBs would
>     >> benefit from a 'multiple-active-sessions' approach... I haven't
>     given
>     >> it much thought, but I don't see any advantage - bear with me :
>     >>
>     >> - in the EJB world, we own the client side proxy. We can impose
>     >> strict affinity. An invocation arriving at a node that is not
>     >> carrying the primary session copy will be an exceptional occurance.
>     >>
>     >> - If you go with the 'single-active-session' model, and an
>     invocation
>     >> does land on a secondary, you then pay an exceptional cost -
>     >> deserialisation and promotion (messaging) from secondary to
>     primary.
>     >> This is OK, since you are in an exceptional situation.
>     >>
>     >> - If you go with the 'multiple-active-sessions' approach you
>     have two
>     >> choices regarding deserialisation of replication messages.
>     >>
>     >> 1) you can deserialise them as they arrive - bad idea, because
>     >> deserialisation is extremely expensive and most of the time these
>     >> copies will never be used.
>     >> 2) you can deserialise them lazily - only bother to do the work,
>     >> if/when an invocation arises.
>     >>
>     >> Regardless of which you choose (and I hope you would choose 2),
>     you
>     >> are now in a situation where two copies of a session may
>     diverge from
>     >> each other. Lets say you make a change to one, then you make a
>     change
>     >> to the other, but the replication message from the first session
>     >> arrives at the second session after your second change and wipes it
>     >> out, and the replication message from your second change then
>     >> overwrites the first session with what is now a different value
>     than
>     >> that carried by the second.... you can detect these issues by
>     >> versioning, but the best way to protect against them occuring (see
>     >> reasons for needing a pessimistic algorithm below) is by having
>     some
>     >> form of distributed locking. In effect, the guy with the lock
>     is the
>     >> primary and the guy without it the secondary.
>     >>
>     >> OK, so now we have a working 'multiple-active-sessions' model -
>     but
>     >> hold on, it is doing lazy deserialisation and distributed locking -
>     >> it looks very like the 'single-active-session' model....
>     >>
>     >> Does that help ?
>     >>
>     >>>
>     >>> We can abstract things like a ReplicationManager that
>     >>> handles/controls no of replicas etc.. and a ReplicatedSession
>     which
>     >>> decides wether it's active or passive (backup) based on the
>     >>> parameters passed to the ReplicatedSessionFactory at create time
>     >>> from the ReplicationManager.
>     >>
>     >>
>     >>
>     >> sure - and all of these things are already pluggable in WADI.
>     >>
>     >>>
>     >>> The ReplicationManager impl could be the stratergy that decides
>     >>> wether it maintains n of active replicas or 1 active and n backups
>     >>> or any other stratergy.
>     >>
>     >>
>     >>
>     >> Yes it could - but I think that this is still being driven by your
>     >> attachment to the multiple-active-sessions model and I do not see
>     >> that as viable.
>     >>
>     >>>
>     >>> Also the ReplicatedSession could impl stratergies like in
>     >>> MemoryReplication or PassiveReplication (based on active or
>     passive)
>     >>> or anything else. And PassiveReplication can be extended to file
>     >>> based, database backed (not recomended) or anything else.
>     >>>
>     >>
>     >>
>     >>
>     >> WADI's replication strategy is already pluggable. We have a
>     basic DB
>     >> replication scheme and are working on the in-vm scheme. Other
>     schemes
>     >> could easily be added.
>     >>
>     >>> If we open up the API and let the user choose the stratergy they
>     >>> want then we are delaying our concerns to the user level and allow
>     >>> them to make the decesion.
>     >>> I am sure we cannot address every situation, and the user is the
>     >>> best to judge about there env.
>     >>>
>     >>> But we can always provide some sensible stratergies and
>     >>> recomendations and use cases around them to make an informed
>     decesion.
>     >>>
>     >>> Then We can leave the decesion to the user about
>     >>> spec-complient/performance.
>     >>>
>     >>> What do u think??
>     >>
>     >>
>     >>
>     >> Unless you can demonstrate a clear win for a strategy that is
>     >> non-compliant, I would be very hesitant to ship one.
>     >>
>     >> WADI is designed so that pretty much everything that you might want
>     >> to plug is pluggable. But the larger the piece that you want to
>     plug
>     >> in, the more work you would have to do writing it and making sure
>     >> that it did not collide with any other fn-ality.
>     >>
>     >>>
>     >>> >If a request arrives at a secondary, primary and secondary
>     swap roles
>     >>> >and processing happens locally.
>     >>> >If a request arrives on a node with no copy of the relevant
>     >>> session, it
>     >>> >may be relocated to the primary, or the primary to it.
>     >>>
>     >>> 1. Do u plan to have an abstraction around the above concerns
>     as well??
>     >>>     So we can have impls of different stratergies, So people can
>     >>> decide wether they want to relocate the primary or the request.
>     >>
>     >>
>     >>
>     >> yes - this decision is pluggable.
>     >>
>     >>>
>     >>>     In case of a relocation of either request or session I
>     assume u
>     >>> have hidden the impls behind an interface/API sort of thing so
>     ppl
>     >>> can do different impls of the same stratergies or impl their own
>     >>> stratergy.
>     >>
>     >>
>     >>
>     >> yes
>     >>
>     >>>
>     >>> 2. In the event of a primary and secondary swapping roles or
>     having
>     >>> n of active replicas don't we need some sort of distributed
>     locking
>     >>> mechanism.
>     >>> I heard that in memory locking should be optimistic and storage
>     >>> backed replicas should be pessimistic locking.
>     >>
>     >>
>     >>
>     >> session locking has to be pessimistic, because changes are made by
>     >> app, not container code. So a collision of changes could not be
>     >> resolved by the container, so it cannot be allowed to happen.
>     >>
>     >> WADI contains a distributed locking mechanism within its
>     Partitioning
>     >> system. When a copy is promoted, a message will pass from it
>     >> (containing its version number), to its partition (where it
>     will take
>     >> a lock and find the location of the primary), on to the primary
>     >> (where it will compare version numbers), back to the secondary
>     (with
>     >> a possible update, if its version is out of date) and finally
>     back to
>     >> the partition (where the primary's new location will be stored and
>     >> the lock released). local locking will also occur around both
>     >> secondary and primary whilst they are involved in this
>     interaction.
>     >>
>     >>>
>     >>> I hope I haven't got the too mixed up :)
>     >>
>     >>
>     >>
>     >> No, I don' think so, but I do think that you need to take a careful
>     >> look at exactly how you think a mutiple-active-sessions model
>     might
>     >> work and whether this would, in fact, be any different from the
>     model
>     >> that I am proposing.
>     >>
>     >>>
>     >>> Can u please touch on this problem as my knoweldge is limited on
>     >>> this area.
>     >>
>     >>
>     >>
>     >> is this enough detail ? :-)
>     >>
>     >>
>     >> Jules
>     >>
>     >>>
>     >>> Regards,
>     >>>
>     >>> Rajith.
>     >>>
>     >>>
>     >>> On 1/17/06, *Jules Gosnell* <jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>
>     >>> <mailto: jules@coredevelopers.net
>     <mailto:jules@coredevelopers.net>>> wrote:
>     >>>
>     >>>     Rajith Attapattu wrote:
>     >>>
>     >>>     >
>     >>>     >  Hi,
>     >>>     >
>     >>>     > Some of these questions came up after reading the thread on
>     >>> totem.
>     >>>     > However I started the new thread so that searching is
>     easy and
>     >>> also
>     >>>     > want distract the intense discussions on totem with
>     out-of-topic
>     >>>     > questions.
>     >>>     >
>     >>>     > Jules Gosnel wrote
>     >>>     >
>     >>>     > >This is not something that is really considered a
>     significant
>     >>>     saving in
>     >>>     > >WADI (see my last posting's explanation of why you only
>     want one
>     >>>     > >'active' copy of a session). WADI will keep session backups
>     >>>     serialised,
>     >>>     > >to save resources being constantly expended deserialising
>     >>> session
>     >>>     > >backups that may never be accessed. I guess actually,
>     you could
>     >>>     consider
>     >>>     > >that WADI will do a lazy deserialisation in the case
>     that you
>     >>> have
>     >>>     > >outlined, as primary and secondary copies will actually
>     swap
>     >>>     roles with
>     >>>     > >attendant serialisation/passivation and
>     >>> deserialisation/activation
>     >>>     > >coordinated by messages.
>     >>>     >
>     >>>     > >If you are running a reasonable sized cluster ( e.g. 30
>     nodes -
>     >>>     it's all
>     >>>     > >relative) with a small number of backups configured (
>     e.g. 1),
>     >>>     then, in
>     >>>     > >the case of a session affinity brekdown (due to the
>     leaving of a
>     >>>     > >primary's node), you have a 1/30 chance that the
>     request will
>     >>>     hit the
>     >>>     > >primary, a 1/30 that you will hit the secondary and a 28/30
>     >>>     that you
>     >>>     > >will miss :-) So, you are right :-)
>     >>>     >
>     >>>     > So just to figure out if I understand this correctly.
>     >>>     >
>     >>>     > 1.) WADI only has one active and one-two backups at most (I
>     >>>     assume the
>     >>>     > no of backups is configurable)
>     >>>
>     >>>     replication is under implementation at the moment. Any
>     number of
>     >>>     backups
>     >>>     should be configurable, but the more you have the less
>     >>> performant you
>     >>>     are. You trade off safety for speed.
>     >>>
>     >>>     >
>     >>>     > 2.) WADI is built up on the assumption of session
>     affinity. So
>     >>> the
>     >>>     > probability of missing the primary and the secondary
>     >>> backup(s)  goes
>     >>>     > up as the cluster grows according to your example
>     >>>
>     >>>     WADI will work without session affinity, however, as you would
>     >>> expect,
>     >>>     this will not perform as well as it might. If you switch on
>     >>> affinity,
>     >>>     you will drastically cut down the amount of request/session
>     >>> relocation
>     >>>     and most interactions should become local.
>     >>>
>     >>>     Switch off affinity, and of course, your chances of hitting a
>     >>> copy of
>     >>>     the session will go down. There are a fixed number of
>     sessions and
>     >>>     you
>     >>>     are increasing the number of nodes... If you are intending
>     to use
>     >>>     an lb
>     >>>     without affinity, then you should really reconsider. The
>     costs are
>     >>>     tiny
>     >>>     and the gains enormous. Affinity is a standard feature on any
>     >>> serious
>     >>>     HTTP LB.
>     >>>
>     >>>     >
>     >>>     > 3.) How does WADI handle a situation where there is no
>     session
>     >>>     affinity??
>     >>>
>     >>>     If a request lands on the primary, processing occurs locally.
>     >>>     If a request arrives at a secondary, primary and secondary
>     swap
>     >>> roles
>     >>>     and processing happens locally.
>     >>>     If a request arrives on a node with no copy of the relevant
>     >>>     session, it
>     >>>     may be relocated to the primary, or the primary to it.
>     >>>
>     >>>     >
>     >>>     > 4.) Have you compared the overhead of maintaining session
>     >>>     affinity vs
>     >>>     > having R replicas (all-Active) to service the client.
>     >>>
>     >>>     I have worked on impls using both approaches and am satisfied
>     >>> that my
>     >>>     most recent approach will be the most performant.
>     >>>
>     >>>     >
>     >>>     > >If, however,  you did your deserialisation of replicants
up
>     >>>     front and
>     >>>     > thus avoided further messages when a secondary was hit, by
>     >>>     maintaining
>     >>>     > >all copies 'active' (I think you would not be spec
>     compliant
>     >>> if you
>     >>>     > did this),
>     >>>     >
>     >>>     > 1.) What do u mean by spec here ?? Are u talking about
>     the WADI
>     >>>     spec?
>     >>>
>     >>>     There is no WADI spec :-) - I'm talking about the servlet
>     spec -
>     >>>     specifically :
>     >>>
>     >>>     SRV 7.7.2 - "Within an application marked as
>     distributable, all
>     >>>     requests
>     >>>     that are part of a session must be handled by one Java Virtual
>     >>>     Machine1
>     >>>     ( JVM ) at a time." and "Containers must notify any session
>     >>> attributes
>     >>>     implementing the HttpSessionActivationListener during
>     migration
>     >>> of a
>     >>>     session. They must notify listeners of passivation prior to
>     >>>     serialization of a session, and of activation after
>     >>>     deserialization of a
>     >>>     session."
>     >>>
>     >>>     These two constraints make it, IMHO, much more difficult
>     to try
>     >>>     implementing any system that maintains multiple 'active', or
>     >>>     'primary'
>     >>>     copies of a session. The system needs to be absolutely clear
>     >>> where the
>     >>>     single 'active' copy is at any one time, in order to remain
>     >>> compliant.
>     >>>     To ensure that activation/passivation semantics work OK,
>     only this
>     >>>     session may be activated, whilst the other 'secondary'
>     copies are
>     >>>     passivated. By leaving the secondaries in serialised form,
>     you save
>     >>>     further cycles and arrive at WADI's current design.
>     >>>
>     >>>     >
>     >>>     > 2.) Assuming sombody wants to do session replication (All
>     >>>     > Active) instead of (one Active and "n" backups) is there
>     >>> provision
>     >>>     > within the WADI api to plug in this stratergy?
>     >>>
>     >>>     I'm giving this some thought in terms of SFSB support, I'm not
>     >>>     aware of
>     >>>     similar constraints in the EJB world...
>     >>>
>     >>>     I guess we could relax this constraint in the web world, but I
>     >>> am not
>     >>>     sure that I think that this is a good idea. Can you see a
>     way to
>     >>>     do this
>     >>>     and maintain spec compliance and performance ?
>     >>>
>     >>>     >
>     >>>     > If u remeber we talked about extention points within WADI.
>     >>>     >
>     >>>     > 1.) Is there a doc that describes WADI architecture
>     >>>
>     >>>     Not as yet, just a website with various resources hanging
>     of it.
>     >>>     WADI is
>     >>>     still relatively young. The best source of architecture
>     info is the
>     >>>     conversations that we have been having.
>     >>>
>     >>>     >
>     >>>     > 2.) Is there a doc that describes these extention points
>     and how
>     >>>     to do
>     >>>     > it?? (Looking for a little more info than the API doc)
>     >>>
>     >>>     WADI is put together using Spring. You just check out the
>     >>> javadoc and
>     >>>     plug the pojos together. A lot of what we have been talking
>     >>> about is
>     >>>     architectural design and not implemented (although the
>     >>>     primary/secondary
>     >>>     stuff is all in and working).
>     >>>
>     >>>     regards,
>     >>>
>     >>>
>     >>>     Jules
>     >>>
>     >>>     >
>     >>>     > Thanks,
>     >>>     >
>     >>>     > Rajith.
>     >>>     >
>     >>>
>     >>>
>     >>>
>     >>>     --
>     >>>     "Open Source is a self-assembling organism. You dangle a
>     piece of
>     >>>     string into a super-saturated solution and a whole
>     operating-system
>     >>>     crystallises out around it."
>     >>>
>     >>>     /**********************************
>     >>>     * Jules Gosnell
>     >>>     * Partner
>     >>>     * Core Developers Network (Europe)
>     >>>     *
>     >>>     *    www.coredevelopers.net
>     <http://www.coredevelopers.net> < http://www.coredevelopers.net>
>     >>>     *
>     >>>     * Open Source Training & Support.
>     >>>     **********************************/
>     >>>
>     >>>
>     >>
>     >>
>     >
>     >
>
>
>     --
>     "Open Source is a self-assembling organism. You dangle a piece of
>     string into a super-saturated solution and a whole operating-system
>     crystallises out around it."
>
>     /**********************************
>     * Jules Gosnell
>     * Partner
>     * Core Developers Network (Europe)
>     *
>     *    www.coredevelopers.net <http://www.coredevelopers.net>
>     *
>     * Open Source Training & Support.
>     **********************************/
>
>


-- 
"Open Source is a self-assembling organism. You dangle a piece of
string into a super-saturated solution and a whole operating-system
crystallises out around it."

/**********************************
 * Jules Gosnell
 * Partner
 * Core Developers Network (Europe)
 *
 *    www.coredevelopers.net
 *
 * Open Source Training & Support.
 **********************************/


Mime
View raw message