axis-java-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajith Attapattu" <>
Subject Re: [Axis2] Adding ClusterManager code the the codebase
Date Tue, 06 Feb 2007 22:23:30 GMT
Hello Sanjaya,

Thanks for the reply, comments marked with [RA]

Rajith Attapattu
Red Hat.

On 2/5/07, Sanjaya Karunasena <> wrote:
> Hi Rajith,
>  Please find my comments below marked with [SK]
> Thanks
> Sanjaya
> On Monday 05 February 2007 20:28, Rajith Attapattu wrote:
> > Hello Sanjaya,
> >
> > comments are inline marked with [RA]
> >
>  > Rajith Attapattu
> > Red Hat.
> >
> >
> > In a distributed scenario having the ability to handle concurrency is
> >
> > > important. How do you tackle conflicting updates? There should be a
> > > to rollback/commit. In otherwords the interface should be
> > > transactionable. Also
> > > having the ability to distinguish between read-only vs read/write
> > > to
> > > the context is an opportunity to apply some optimizations to the
> > > implementation.
> >
> > [RA] This is where locking  comes into picture. There are 2 different
> > of locking mechanisms, pesimistic locking and optimistic locking.
> > Most databases use pesimistic locking and is expensive and slow, while
> > optimistic locking is relatively cheap. Negotiating concurrency issues
> > best handled by the underlying infrastructure. Apache Tribes (the impl I
> > used and is using total ordering) and Richochet (garuntees are lower
> > VS) are both low level reliable multicasting infrastructure and have no
> > support for locking.
> > WADI from codehaus specializes in locking. But for now we decided
> > back to go without any locking for the following reasons.
> >
> > WADI is written for session replication and is difficult to adapt as is
> > the context replication scenario we have with out some modifications.
> > Sanjiva and I debated on this issue for a while and agreed that we live
> > with the concurrency issues for the time being.
> > What we can do is to try to utilise WADI to provide locking support for
> > implementation as the second step.
> [SK] Locking, which comes under pessimistic concurrency control is good
> databases due to the high probability of conflicts. Optimistic concurrency
> work best when there are high number of reads with intermittent writes.
> which way to go depends on what do we expect in the context. We can put
> constrains or let the end user to configure it. But I don't think this is
> good place to over engineer things. We should keep this very simple. I
> this is similar to the HTTPSession. We don't expect application developers
> use the HTTPSession as their in memory database. Am I right?

[RA] Yes, pessimistic locking is expensive and slow. If we really want to do
locking, then we should do optimistic locking (WADI supports this).
But for the first phase I would like to go without locking. Tomcat
clustering is built on Tribes and they have no distributed locking.
As you correctly pointed out the contexts are like the HTTPSession and
shouldn't be abused by the Web Service developer to shove anything and
everything into it and do frequent writes. This should be clearly stated in

The expectation is to use some simple state (mostly read only with
intermittent writes), just like a HTTPSession.

However there is a difference btw the HTTPSession case and the context case
that makes things a bit hard. With HTTPSession if we use session affinity
a.k.a sticky sessions then we make sure requests for the same session is
routed to the same node. So even w/o locking we can get rid of the
concurrency issues. The replication happens merely for failover.

But with Axis2 (if HTTP is not used as transport) we cannot do session
afinity. So we might face concurrency issues. Unless we use Synapse as a
Load Balancer that knows the interpret the serviceGroupId as the session

>  >
> > Detecting conflicts and rolling back is best decided at the
> > level  (for example  WADI)  rather than at the application level.
> > The logic for rolling back is complicated and if a rollback occurs we
> > to notify the web service client. Currently there is no standard WS way
> > doing that.
> > (Not sure WS-Context has anything regarding this)
> >
> > We can have dirty reads (cheap) and do locking only for updates. But
> [SK] From the application developer point of view Axis2 itself is
> infrastructure. Obviously we should not pull him in to the picture.
> application code which implements a refine requirement can provide some
> (like whether its OK to have dirty reads).

[RA] The only way to avoid  dirty reads completly is to do pessimistic
locking :(
For now lets keep things simple. Later let us try to come up with a way to
layer some locking mechanism on top of the cluster impl.

> >
> > I suggest that this interface should provide some means to do
> >
> > > that too.
> >
> > [RA] As I said, the application should not be figuring out wether to
> > rollback or not, the infrastructure should.
> > so it should be the reverse. there should be a way for the
> > to notify the client about a rollback.
> >
> [SK] Agreed. But it will be good to shoot for some thing where there is no

> need to rollback.
> > I think the intention of abstracting the cluster manager to an interface
> >
> > > to
> > > provide extensibility so that we can have different cluster manager
> > > implementations based on different group communication techniques. But
> > > think right now the interface has introduced more coupling.
> > >
> > > Following is an approach which could reduce coupling and enable more
> > > flexibility and extensibility.
> > >
> > > Do we have the ability to register a listener to the Context? What I
> > > is
> > > an instance of a class which implements an interface ContextListner.
> > > ContextListner has one method "notify" which take a parameter of
> > > type "Event". The ClusterManager interface can extend the
> > > (BTW, Should ClusterManger be an abstract class?)
> >
> > [RA} unfortunately your replication points can be outside of the context

> > heirachy.
> [SK] I am sorry, I did not understand this. What else do you want to
> replicate. In a cluster what we need is state information to be
> There are two places where usually an application developer maintain
> the session and the database. I guess with the Axis2 context option he get
> third place. HTTPSession and databases should not be our problem. Let that
> be handled by the infrastructure providers of those two. But yes, we need
> have a story as to how to get end to end state replication working.

[RA]Sorry for not being clear. What I meant was the decesion points to
replicate (the exact replication points)
may fall outside of the context heirachy. That's why I think we need the
ClusterManager (atleast for the time being).
For example update state.
We can do this on per updateBasis (we can have a event listener like u said
that creates event for an add/update/remove proeprty),
instead, it's easy to call updateState at the end of the invocation from the
engine or anywhere deem suitable and then figure out the net state change
and then replicate. So we need more flexibility than events.
Does that make sense ?

> > For example instead of replicating every  add/remove change happening
> > inside an invocation we can choose to replicate the net result (change
> > state) at the end of the invocation. I think this is more
efficient.  The
> > decesion to replicate something should not be limited to be taken within
> > the context heirachy only.
> [SK] Yes this is efficient but leads to more conflicts. Why not let the
> user (Application developer) decide that since he knows best the behavior
> his application implementation.
[RA] If you look at HTTPSession, the service author have no way of saying
how or when to replicate.
If the service author calls abuses this privilege then  we have a
performance issue on hand.
Also I am not sure how the above way will lead to less conflicts? Can u
explain more?

> > There should be flexibility to call the ClusterManager from anywhere. So
> > context listener is not sufficient.
> >
> [SK] Well this sounds bit dangerous :-). I am sorry, I don't have enough
> context here.

[RA] Sorry what I meant was the decesion to replicate or points of
replication can be anywhere within the system.
(Not the state, the state is only encapsulated inside the ctx heirachy).

> > Furthermore your coupling is only reduced to the cluster manager. I also
> > didn't like the prospect of calling cluster manager from all over the
> > place, but given the current design of the context heirachy we have no
> > choice.
> >
> [SK] Well this actually could provide a good extension point to even
> application developers. In application server implementation of
> there is a way to register a listener. I have used it many times to clean
> stale data in the session. But I may be wrong in this context.

> > The other more cleaner approach that I proposed a while back was to use
> > Aspects. We can use aspects to capture replication points using load
> > weaving in a more non intrusive manner. This way the context code is not
> > littered with calls to the Cluster manager. I do have experimental
> > on my local copy, but it still needs more work.
> >
> [SK] Aspect oriented approach work best when there are cross cutting
> I am not sure about the performance implications. I have used it for
> tracing, throttling, etc. But certainly worth evaluating.
[RA] I don't think it's a big problem with performance, but dims pointed out
other issues where byte code manipulation may create to other non technical
realted issues.
My point cuts are the replication points. AOP will allow me to intercept
these points in a non intrusive manner.

> > This way, in a particular node when there is change in the Context the
> >
> > > ClusterManager instance in that node get notify. Then that
> > > instance can mulitcast (reliable mulitcasting) the change to other
> > > ClusterManger instances running in other nodes. Since each
> > > instance has a handler to the Context it can apply the change locally.
> > > the
> > > reliable multicast is configured to keep the "total order" in message
> > > ordering you don't have to worry about the conflicts due to
> >
> > [RA] This is what happens currently.  Context creations are notified
> > immediatly and state updates are notified at the end of  an invocation.
> > Tribes which is a  total odering reliable multicast then does the work.
> > However this will not prevent concurrency issues completely. Unless we
> > pesimistic locking we would still experiance dirty reads and potential
> > write/write conflicts.
> >
> [SK] Well, total ordering to work, you need to maintain total ordering of
> events. In otherwords all the events occurred need to be notified. We can
> check with some carefull analysis whether causal ordering could work here.
> that way things will be more efficient.

[RA] I made mistake, Phlips confirmed that he hasn't implemented total
ordering. Even JGroups doesn't have it.
There is evs4j from Guglielmo Lichtner which claims total odering. The
others seem to have different levels of garuntees.

> Different group communication implementations are geared towards different
> attributes. Application developer should have the flexibility to choose
> one to go with. So we may want to enable different ClusterManger
> implementations to come in using different group communication
> implementations.

[RA] It is simple, all you need to do is to add the following to your
<cluster class="org.apache.axis2.cluster.tribes.ClusterManager" />

You can put any implementation that implements the ClusterManager interface.

> However things can be futher simplify if we can make an assumption of a
> session. I think that's some thing you can enforce and then manage simple
> complexities like transparent node recovery. With little feedback from the
> application developer this could also lead to flexible scalable
> implementation.
[RA]  see my above comment on session affinity and why it may not work for

> I am just throwing out ideas so that we can trash out things. I suppose
> is not information overloading :-).

[RA] not at all, I really like this conversation.

> ---------------------------------------------------------------------
>  To unsubscribe, e-mail:
> For additional commands, e-mail:

View raw message