ws-sandesha-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amila Suriarachchi" <amilasuriarach...@gmail.com>
Subject Re: Sandesha2 synchronization and dead lock handling.
Date Wed, 29 Oct 2008 13:44:08 GMT
On Tue, Oct 28, 2008 at 11:26 AM, Amila Suriarachchi <
amilasuriarachchi@gmail.com> wrote:

>
>
> On Fri, Oct 24, 2008 at 7:54 PM, Thomas McKiernan <MCKIERNA@uk.ibm.com>wrote:
>
>> I think Andrew was saying that there is already an implicit order for the
>> beans.
>
>
> yes this is correct. most of the transactions follows a same order except
> some transactions.
> What about making an explicit order so that we can change the code whenever
> found an exception.
>
> we can have an order like RMSBean, RMDBean, MessageBean, SenderBean.
>
>
>> The point is people keep changing the code and breaking that order.
>> Therefore this is not quite a hack but an attempt to make explicit what is
>> already implied in the open src code.
>>
>> It would not work for the distributed case but, assuming each distributed
>> thread was behaving correctly i.e. obeying the correct locking order then
>> this would not be an issue.
>> And this would catch any thread disobeying that ordering.
>
>
Please see the attached patch I have added some similar thing as you have
mentioned. Here I could not locate the InvokerBean.

Then I proceed with this change. I found a lot of places where it does not
follow the given order. In order to proceed I put another set of methods
called locks which are called earlier to make the sequence correct.

In this way I found a lot of places which does not follow this order. Of
course this may not be the correct order to follow.

Based on the above observations we can possibly have following options to
address this problem.

1. Although there are dead locks sequence get finished. I think this is the
way Sandesha2 is written. if there is an error it can roll back and proceed.
But lot of people may not like it.

2. To implement a possible deadlock detection system (As suggested above )
and remove them either using some dummy locks or reformatting the code. Here
reformatting the code may not feasible all the time.

Any thoughts?

thanks,
Amila.

>
> As I understood in your method what you try to do is to have an explicit
> order of accessing the
> beans and raise an exception if one transaction make an exception.
>
> In this case also don't we have to go and change the transaction to have it
> correct order (the explicit order you have define). Otherwise it keep on
> failing.
>
> But this will certainly improve the transaction debugging.
>
> thanks,
> Amila.
>
>
>
>>
>>
>> ----------------------------------
>> Thomas McKiernan
>>
>> WebSphere Messaging Development,
>> IBM United Kingdom Limited
>>
>> Internal Phone: 248241
>> External Phone: +44 (0)1962 818241
>> Mobile: +44 (0)789 1737497
>> Email: MCKIERNA@uk.ibm.com
>>
>> Mail Point 211, IBM, Hursley Park, Winchester, Hampshire, England, SO21
>> 2JN
>>
>>
>> Caminante, no hay camino
>> Se hace camino al andar.
>> ("Walker, there is no path; the path is made by walking.")  Antonio
>> Machado
>>
>>
>>
>> From:
>> "Amila Suriarachchi" <amilasuriarachchi@gmail.com>
>> To:
>> Thomas McKiernan/UK/IBM@IBMGB
>> Cc:
>> Andrew K Gatford/UK/IBM@IBMGB, "sandesha-dev@ws.apache.org"
>> <sandesha-dev@ws.apache.org>
>> Date:
>> 24/10/2008 14:06
>> Subject:
>> Re: Sandesha2 synchronization and dead lock handling.
>>
>>
>>
>>
>>
>> On Fri, Oct 24, 2008 at 3:55 PM, Thomas McKiernan <MCKIERNA@uk.ibm.com>
>> wrote:
>> How about a lock manager impl independent of any particular store's impl.
>> It could be abstract if necessary.
>>
>> Basically, this has a hierarchy of classes (beans) hard coded.
>> If you use a store to access a bean then the store impl's tran calls into
>> the independent lock manager.
>>
>> I feel this is a kind hack for the problem. And also as Andrew has
>> mentioned this won't work in a
>> distributed environment.
>> For me the correct solution is to go through all the transactions and make
>> an order of which
>> transactions access the beans. But apparently  this is also seems to be
>> difficult since a lot
>> of transactions has start and commits.
>> So have to think bit more.
>>
>> thanks,
>> Amila.
>>
>>
>>
>>
>> Any attempt to enlist outside of the locking hierarchy results in a hard
>> runtime error and a rollback of the tran.
>>
>> Is this too naive?
>>
>> ----------------------------------
>> Thomas McKiernan
>>
>> WebSphere Messaging Development,
>> IBM United Kingdom Limited
>>
>> Internal Phone: 248241
>> External Phone: +44 (0)1962 818241
>> Mobile: +44 (0)789 1737497
>> Email: MCKIERNA@uk.ibm.com
>>
>> Mail Point 211, IBM, Hursley Park, Winchester, Hampshire, England, SO21
>> 2JN
>>
>>
>> Caminante, no hay camino
>> Se hace camino al andar.
>> ("Walker, there is no path; the path is made by walking.")  Antonio
>> Machado
>>
>>
>>
>> From:
>> Andrew K Gatford/UK/IBM@IBMGB
>> To:
>> "Amila Suriarachchi" <amilasuriarachchi@gmail.com>
>> Cc:
>> "sandesha-dev@ws.apache.org" <sandesha-dev@ws.apache.org>
>> Date:
>> 24/10/2008 11:07
>> Subject:
>> Re: Sandesha2 synchronization and dead lock handling.
>>
>>
>>
>> I went through similar pain when implementing a StorageManager and
>> encountered a number of deadlocks similar to the ones that you describe.
>> What I have gradually done is eliminate these in both the InMemory store
>> and my store by changing the ordering the beans were taken in.
>>
>> In general the beans are taken in this order.
>>
>> RMSBean or RMDBean followed by
>> SenderBean or InvokerBean.
>>
>> In cases where both the RMSBean and RMDBean are locked, they tend to be
>> taken in that order - RMS followed by RMD.
>> The one thing that I do know is that it is fairly easy to introduce new
>> deadlocks by slightly altering the order that beans are read.
>>
>> The one question I have is how does the jdbc store handle multiple threads
>>
>> accessing multiple sequences, or even a single sequence, but with multiple
>>
>> threads sending multiple requests.  From my experience this is where we
>> have found a lot of problems in the InMemory store and I expect to be even
>>
>> more painful with a jdbc store.
>>
>> Andrew Gatford
>> Technical Project Lead
>> Websphere ESB Foundation Technologies
>> Hursley MP211
>> IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21 2JN
>> Telephone :
>> Internal (7) 245743
>> External 01962 815743
>> Internet : gatfora@uk.ibm.com
>>
>>
>>
>> From:
>> "Amila Suriarachchi" <amilasuriarachchi@gmail.com>
>> To:
>> "sandesha-dev@ws.apache.org" <sandesha-dev@ws.apache.org>
>> Date:
>> 24/10/2008 10:30
>> Subject:
>> Sandesha2 synchronization and dead lock handling.
>>
>>
>>
>> hi all,
>>
>> This is regarding the issue [1].
>>
>> First of all as I learned Sandesha2 uses different beans to keep the state
>>
>> of the sequence and the messages. In a dual channel mode
>> different threads can access these beans and update them concurrently. So
>> the synchronization of these beans done by using the
>> storage level transactions. Therefore Sandesha2 needs an storage which
>> supports isolated transactions.
>>
>> To synchronize these beans the transactions must be completely isolated.
>> i.e It should not allow simultaneous reads of
>> same record from different transactions. Therefore I think the problem I
>> saw on[1] because not isolating the transactions properly.
>>
>> Then I increased the transaction isolation to fix the above problem. It
>> fixed that problem but results in dead locks.
>> The reason I believe for this dead locks is that different transactions
>> try to access the data base tables in different order.
>> But unfortunately I could not fix the issue.
>>
>> Normally these types of dead locks are prevented by accessing resources in
>>
>> same order. Does Sandesha2 follows such a order or any
>> other technique?
>>
>> Or is there any other reason for this dead locks and synchronization
>> problems? Can someone
>> have a better idea of Sandesha2 Design shed some light on this?
>>
>> thanks,
>> Amila.
>>
>>
>> [1] http://issues.apache.org/jira/browse/SANDESHA2-179
>> --
>> Amila Suriarachchi
>> WSO2 Inc.
>> blog: http://amilachinthaka.blogspot.com/
>>
>>
>>
>>
>>
>>
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number
>> 741598.
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: sandesha-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: sandesha-dev-help@ws.apache.org
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number
>> 741598.
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Amila Suriarachchi
>> WSO2 Inc.
>> blog: http://amilachinthaka.blogspot.com/
>>
>>
>>
>>
>>
>>
>>
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number
>> 741598.
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Amila Suriarachchi
> WSO2 Inc.
> blog: http://amilachinthaka.blogspot.com/
>



-- 
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/

Mime
View raw message