jakarta-jcs-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Cooke <mpcoo...@lineone.net>
Subject Re: Thread deadlock in CacheEventQueue class
Date Fri, 14 Jan 2005 10:29:39 GMT
Aaron,

As far as I know the deadlock was not recoverable and did not occur when 
iterating the local memory cache but rather on put operations triggered 
by a master/remotecache update.
The problem once it occurred stopped all updates occurring I think it 
was blocking the local puts triggered from the mastercache by holding a 
lock.
I agree automated tests are difficult, I tried, it required high load 
from a stress tool difficult to setup in an automated test.

On an unrelated note, we tried the threadpool timeout code 1.1.3dev 
yesterday. It seemed to work in testing but once we staged to production 
it turned out that deletions were no longer getting propagated from the 
remote/mastercache resulting in content never updating on the client 
machines. I tried various config combinations, with threadpool, without 
threadpool and with/without removeOnPut enabled. I got the same error in 
every case on the mastercache (failure to enqueue remove events) and had 
to roll back to the version prior to thread pools. Unfortunately during 
testing we had only checked the initial items were received, we never 
tested items were updated - so this was partially my own fault.
Having retested the code now on test servers i definitely can't get the 
remove events working in the 1.1.3dev build.

If you haven't tested that deletion are propagating with the new 
threadpool code i suggest you do so, if you have tested this could you 
send me a copy of your working cache.ccf and remote.cache.ccf in case i 
missed something.

Many Thanks,
Matt.

Aaron Smuts wrote:

> I suspect that it was the remove / put problem that I solved in 1.1.2.
> It was the only known problem of that sort.  It may be due to iteration
> but I thought this was resolved.  Iteration is very problematic on an
> active collection.  There is no really good way to do it.  We basically
> copy the keys in fail fast mode.  Were you seeing unrecoverable deadlock
> or just temporary lags?
> 
> I'll try some more strenuous testing.  The big problem is that it is
> very difficult to create automated tests for the remote and lateral
> caches.  I mainly rely on the tester script.  I have a random function
> that will beat the hell out of the cache. . . . .  
> 
> Aaron
> 
> 
>>-----Original Message-----
>>From: Matthew Cooke [mailto:matthew@connextra.com]
>>Sent: Tuesday, January 11, 2005 8:52 AM
>>To: turbine-jcs-user@jakarta.apache.org
>>Subject: RE: Thread deadlock in CacheEventQueue class
>>
>>We had major problems when we used the remotecache configuration in
> 
> push
> 
>>mode rather than delete mode (removeOnPut).
>>
>>At the time we strongly suspected a deadlock caused by the local puts
>>placing cached items back into the local memory caches when they
>>recieved async put events from the remotecache.
>>
>>Could the deadlock we were seeing when using remotePuts have been the
>>same thing? We are quite keen to try remotePuts again as this
>>functionality would be very useful to us (would allow iteration over
>>whole caches on individual machines). As the deadlock was only
> 
> occuring
> 
>>under high load we don't have a reliable way to test.
>>
>>Any advice appreciated!
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail:
> 
> turbine-jcs-user-unsubscribe@jakarta.apache.org
> 
>>For additional commands, e-mail:
> 
> turbine-jcs-user-help@jakarta.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org


Mime
View raw message