jackrabbit-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sbarriba" <sbarr...@yahoo.co.uk>
Subject RE: Concurrent Bottleneck within JackRabbit due to Lock Contention
Date Tue, 01 Jul 2008 17:04:04 GMT
Hi Marcel,
Thanks for the response. The issue is just about contention- although the app server doesn't
lock up it reaches a critical point of contention after which the thread pool grows rapidly
(to 400+) and context switching results in virtually 0 throughput for up to 2 hours (with
our application).

The issue is primarily a JRockit one. We've run the following tests with the same hardware,
same application, same load 20 thread load test (fetching 3000 JCR nodes per request):

Weblogic 9.2 with JRockit (jrockit_1.5.0_12)
--------------------------------------------
Throughput under load is < 1 per second
After a few seconds always 19 threads stuck in 
"Blocked trying to get lock: EDU/oswego/cs/dl/util/concurrent"
CPU Usage 50% to 60%
Conclusion: Application is lock bound

Tomcat + JRockit (jrockit_1.5.0_12)
------------------------------------
As above. Lock bound.

Tomcat + Sun JVM 
----------------
Through-put 15+ per second
No threads listing "Blocked trying to get lock: EDU/oswego/cs/dl/util/concurrent"
Threads in a mixture of areas including Lucene, MySql, app code (as expected)
CPU Usage 90%+ 
Conclusion: Application is now CPU bound

We're currently testing the latest JRockit version which appears to improve throughput by
a significant factor and suffers less from the contention issue.

Regards,
Shaun



-----Original Message-----
From: Marcel Reutegger [mailto:marcel.reutegger@gmx.net] 
Sent: 01 July 2008 09:28
To: users@jackrabbit.apache.org
Subject: Re: Concurrent Bottleneck within JackRabbit due to Lock Contention

Hi Shaun,

so far we did not receive reports about this kind of contention. might be 
specific to JRockit. what version of JRockit is this? Are you able to test your 
application with a JVM from a different vendor? e.g. Sun.

this is really just contention, right? or does jackrabbit lock up completely, 
which would indicate a dead lock.

regards
  marcel

sbarriba wrote:
> Hi all,
> With an application which makes 'heavy' usage of JackRabbit we see its
> throughput throttled by JackRabbits usage of 
> 
> org/apache/jackrabbit/core/state/DefaultISMLocking.acquireReadLock
> 
> Under load we can easily see 400+ threads (in Weblogic 9.2) all fighting to
> acquire a read lock. Even under very light load a thread dump will show
> multiple threads in the acquireReadLock method.
> 
> We've tried all possible configurations to attempt to alleviate this
> bottleneck but I'm wondering if this is:
> 
> a) a fundamental bottleneck with JackRabbit, or
> a) an expected bottleneck in JackRabbit (required to proect cache usage),
> which implies we're trying to drive JackRabbit too hard? 
> 
> There are NO write operations happening, just reads hence my slight surprise
> we're seeing such contention.
> 
> I'm sending this post after multiple days of trying the things outlined
> below.
> 
> Regards,
> Shaun
> 
> 
> Background to Post
> -------------------
> 
> This post follows my previous posts trying to understand the locking
> strategy. We've tried the following, without success, to improve throughput:
> 
> -	Modified DefaultISMLocking strategy using the patch included with
> https://issues.apache.org/jira/browse/JCR-1334
> -	Modified DefaultISMLocking strategy using the “noLockHack” option
> -	The latest JackRabbit 1.4.4 library
> -	FineGrainedISMLocking
> 
> We've even tried the following JRockit specifics:
> 
> -	JVM –XnoOpt  option
> -	JVM  -XXdisableFatSpin option
> 
> In all cases the contention exists.
> 
> 
> -----Original Message-----
> From: sbarriba [mailto:sbarriba@yahoo.co.uk] 
> Sent: 27 June 2008 17:30
> To: users@jackrabbit.apache.org
> Subject: RE: Locking issues with XAItemStateManager - help appreciated
> 
> Hi all,
> Can anyone help me understand why a JackRabbit based web-app that only reads
> the repository would end up with a large number of threads locked when there
> are no writer threads under load? 
> 
> We're struggling a little to get to the bottom of this.
> All help appreciated.
> Regards,
> Shaun
> 
> "[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default
> (self-tuning)'" id=17 idx=0x4c tid=26655 prio=1 alive, in native, blocked,
> daemon
>     -- Blocked trying to get lock:
> EDU/oswego/cs/dl/util/concurrent/WriterPreferenceReadWriteLock$ReaderLock@0x
> 36cd5700[unlocked]
>     at pthread_cond_wait@@GLIBC_2.3.2+170(:0)@0x376f708a7a
>     at tsiWaitForSignalForever+58(:0)@0x2a95712169
>     at tsiWaitForSignal+39(:0)@0x2a957121c4
>     at tsiWaitForLockSignal+39(:0)@0x2a9571229e
>     at tsWaitForJavaLockSignal+31(:0)@0x2a9571234d
>     at RJNI_jrockit_vm_Threads_waitForUnblockSignal+14(:0)@0x2a95708e6c
>     at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
>     at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1630)[optimized]
>     at jrockit/vm/Locks.lockFat(Locks.java:1731)[optimized]
>     at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1277)[optimized]
>     at jrockit/vm/Locks.monitorEnter(Locks.java:2389)[inlined]
>     at jrockit/vm/Locks.monitorEnterForced(Locks.java:872)[optimized]
>     at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
>     at jrockit/vm/Locks.monitorEnterUnmatched(Ljava/lang/Object;)V(Native
> Method)
>     at
> EDU/oswego/cs/dl/util/concurrent/WriterPreferenceReadWriteLock$ReaderLock....a
> c
> quire()V(Unknown Source)[optimized]
>     at
> org/apache/jackrabbit/core/state/DefaultISMLocking$ReadLockImpl.<init>(Defau
> ltISMLocking.java:103)[inlined]
>     at
> org/apache/jackrabbit/core/state/DefaultISMLocking$ReadLockImpl.<init>(Defau
> ltISMLocking.java:97)[inlined]
>     at
> org/apache/jackrabbit/core/state/DefaultISMLocking.acquireReadLock(DefaultIS
> MLocking.java:65)[optimized]
>     at
> org/apache/jackrabbit/core/state/SharedItemStateManager.acquireReadLock(Shar
> edItemStateManager.java:1438)[inlined]
>     at
> org/apache/jackrabbit/core/state/SharedItemStateManager.hasItemState(SharedI
> temStateManager.java:269)[optimized]
>     at
> org/apache/jackrabbit/core/state/LocalItemStateManager.hasItemState(LocalIte
> mStateManager.java:178)[inlined]
>     at
> org/apache/jackrabbit/core/state/XAItemStateManager.hasItemState(XAItemState
> Manager.java:252)[optimized]
>     at
> org/apache/jackrabbit/core/state/SessionItemStateManager.getItemState(Sessio
> nItemStateManager.java:174)[optimized]
>     at
> org/apache/jackrabbit/core/ItemManager.createItemInstance(ItemManager.java:5
> 64)[inlined]
>     at
> org/apache/jackrabbit/core/ItemManager.getItem(ItemManager.java:395)[inlined
> ]
>     at
> org/apache/jackrabbit/core/LazyItemIterator.prefetchNext(LazyItemIterator....j
> a
> va:90)[inlined]
>     at
> org/apache/jackrabbit/core/LazyItemIterator.next(LazyItemIterator.java:203)[
> optimized]
>     ^-- Holding lock: org/apache/jackrabbit/core/ItemManager@0x8e7b568[thin
> lock]
> -----Original Message-----
> From: KÖLL Claus [mailto:C.KOELL@TIROL.GV.AT] 
> Sent: 25 June 2008 16:18
> To: users@jackrabbit.apache.org
> Subject: AW: Locking issues with XAItemStateManager - help appreciated
> 
> Hi Shaun,
> 
> In my case the problem was that the applicationserver (websphere)
> has changed the thread between acquiring writelock and the downgrade to a
> readlock
> so we are running into deadlock.
> 
> Have you tested the patched DefaultISMLocking Class ?
> 
> -----Ursprüngliche Nachricht-----
> Von: sbarriba [mailto:sbarriba@yahoo.co.uk] 
> Gesendet: Mittwoch, 25. Juni 2008 12:52
> An: users@jackrabbit.apache.org
> Betreff: RE: Locking issues with XAItemStateManager - help appreciated
> 
> 
> Apologies for the multitude of emails.
> 
> .......would switching from DefaultISMLocking to the FineGrainedISMLocking
> implementation help the situation?
> 
> I believe this can be done with the following config within the workspace
> config.
> 
> <ISMLocking class="[ClassName]"/>
> 
> Regards,
> Shaun
> 
> -----Original Message-----
> From: sbarriba [mailto:sbarriba@yahoo.co.uk] 
> Sent: 25 June 2008 11:35
> To: users@jackrabbit.apache.org
> Subject: RE: Locking issues with XAItemStateManager - help appreciated
> 
> Hi Claus,
> Thanks for the quick response.
> 
> We're not intentionally using an XA transaction. We're using Spring +
> Transaction Manager + Hibernate elsewhere to satisfy the request.
> 
> .......so a few quick questions:
> 
> * Is there a way to force JackRabbit not to use XA? What are the alternative
> ItemStateManager(s)?
> * Does using XA cause more locking?
> 
> Thanks in advance,
> Regards,
> Shaun
> 
> -----Original Message-----
> From: KÖLL Claus [mailto:C.KOELL@TIROL.GV.AT] 
> Sent: 25 June 2008 11:24
> To: users@jackrabbit.apache.org
> Subject: AW: Locking issues with XAItemStateManager - help appreciated
> 
> hi,
> 
> do you use a xa transaction ?
> take a look @ https://issues.apache.org/jira/browse/JCR-1334
> i see in your stack that the lock comes from acquireReadLock()
> this was also in my issue
> feel free do test the PatchedDefaultISMLocking.java by Marcel
> 
> BR,
> claus
> 
> -----Ursprüngliche Nachricht-----
> Von: sbarriba [mailto:sbarriba@yahoo.co.uk] 
> Gesendet: Mittwoch, 25. Juni 2008 12:16
> An: users@jackrabbit.apache.org
> Betreff: Locking issues with XAItemStateManager - help appreciated
> 
> Hi all,
> 
> As follow up to a previous thread we're seeing lots and lots of contention
> around the following lock. We're using Weblogic 9.1 / JRocket 27.4.0 (1.5).
> 
>  
> 
> Even with very little load on the app a thread dump shows active threads at
> exactly this point. As the concurrent load increases the contention
> increases until the app is continually thrashing on these locks and stops
> responding.
> 
>  
> 
> Is there a way to configure JackRabbit to reduce the amount of locking?
> 
> For example, I note the use of XAItemStateManager in the stack - is there an
> alternative ItemStateManager implementation which requires less locking?
> 
>  
> 
> All help appreciated.
> 
>  
> 
> Regards,
> 
> Shaun
> 
>  
> 
>  
> 
> "[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default
> (self-tuning)'" id=133 idx=0x20c tid=6426 prio=5 alive, in native, daemon
> 
>     at jrockit/vm/Locks.monitorEnterUnmatched(Ljava/lang/Object;)V(Native
> Method)
> 
>     at
> EDU/oswego/cs/dl/util/concurrent/WriterPreferenceReadWriteLock$ReaderLock.....
> ..
> a
> c
> quire()V(Unknown Source)
> 
>     at
> org/apache/jackrabbit/core/state/DefaultISMLocking$ReadLockImpl.<init>(Defau
> ltISMLocking.java:103)
> 
>     at
> org/apache/jackrabbit/core/state/DefaultISMLocking$ReadLockImpl.<init>(Defau
> ltISMLocking.java:97)
> 
>     at
> org/apache/jackrabbit/core/state/DefaultISMLocking.acquireReadLock(DefaultIS
> MLocking.java:65)
> 
>     at
> org/apache/jackrabbit/core/state/SharedItemStateManager.acquireReadLock(Shar
> edItemStateManager.java:1438)[inlined]
> 
>     at
> org/apache/jackrabbit/core/state/SharedItemStateManager.getItemState(SharedI
> temStateManager.java:237)[optimized]
> 
>     at
> org/apache/jackrabbit/core/state/LocalItemStateManager.getPropertyState(Loca
> lItemStateManager.java:118)[inlined]
> 
>     at
> org/apache/jackrabbit/core/state/LocalItemStateManager.getItemState(LocalIte
> mStateManager.java:150)[inlined]
> 
>     at
> org/apache/jackrabbit/core/state/XAItemStateManager.getItemState(XAItemState
> Manager.java:226)[optimized]
> 
>     ^-- Holding lock:
> org/apache/jackrabbit/core/state/XAItemStateManager@0x3ea1ae88[thin lock]
> 
>     at
> org/apache/jackrabbit/core/state/SessionItemStateManager.getItemState(Sessio
> nItemStateManager.java:175)[optimized]
> 
>     at
> org/apache/jackrabbit/core/ItemManager.createItemInstance(ItemManager.java:5
> 64)[inlined]
> 
>     at
> org/apache/jackrabbit/core/ItemManager.getItem(ItemManager.java:395)[inlined
> ]
> 
>     at
> org/apache/jackrabbit/core/LazyItemIterator.prefetchNext(LazyItemIterator.....
> ..
> j
> a
> va:90)[inlined]
> 
>     at
> org/apache/jackrabbit/core/LazyItemIterator.next(LazyItemIterator.java:203)[
> optimized]
> 
>     ^-- Holding lock: org/apache/jackrabbit/core/ItemManager@0x3ea1a4c0[thin
> lock]
> 
>  
> 
> 
> 
> 
> 
> 



Mime
View raw message