jackrabbit-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcel Reutegger <marcel.reuteg...@gmx.net>
Subject Re: Concurrent Bottleneck within JackRabbit due to Lock Contention
Date Tue, 01 Jul 2008 08:28:22 GMT
Hi Shaun,

so far we did not receive reports about this kind of contention. might be 
specific to JRockit. what version of JRockit is this? Are you able to test your 
application with a JVM from a different vendor? e.g. Sun.

this is really just contention, right? or does jackrabbit lock up completely, 
which would indicate a dead lock.

regards
  marcel

sbarriba wrote:
> Hi all,
> With an application which makes 'heavy' usage of JackRabbit we see its
> throughput throttled by JackRabbits usage of 
> 
> org/apache/jackrabbit/core/state/DefaultISMLocking.acquireReadLock
> 
> Under load we can easily see 400+ threads (in Weblogic 9.2) all fighting to
> acquire a read lock. Even under very light load a thread dump will show
> multiple threads in the acquireReadLock method.
> 
> We've tried all possible configurations to attempt to alleviate this
> bottleneck but I'm wondering if this is:
> 
> a) a fundamental bottleneck with JackRabbit, or
> a) an expected bottleneck in JackRabbit (required to proect cache usage),
> which implies we're trying to drive JackRabbit too hard? 
> 
> There are NO write operations happening, just reads hence my slight surprise
> we're seeing such contention.
> 
> I'm sending this post after multiple days of trying the things outlined
> below.
> 
> Regards,
> Shaun
> 
> 
> Background to Post
> -------------------
> 
> This post follows my previous posts trying to understand the locking
> strategy. We've tried the following, without success, to improve throughput:
> 
> -	Modified DefaultISMLocking strategy using the patch included with
> https://issues.apache.org/jira/browse/JCR-1334
> -	Modified DefaultISMLocking strategy using the “noLockHack” option
> -	The latest JackRabbit 1.4.4 library
> -	FineGrainedISMLocking
> 
> We've even tried the following JRockit specifics:
> 
> -	JVM –XnoOpt  option
> -	JVM  -XXdisableFatSpin option
> 
> In all cases the contention exists.
> 
> 
> -----Original Message-----
> From: sbarriba [mailto:sbarriba@yahoo.co.uk] 
> Sent: 27 June 2008 17:30
> To: users@jackrabbit.apache.org
> Subject: RE: Locking issues with XAItemStateManager - help appreciated
> 
> Hi all,
> Can anyone help me understand why a JackRabbit based web-app that only reads
> the repository would end up with a large number of threads locked when there
> are no writer threads under load? 
> 
> We're struggling a little to get to the bottom of this.
> All help appreciated.
> Regards,
> Shaun
> 
> "[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default
> (self-tuning)'" id=17 idx=0x4c tid=26655 prio=1 alive, in native, blocked,
> daemon
>     -- Blocked trying to get lock:
> EDU/oswego/cs/dl/util/concurrent/WriterPreferenceReadWriteLock$ReaderLock@0x
> 36cd5700[unlocked]
>     at pthread_cond_wait@@GLIBC_2.3.2+170(:0)@0x376f708a7a
>     at tsiWaitForSignalForever+58(:0)@0x2a95712169
>     at tsiWaitForSignal+39(:0)@0x2a957121c4
>     at tsiWaitForLockSignal+39(:0)@0x2a9571229e
>     at tsWaitForJavaLockSignal+31(:0)@0x2a9571234d
>     at RJNI_jrockit_vm_Threads_waitForUnblockSignal+14(:0)@0x2a95708e6c
>     at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
>     at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1630)[optimized]
>     at jrockit/vm/Locks.lockFat(Locks.java:1731)[optimized]
>     at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1277)[optimized]
>     at jrockit/vm/Locks.monitorEnter(Locks.java:2389)[inlined]
>     at jrockit/vm/Locks.monitorEnterForced(Locks.java:872)[optimized]
>     at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
>     at jrockit/vm/Locks.monitorEnterUnmatched(Ljava/lang/Object;)V(Native
> Method)
>     at
> EDU/oswego/cs/dl/util/concurrent/WriterPreferenceReadWriteLock$ReaderLock...a
> c
> quire()V(Unknown Source)[optimized]
>     at
> org/apache/jackrabbit/core/state/DefaultISMLocking$ReadLockImpl.<init>(Defau
> ltISMLocking.java:103)[inlined]
>     at
> org/apache/jackrabbit/core/state/DefaultISMLocking$ReadLockImpl.<init>(Defau
> ltISMLocking.java:97)[inlined]
>     at
> org/apache/jackrabbit/core/state/DefaultISMLocking.acquireReadLock(DefaultIS
> MLocking.java:65)[optimized]
>     at
> org/apache/jackrabbit/core/state/SharedItemStateManager.acquireReadLock(Shar
> edItemStateManager.java:1438)[inlined]
>     at
> org/apache/jackrabbit/core/state/SharedItemStateManager.hasItemState(SharedI
> temStateManager.java:269)[optimized]
>     at
> org/apache/jackrabbit/core/state/LocalItemStateManager.hasItemState(LocalIte
> mStateManager.java:178)[inlined]
>     at
> org/apache/jackrabbit/core/state/XAItemStateManager.hasItemState(XAItemState
> Manager.java:252)[optimized]
>     at
> org/apache/jackrabbit/core/state/SessionItemStateManager.getItemState(Sessio
> nItemStateManager.java:174)[optimized]
>     at
> org/apache/jackrabbit/core/ItemManager.createItemInstance(ItemManager.java:5
> 64)[inlined]
>     at
> org/apache/jackrabbit/core/ItemManager.getItem(ItemManager.java:395)[inlined
> ]
>     at
> org/apache/jackrabbit/core/LazyItemIterator.prefetchNext(LazyItemIterator...j
> a
> va:90)[inlined]
>     at
> org/apache/jackrabbit/core/LazyItemIterator.next(LazyItemIterator.java:203)[
> optimized]
>     ^-- Holding lock: org/apache/jackrabbit/core/ItemManager@0x8e7b568[thin
> lock]
> -----Original Message-----
> From: KÖLL Claus [mailto:C.KOELL@TIROL.GV.AT] 
> Sent: 25 June 2008 16:18
> To: users@jackrabbit.apache.org
> Subject: AW: Locking issues with XAItemStateManager - help appreciated
> 
> Hi Shaun,
> 
> In my case the problem was that the applicationserver (websphere)
> has changed the thread between acquiring writelock and the downgrade to a
> readlock
> so we are running into deadlock.
> 
> Have you tested the patched DefaultISMLocking Class ?
> 
> -----Ursprüngliche Nachricht-----
> Von: sbarriba [mailto:sbarriba@yahoo.co.uk] 
> Gesendet: Mittwoch, 25. Juni 2008 12:52
> An: users@jackrabbit.apache.org
> Betreff: RE: Locking issues with XAItemStateManager - help appreciated
> 
> 
> Apologies for the multitude of emails.
> 
> .......would switching from DefaultISMLocking to the FineGrainedISMLocking
> implementation help the situation?
> 
> I believe this can be done with the following config within the workspace
> config.
> 
> <ISMLocking class="[ClassName]"/>
> 
> Regards,
> Shaun
> 
> -----Original Message-----
> From: sbarriba [mailto:sbarriba@yahoo.co.uk] 
> Sent: 25 June 2008 11:35
> To: users@jackrabbit.apache.org
> Subject: RE: Locking issues with XAItemStateManager - help appreciated
> 
> Hi Claus,
> Thanks for the quick response.
> 
> We're not intentionally using an XA transaction. We're using Spring +
> Transaction Manager + Hibernate elsewhere to satisfy the request.
> 
> .......so a few quick questions:
> 
> * Is there a way to force JackRabbit not to use XA? What are the alternative
> ItemStateManager(s)?
> * Does using XA cause more locking?
> 
> Thanks in advance,
> Regards,
> Shaun
> 
> -----Original Message-----
> From: KÖLL Claus [mailto:C.KOELL@TIROL.GV.AT] 
> Sent: 25 June 2008 11:24
> To: users@jackrabbit.apache.org
> Subject: AW: Locking issues with XAItemStateManager - help appreciated
> 
> hi,
> 
> do you use a xa transaction ?
> take a look @ https://issues.apache.org/jira/browse/JCR-1334
> i see in your stack that the lock comes from acquireReadLock()
> this was also in my issue
> feel free do test the PatchedDefaultISMLocking.java by Marcel
> 
> BR,
> claus
> 
> -----Ursprüngliche Nachricht-----
> Von: sbarriba [mailto:sbarriba@yahoo.co.uk] 
> Gesendet: Mittwoch, 25. Juni 2008 12:16
> An: users@jackrabbit.apache.org
> Betreff: Locking issues with XAItemStateManager - help appreciated
> 
> Hi all,
> 
> As follow up to a previous thread we're seeing lots and lots of contention
> around the following lock. We're using Weblogic 9.1 / JRocket 27.4.0 (1.5).
> 
>  
> 
> Even with very little load on the app a thread dump shows active threads at
> exactly this point. As the concurrent load increases the contention
> increases until the app is continually thrashing on these locks and stops
> responding.
> 
>  
> 
> Is there a way to configure JackRabbit to reduce the amount of locking?
> 
> For example, I note the use of XAItemStateManager in the stack - is there an
> alternative ItemStateManager implementation which requires less locking?
> 
>  
> 
> All help appreciated.
> 
>  
> 
> Regards,
> 
> Shaun
> 
>  
> 
>  
> 
> "[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default
> (self-tuning)'" id=133 idx=0x20c tid=6426 prio=5 alive, in native, daemon
> 
>     at jrockit/vm/Locks.monitorEnterUnmatched(Ljava/lang/Object;)V(Native
> Method)
> 
>     at
> EDU/oswego/cs/dl/util/concurrent/WriterPreferenceReadWriteLock$ReaderLock....
> ..
> a
> c
> quire()V(Unknown Source)
> 
>     at
> org/apache/jackrabbit/core/state/DefaultISMLocking$ReadLockImpl.<init>(Defau
> ltISMLocking.java:103)
> 
>     at
> org/apache/jackrabbit/core/state/DefaultISMLocking$ReadLockImpl.<init>(Defau
> ltISMLocking.java:97)
> 
>     at
> org/apache/jackrabbit/core/state/DefaultISMLocking.acquireReadLock(DefaultIS
> MLocking.java:65)
> 
>     at
> org/apache/jackrabbit/core/state/SharedItemStateManager.acquireReadLock(Shar
> edItemStateManager.java:1438)[inlined]
> 
>     at
> org/apache/jackrabbit/core/state/SharedItemStateManager.getItemState(SharedI
> temStateManager.java:237)[optimized]
> 
>     at
> org/apache/jackrabbit/core/state/LocalItemStateManager.getPropertyState(Loca
> lItemStateManager.java:118)[inlined]
> 
>     at
> org/apache/jackrabbit/core/state/LocalItemStateManager.getItemState(LocalIte
> mStateManager.java:150)[inlined]
> 
>     at
> org/apache/jackrabbit/core/state/XAItemStateManager.getItemState(XAItemState
> Manager.java:226)[optimized]
> 
>     ^-- Holding lock:
> org/apache/jackrabbit/core/state/XAItemStateManager@0x3ea1ae88[thin lock]
> 
>     at
> org/apache/jackrabbit/core/state/SessionItemStateManager.getItemState(Sessio
> nItemStateManager.java:175)[optimized]
> 
>     at
> org/apache/jackrabbit/core/ItemManager.createItemInstance(ItemManager.java:5
> 64)[inlined]
> 
>     at
> org/apache/jackrabbit/core/ItemManager.getItem(ItemManager.java:395)[inlined
> ]
> 
>     at
> org/apache/jackrabbit/core/LazyItemIterator.prefetchNext(LazyItemIterator....
> ..
> j
> a
> va:90)[inlined]
> 
>     at
> org/apache/jackrabbit/core/LazyItemIterator.next(LazyItemIterator.java:203)[
> optimized]
> 
>     ^-- Holding lock: org/apache/jackrabbit/core/ItemManager@0x3ea1a4c0[thin
> lock]
> 
>  
> 
> 
> 
> 
> 
> 


Mime
View raw message