Mailing-List: contact dev-help@jackrabbit.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@jackrabbit.apache.org
Message-ID: <158456303.201461268304868426.JavaMail.jira@brutus.apache.org>
Date: Thu, 11 Mar 2010 10:54:28 +0000 (UTC)
From: "Robert Sauer (JIRA)" <jira@apache.org>
To: dev@jackrabbit.apache.org
Subject: [jira] Commented: (JCR-2554) Deadlock inside XASession on Weblogic
In-Reply-To: <5342166.176091268224047193.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/JCR-2554?page=3Dcom.atlassian.j=
ira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D128440=
10#action_12844010 ]=20

Robert Sauer commented on JCR-2554:
-----------------------------------

Essentially yes. I had breakpoint on all critical places, and this one was =
trigegred.

I assume the call sequences is as follows (having the JCA adapter driven by=
 a WLS workmanager pool):
1. thread-a invokes prepare on behalf of XID_1
  - the write lock is granted to thread-a
  - XID_1 is stored as the ActiveXID
2. thread-a now invokes prepare on behalf of XID_2
  - tries to acquire the write lock, theoretically it should block until XI=
D_1 was committed
  - but due to the previous association between the write lock and thread-a=
 locking succeeds
  - thread-a now tries to set the ActiveXID to XID_2, but as XID_1 is still=
 active, it will just trace a warning
3. thread-b invokes commit on behalf of XID_1
  - the ActiveXID is cleared
  - the write lock gets released
4. eventually some thread invokes commit on behalf of XID_2, but as the Act=
iveXID was not set, signalling readers fails

I did not know how to easily fix 2. as the lock-to-thread association seems=
 to be fundamentally wrong in this case. So I started with the different ap=
proach contained in the patch.

> Deadlock inside XASession on Weblogic
> -------------------------------------
>
>                 Key: JCR-2554
>                 URL: https://issues.apache.org/jira/browse/JCR-2554
>             Project: Jackrabbit Content Repository
>          Issue Type: Bug
>          Components: jackrabbit-core
>    Affects Versions: 1.6.1, 2.0.0
>         Environment: Weblogic 9.2
>            Reporter: Robert Sauer
>            Assignee: Claus K=C3=B6ll
>            Priority: Critical
>         Attachments: ConcurrentReaders.java, jr-core-xid-aware-ism-lockin=
g1.patch
>
>
> In one of our client deployments on WebLogic 9.2 we observed JackRabbit s=
essions going stale in a load test. This was observed against release 1.6.1=
 (to which we migrated due to concurrency related issues JCR-2081 and JCR-2=
237). Same effect with 2.0.0.
> =20
> I could finally reproduce this issue locally. And it seems to boil down t=
o WLS invoking the sequence of <prepare> ... <release> ... <commit> on one =
XA session from multiple threads, as it seems breaking assumptions of the t=
hread-bound java.util.concurrent-RWLock based DefaultISMLocking class.
> Effectively the setActiveXid(..) method on DefaultISMLocking$RWLock fails=
 as the old active XID was not yet cleared. With the result of more and mor=
e sessions deadlocking in below's invocation stack.
> {code}
> "[ACTIVE] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-t=
uning)'" daemon prio=3D1 tid=3D0x33fc3ec0 nid=3D0x2324 in Object.wait() [0x=
2156a000..0x2156beb0] at java.lang.Object.wait(Native Method) - waiting on =
<0x68a54698> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteL=
ock$WriterLock) at java.lang.Object.wait(Object.java:474) at EDU.oswego.cs.=
dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(Unknown=
 Source) - locked <0x68a54698> (a EDU.oswego.cs.dl.util.concurrent.WriterPr=
eferenceReadWriteLock$WriterLock) at org.apache.jackrabbit.core.state.Defau=
ltISMLocking$1.<init>(DefaultISMLocking.java:64) at org.apache.jackrabbit.c=
ore.state.DefaultISMLocking.acquireWriteLock(DefaultISMLocking.java:61) at =
org.apache.jackrabbit.core.version.AbstractVersionManager.acquireWriteLock(=
AbstractVersionManager.java:146) at org.apache.jackrabbit.core.version.XAVe=
rsionManager$1.prepare(XAVersionManager.java:562) at org.apache.jackrabbit.=
core.TransactionContext.prepare(TransactionContext.java:154) - locked <0x6d=
c2ad88> (a org.apache.jackrabbit.core.TransactionContext) at org.apache.jac=
krabbit.core.XASessionImpl.prepare(XASessionImpl.java:331) at org.apache.ja=
ckrabbit.jca.TransactionBoundXAResource.prepare(TransactionBoundXAResource.=
java:68) at weblogic.connector.security.layer.AdapterLayer.prepare(AdapterL=
ayer.java:397) at weblogic.connector.transaction.outbound.XAWrapper.prepare=
(XAWrapper.java:297) at weblogic.transaction.internal.XAServerResourceInfo.=
prepare(XAServerResourceInfo.java:1276) at weblogic.transaction.internal.XA=
ServerResourceInfo.prepare(XAServerResourceInfo.java:499) at weblogic.trans=
action.internal.ServerSCInfo$1.execute(ServerSCInfo.java:335) at weblogic.k=
ernel.Kernel.executeIfIdle(Kernel.java:243) at weblogic.transaction.interna=
l.ServerSCInfo.startPrepare(ServerSCInfo.java:326) at weblogic.transaction.=
internal.ServerTransactionImpl.localPrepare(ServerTransactionImpl.java:2516=
) at weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(Serv=
erTransactionImpl.java:2211) at weblogic.transaction.internal.ServerTransac=
tionImpl.internalCommit(ServerTransactionImpl.java:266) at weblogic.transac=
tion.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:227) =
at weblogic.transaction.internal.TransactionManagerImpl.commit(TransactionM=
anagerImpl.java:283) at org.springframework.transaction.jta.JtaTransactionM=
anager.doCommit(JtaTransactionManager.java:1028) at org.springframework.tra=
nsaction.support.AbstractPlatformTransactionManager.processCommit(AbstractP=
latformTransactionManager.java:709) at org.springframework.transaction.supp=
ort.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionMa=
nager.java:678)
> {code}

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.