Mailing-List: contact users-help@jackrabbit.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@jackrabbit.apache.org
Received-SPF: pass (nike.apache.org: domain of aas.ojj@gmail.com designates
 209.85.221.194 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=SE10fcsnjIFJ93g7NkTupGmLIjbD7LYaul+LtK7zTywhaWczkDm+I3FXx50TR8TSNW
         oYLmm1rFDXry3uPxLY3uuV4TiBYTLZc8QMMWLjuk5/ySW576GFfCS46YduoqNKczGyPJ
         VE0cxjwrZp49lLz6x37F650+D6jwUlaSL7HTo=
MIME-Version: 1.0
Date: Tue, 28 Jul 2009 15:00:15 +0530
Message-ID: <f65422c50907280230j303f4852j37bf00ff43f8ce8c@mail.gmail.com>
Subject: Deadlock in cluster deployment
From: aasoj j <aas.ojj@gmail.com>
To: users@jackrabbit.apache.org
Content-Type: multipart/alternative; boundary=0016364ee316d7b9f8046fc0b7be

--0016364ee316d7b9f8046fc0b7be
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi,

We have a 3 host Jackrabbit cluster deployment (version 1.5.4). In our case
new data nodes are added to the repository frequently (500 new nodes in
different parts of the tree every 15 minutes), and a huge number of reads
are serviced (50 per second).

After working for a few hours we observe that most of the threads end up in
a wait state:
"http-0.0.0.0-4080-37" Id=3098 in WAITING on
lock=EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock@42ae9b4d
   at java.lang.Object.wait(Native Method)
   at java.lang.Object.wait(Object.java:485)
   at
EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(Unknown
Source)
   at
org.apache.jackrabbit.core.state.DefaultISMLocking$ReadLockImpl.<init>(DefaultISMLocking.java:84)
   at
org.apache.jackrabbit.core.state.DefaultISMLocking$ReadLockImpl.<init>(DefaultISMLocking.java:78)
   at
org.apache.jackrabbit.core.state.DefaultISMLocking.acquireReadLock(DefaultISMLocking.java:44)
   at
org.apache.jackrabbit.core.state.SharedItemStateManager.acquireReadLock(SharedItemStateManager.java:1409)
   at
org.apache.jackrabbit.core.state.SharedItemStateManager.hasItemState(SharedItemStateManager.java:286)
   at
org.apache.jackrabbit.core.state.XAItemStateManager.hasItemState(XAItemStateManager.java:295)
   at
org.apache.jackrabbit.core.state.SessionItemStateManager.getItemState(SessionItemStateManager.java:181)
   at
org.apache.jackrabbit.core.HierarchyManagerImpl.getItemState(HierarchyManagerImpl.java:150)
   at
org.apache.jackrabbit.core.HierarchyManagerImpl.resolvePath(HierarchyManagerImpl.java:113)
   at
org.apache.jackrabbit.core.HierarchyManagerImpl.resolvePath(HierarchyManagerImpl.java:325)
   at
org.apache.jackrabbit.core.CachingHierarchyManager.resolvePath(CachingHierarchyManager.java:135)
   at
org.apache.jackrabbit.core.HierarchyManagerImpl.resolvePath(HierarchyManagerImpl.java:363)
   at org.apache.jackrabbit.core.ItemManager.getItem(ItemManager.java:435)
   at org.apache.jackrabbit.core.SessionImpl.getItem(SessionImpl.java:817)

This is usually observed after new nodes are added to the repository in one
of the host.

We also observed the following NPE once.
ERROR org.apache.jackrabbit.core.cluster.ClusterNode:292 Unexpected error
while syncing of journal: null
java.lang.NullPointerException
        at
org.apache.jackrabbit.core.state.XAItemStateManager.stateModified(XAItemStateManager.java:580)
        at
org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateModified(StateChangeDispatcher.java:111)
        at
org.apache.jackrabbit.core.state.SharedItemStateManager.stateModified(SharedItemStateManager.java:400)
        at
org.apache.jackrabbit.core.state.ItemState.notifyStateUpdated(ItemState.java:244)
        at
org.apache.jackrabbit.core.state.ChangeLog.persisted(ChangeLog.java:285)
        at
org.apache.jackrabbit.core.state.SharedItemStateManager.doExternalUpdate(SharedItemStateManager.java:1186)
        at
org.apache.jackrabbit.core.state.SharedItemStateManager.externalUpdate(SharedItemStateManager.java:1109)
        at
org.apache.jackrabbit.core.RepositoryImpl$WorkspaceInfo.externalUpdate(RepositoryImpl.java:2098)
        at
org.apache.jackrabbit.core.cluster.ClusterNode.process(ClusterNode.java:852)
        at
org.apache.jackrabbit.core.cluster.ChangeLogRecord.process(ChangeLogRecord.java:384)
        at
org.apache.jackrabbit.core.cluster.ClusterNode.consume(ClusterNode.java:798)
        at
org.apache.jackrabbit.core.journal.AbstractJournal.doSync(AbstractJournal.java:213)
        at
org.apache.jackrabbit.core.journal.AbstractJournal.sync(AbstractJournal.java:188)
        at
org.apache.jackrabbit.core.cluster.ClusterNode.sync(ClusterNode.java:315)
        at
org.apache.jackrabbit.core.cluster.ClusterNode.run(ClusterNode.java:286)
        at java.lang.Thread.run(Thread.java:619)


It seems that these issues are similar to issues below.

   1.
   http://www.nabble.com/-jira--Created%3A-%28JCR-2168%29-Avoid-premature-publication-of-XAItemStateManager-td24145090.html#a24200238
   .
   2. https://issues.apache.org/jira/browse/JCR-2171


Please advise.

Thanks in advance

--0016364ee316d7b9f8046fc0b7be--