jackrabbit-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ian Boston (JIRA)" <j...@apache.org>
Subject [jira] Commented: (JCR-929) Under Heavy load in a Cluster HTTP Threads Block and stall requests
Date Fri, 25 May 2007 16:08:16 GMT

    [ https://issues.apache.org/jira/browse/JCR-929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499128
] 

Ian Boston commented on JCR-929:
--------------------------------

Doing that causes the ClusterNode thread to timeout but the 2 http threads go into waiting
state, so they must be interlocking on waits somewhere. 

I cant really do an attempt in the main threads since if the operation doesnt get to the journal
then it wont propagate, and doing a back off would have to back off far enough to undo the
interlock.


The pattern looks slightly different now,  to I might have fixed the first problem.... sorry
about all the long stack traces but  its going to be confusing if not in context.

     Starting Thread Monitor ==================
Thread Transient File Reaper waiting by java.lang.ref.ReferenceQueue$Lock@8da92 ::WAITING
at org.apache.jackrabbit.util.TransientFileFactory$ReaperThread.run(TransientFileFactory.java:148)
Thread Transient File Reaper waiting by java.lang.ref.ReferenceQueue$Lock@b410a ::WAITING
at org.apache.jackrabbit.util.TransientFileFactory$ReaperThread.run(TransientFileFactory.java:148)
Thread TP-Processor3 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@64886c
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread TP-Processor2 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@388e0b
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread TP-Processor1 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@f59ac1
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor25 waiting by EDU.oswego.cs.dl.util.concurrent.ReentrantLock@eb5563
::WAITING at EDU.oswego.cs.dl.util.concurrent.ReentrantLock.acquire(null:-1)
     at java.lang.Object.wait(Object.java:-2)
     at java.lang.Object.wait(Object.java:474)
     at EDU.oswego.cs.dl.util.concurrent.ReentrantLock.acquire(null:-1)
     at org.apache.jackrabbit.core.lock.LockManagerImpl.acquire(LockManagerImpl.java:599)
     at org.apache.jackrabbit.core.lock.LockManagerImpl.nodeAdded(LockManagerImpl.java:840)
     at org.apache.jackrabbit.core.lock.LockManagerImpl.onEvent(LockManagerImpl.java:745)
     at org.apache.jackrabbit.core.observation.EventConsumer.consumeEvents(EventConsumer.java:231)
     at org.apache.jackrabbit.core.observation.ObservationDispatcher.dispatchEvents(ObservationDispatcher.java:201)
     at org.apache.jackrabbit.core.observation.EventStateCollection.dispatch(EventStateCollection.java:424)
     at org.apache.jackrabbit.core.state.SharedItemStateManager$Update.end(SharedItemStateManager.java:721)
     at org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:855)
     at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:326)
     at org.apache.jackrabbit.core.state.XAItemStateManager.update(XAItemStateManager.java:313)
     at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:302)
     at org.apache.jackrabbit.core.state.SessionItemStateManager.update(SessionItemStateManager.java:306)
     at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:1214)
     at org.apache.jackrabbit.webdav.simple.DavResourceImpl.addMember(DavResourceImpl.java:517)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.doPut(AbstractWebdavServlet.java:504)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.execute(AbstractWebdavServlet.java:241)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.service(AbstractWebdavServlet.java:193)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
     at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
     at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
     at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
     at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
     at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
     at java.lang.Thread.run(Thread.java:613)
Thread http-8080-Processor23 waiting by EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock@bd12a5
::WAITING at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(null:-1)
     at java.lang.Object.wait(Object.java:-2)
     at java.lang.Object.wait(Object.java:474)
     at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(null:-1)
     at org.apache.jackrabbit.core.journal.AbstractJournal.lockAndSync(AbstractJournal.java:233)
     at org.apache.jackrabbit.core.journal.DefaultRecordProducer.append(DefaultRecordProducer.java:51)
     at org.apache.jackrabbit.core.cluster.ClusterNode$WorkspaceLockChannel.unlocked(ClusterNode.java:637)
     at org.apache.jackrabbit.core.lock.LockManagerImpl.internalUnlock(LockManagerImpl.java:338)
     at org.apache.jackrabbit.core.lock.LockManagerImpl.unlock(LockManagerImpl.java:428)
     at org.apache.jackrabbit.core.lock.XALockManager.unlock(XALockManager.java:103)
     at org.apache.jackrabbit.core.NodeImpl.unlock(NodeImpl.java:4133)
     at org.apache.jackrabbit.webdav.simple.DavResourceImpl.unlock(DavResourceImpl.java:739)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.doUnlock(AbstractWebdavServlet.java:710)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.execute(AbstractWebdavServlet.java:262)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.service(AbstractWebdavServlet.java:193)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
     at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
     at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
     at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
     at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
     at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
     at java.lang.Thread.run(Thread.java:613)
Thread http-8080-Processor22 waiting by EDU.oswego.cs.dl.util.concurrent.ReentrantLock@eb5563
::WAITING at EDU.oswego.cs.dl.util.concurrent.ReentrantLock.acquire(null:-1)
     at java.lang.Object.wait(Object.java:-2)
     at java.lang.Object.wait(Object.java:474)
     at EDU.oswego.cs.dl.util.concurrent.ReentrantLock.acquire(null:-1)
     at org.apache.jackrabbit.core.lock.LockManagerImpl.acquire(LockManagerImpl.java:599)
     at org.apache.jackrabbit.core.lock.LockManagerImpl.getLockInfo(LockManagerImpl.java:356)
     at org.apache.jackrabbit.core.lock.XALockManager.isLocked(XALockManager.java:143)
     at org.apache.jackrabbit.core.NodeImpl.isLocked(NodeImpl.java:4161)
     at org.apache.jackrabbit.webdav.simple.DavResourceImpl.getLock(DavResourceImpl.java:648)
     at org.apache.jackrabbit.webdav.simple.DavResourceImpl.initProperties(DavResourceImpl.java:312)
     at org.apache.jackrabbit.webdav.simple.DavResourceImpl.getProperties(DavResourceImpl.java:271)
     at org.apache.jackrabbit.webdav.MultiStatusResponse.<init>(MultiStatusResponse.java:180)
     at org.apache.jackrabbit.webdav.MultiStatus.addResourceProperties(MultiStatus.java:62)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.doPropFind(AbstractWebdavServlet.java:435)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.execute(AbstractWebdavServlet.java:232)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.service(AbstractWebdavServlet.java:193)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
     at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
     at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
     at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
     at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
     at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
     at java.lang.Thread.run(Thread.java:613)
Thread http-8080-Processor21 waiting by EDU.oswego.cs.dl.util.concurrent.ReentrantLock@eb5563
::WAITING at EDU.oswego.cs.dl.util.concurrent.ReentrantLock.acquire(null:-1)
     at java.lang.Object.wait(Object.java:-2)
     at java.lang.Object.wait(Object.java:474)
     at EDU.oswego.cs.dl.util.concurrent.ReentrantLock.acquire(null:-1)
     at org.apache.jackrabbit.core.lock.LockManagerImpl.acquire(LockManagerImpl.java:599)
     at org.apache.jackrabbit.core.lock.LockManagerImpl.getLockInfo(LockManagerImpl.java:356)
     at org.apache.jackrabbit.core.lock.XALockManager.isLocked(XALockManager.java:143)
     at org.apache.jackrabbit.core.NodeImpl.isLocked(NodeImpl.java:4161)
     at org.apache.jackrabbit.webdav.simple.DavResourceImpl.getLock(DavResourceImpl.java:648)
     at org.apache.jackrabbit.webdav.simple.DavResourceImpl.isLocked(DavResourceImpl.java:855)
     at org.apache.jackrabbit.webdav.simple.DavResourceImpl.addMember(DavResourceImpl.java:501)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.doPut(AbstractWebdavServlet.java:504)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.execute(AbstractWebdavServlet.java:241)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.service(AbstractWebdavServlet.java:193)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
     at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
     at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
     at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
     at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
     at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
     at java.lang.Thread.run(Thread.java:613)
Thread http-8080-Processor20 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@42de17
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor19 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@2ea67
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor18 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@6b1a74
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor17 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@95dc5a
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor16 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@b5cc4d
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor15 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@77e923
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor14 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@6afba3
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor13 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@ae8c15
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor12 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@6eea56
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor11 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@daf576
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor10 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@293c5e
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor9 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@3a774f
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor8 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@6efb3f
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor7 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@30e551
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor6 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@1f6576
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor5 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@3e2817
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor4 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@ba1328
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor3 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@3fbf92
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor2 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@3a3c1f
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8080-Processor1 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@4127c0
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread IndexMerger waiting by org.apache.commons.collections.buffer.BlockingBuffer@44b625
::WAITING at org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread IndexMerger waiting by org.apache.commons.collections.buffer.BlockingBuffer@a7252d
::WAITING at org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread ObservationManager waiting by org.apache.commons.collections.buffer.BlockingBuffer@79d52f
::WAITING at org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread IndexMerger waiting by org.apache.commons.collections.buffer.BlockingBuffer@8088da
::WAITING at org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread IndexMerger waiting by org.apache.commons.collections.buffer.BlockingBuffer@7000ea
::WAITING at org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread ObservationManager waiting by org.apache.commons.collections.buffer.BlockingBuffer@4c15d9
::WAITING at org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread Finalizer waiting by java.lang.ref.ReferenceQueue$Lock@c31c7d ::WAITING
Thread Reference Handler waiting by java.lang.ref.Reference$Lock@9c7650 ::WAITING
     Done Thread Monitor ==================





> Under Heavy load in a Cluster HTTP Threads Block and stall requests
> -------------------------------------------------------------------
>
>                 Key: JCR-929
>                 URL: https://issues.apache.org/jira/browse/JCR-929
>             Project: Jackrabbit
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.3
>         Environment: 2 Node Cluster, OSX, JDK 1.5 with DatabaseJournal, DatabasePersistanceManager,
all content in DB, using WebDAV to load
>            Reporter: Ian Boston
>         Assigned To: Dominique Pfister
>
> Under Heavy load created by mounting both nodes in the cluster in OSX Finder and then
uploading large numebers of files to each node at the same time ( a few 1000), eventually
one of the nodes stops responding and the Finder mount timesout and disconnects.
> Once that happens that node becomes unusable.
> More mount attempts will prompt for a password indicating HTTP is still running, but
will timeout once the connection is authenticated.
> Access by the Web Browser will prompt for a password, conenct and provide a once only
listing of any collection in the workspace. If you try to refresh that collection, the HTTP
request hangs forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message