jackrabbit-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ian Boston (JIRA)" <j...@apache.org>
Subject [jira] Commented: (JCR-929) Under Heavy load in a Cluster HTTP Threads Block and stall requests
Date Fri, 25 May 2007 15:27:16 GMT

    [ https://issues.apache.org/jira/browse/JCR-929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499116
] 

Ian Boston commented on JCR-929:
--------------------------------


There is a stack trace of a node locked in waiting, the HTTP are locked and every request
to this node, that hits the http thread will block in the same wait pattern



     Starting Thread Monitor ==================
Thread Transient File Reaper waiting by java.lang.ref.ReferenceQueue$Lock@4622f ::WAITING
at org.apache.jackrabbit.util.TransientFileFactory$ReaperThread.run(TransientFileFactory.java:148)
Thread TP-Processor3 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@eb38d2
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread TP-Processor2 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@df787e
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread TP-Processor1 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@5fa15c
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor25 waiting by EDU.oswego.cs.dl.util.concurrent.ReentrantLock@5036f6
::WAITING at EDU.oswego.cs.dl.util.concurrent.ReentrantLock.acquire(null:-1)
     at java.lang.Object.wait(Object.java:-2)
     at java.lang.Object.wait(Object.java:474)
     at EDU.oswego.cs.dl.util.concurrent.ReentrantLock.acquire(null:-1)
     at org.apache.jackrabbit.core.lock.LockManagerImpl.acquire(LockManagerImpl.java:599)
     at org.apache.jackrabbit.core.lock.LockManagerImpl.externalLock(LockManagerImpl.java:973)
     at org.apache.jackrabbit.core.cluster.ClusterNode.process(ClusterNode.java:723)
     at org.apache.jackrabbit.core.cluster.ClusterNode.consume(ClusterNode.java:910)
     at org.apache.jackrabbit.core.journal.AbstractJournal.doSync(AbstractJournal.java:191)
     at org.apache.jackrabbit.core.journal.AbstractJournal.lockAndSync(AbstractJournal.java:241)
     at org.apache.jackrabbit.core.journal.DefaultRecordProducer.append(DefaultRecordProducer.java:51)
     at org.apache.jackrabbit.core.cluster.ClusterNode$WorkspaceUpdateChannel.updateCreated(ClusterNode.java:466)
     at org.apache.jackrabbit.core.state.SharedItemStateManager$Update.begin(SharedItemStateManager.java:530)
     at org.apache.jackrabbit.core.state.SharedItemStateManager.beginUpdate(SharedItemStateManager.java:825)
     at org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:855)
     at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:326)
     at org.apache.jackrabbit.core.state.XAItemStateManager.update(XAItemStateManager.java:313)
     at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:302)
     at org.apache.jackrabbit.core.state.SessionItemStateManager.update(SessionItemStateManager.java:306)
     at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:1214)
     at org.apache.jackrabbit.core.NodeImpl.lock(NodeImpl.java:4070)
     at org.apache.jackrabbit.webdav.simple.DavResourceImpl.lock(DavResourceImpl.java:685)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.doLock(AbstractWebdavServlet.java:689)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.execute(AbstractWebdavServlet.java:259)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.service(AbstractWebdavServlet.java:193)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
     at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
     at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
     at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
     at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
     at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
     at java.lang.Thread.run(Thread.java:613)
Thread http-8580-Processor24 waiting by EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock@488a98
::WAITING at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(null:-1)
     at java.lang.Object.wait(Object.java:-2)
     at java.lang.Object.wait(Object.java:474)
     at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(null:-1)
     at org.apache.jackrabbit.core.journal.AbstractJournal.lockAndSync(AbstractJournal.java:228)
     at org.apache.jackrabbit.core.journal.DefaultRecordProducer.append(DefaultRecordProducer.java:51)
     at org.apache.jackrabbit.core.cluster.ClusterNode$WorkspaceLockChannel.unlocked(ClusterNode.java:637)
     at org.apache.jackrabbit.core.lock.LockManagerImpl.internalUnlock(LockManagerImpl.java:338)
     at org.apache.jackrabbit.core.lock.LockManagerImpl.unlock(LockManagerImpl.java:428)
     at org.apache.jackrabbit.core.lock.XALockManager.unlock(XALockManager.java:103)
     at org.apache.jackrabbit.core.NodeImpl.unlock(NodeImpl.java:4133)
     at org.apache.jackrabbit.webdav.simple.DavResourceImpl.unlock(DavResourceImpl.java:739)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.doUnlock(AbstractWebdavServlet.java:710)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.execute(AbstractWebdavServlet.java:262)
     at org.apache.jackrabbit.server.AbstractWebdavServlet.service(AbstractWebdavServlet.java:193)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
     at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
     at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
     at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
     at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
     at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
     at java.lang.Thread.run(Thread.java:613)
Thread http-8580-Processor22 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@40eb2a
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor21 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@f45d1
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor20 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@b0cea2
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor19 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@ce1b2b
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor18 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@59c4c6
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor17 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@85ded7
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor16 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@e90ea7
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor15 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@c11ce0
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor14 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@4ecfa4
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor13 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@225841
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor12 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@d034cf
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor11 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@3b1da2
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor10 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@16b458
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor9 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@9a3769
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor8 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@abd36b
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor7 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@c54d06
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor6 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@43f502
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor5 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@8a7951
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor4 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@22da8f
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor3 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@3af3cb
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor2 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@5ba3ee
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread http-8580-Processor1 waiting by org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@67770f
::WAITING at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
Thread ClusterNode-node2 waiting by EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock@aaab5d
::WAITING at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(null:-1)
     at java.lang.Object.wait(Object.java:-2)
     at java.lang.Object.wait(Object.java:474)
     at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(null:-1)
     at org.apache.jackrabbit.core.journal.AbstractJournal.sync(AbstractJournal.java:160)
     at org.apache.jackrabbit.core.cluster.ClusterNode.sync(ClusterNode.java:283)
     at org.apache.jackrabbit.core.cluster.ClusterNode.run(ClusterNode.java:254)
     at java.lang.Thread.run(Thread.java:613)
Thread IndexMerger waiting by org.apache.commons.collections.buffer.BlockingBuffer@7dfc02
::WAITING at org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread IndexMerger waiting by org.apache.commons.collections.buffer.BlockingBuffer@9d85b ::WAITING
at org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread ObservationManager waiting by org.apache.commons.collections.buffer.BlockingBuffer@5b60bf
::WAITING at org.apache.commons.collections.buffer.BlockingBuffer.remove(BlockingBuffer.java:107)
Thread Finalizer waiting by java.lang.ref.ReferenceQueue$Lock@99d183 ::WAITING
Thread Reference Handler waiting by java.lang.ref.Reference$Lock@1de95a ::WAITING
     Done Thread Monitor ==================

> Under Heavy load in a Cluster HTTP Threads Block and stall requests
> -------------------------------------------------------------------
>
>                 Key: JCR-929
>                 URL: https://issues.apache.org/jira/browse/JCR-929
>             Project: Jackrabbit
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.3
>         Environment: 2 Node Cluster, OSX, JDK 1.5 with DatabaseJournal, DatabasePersistanceManager,
all content in DB, using WebDAV to load
>            Reporter: Ian Boston
>         Assigned To: Dominique Pfister
>
> Under Heavy load created by mounting both nodes in the cluster in OSX Finder and then
uploading large numebers of files to each node at the same time ( a few 1000), eventually
one of the nodes stops responding and the Finder mount timesout and disconnects.
> Once that happens that node becomes unusable.
> More mount attempts will prompt for a password indicating HTTP is still running, but
will timeout once the connection is authenticated.
> Access by the Web Browser will prompt for a password, conenct and provide a once only
listing of any collection in the workspace. If you try to refresh that collection, the HTTP
request hangs forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message