manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erlend GarĂ¥sen <e.f.gara...@usit.uio.no>
Subject Re: max_pred_locks_per_transaction
Date Fri, 01 Feb 2013 11:45:15 GMT
On 25.01.13 12.24, Karl Wright wrote:

> Leaving logging at the default values would have shown the ERROR
> message you have below.  So the cause for the pause must have been
> something else.

Agree.

> When ManifoldCF seems to make no progress, the first thing to do is
> look at the simple history and see if it is retrying on something for
> some reason.  If that is not helpful, get a thread dump.  You can use
> jstack for that purpose.

Resin has some health check tools I can try to configure to give me such 
information. I have included a thread dump in the end of this email.

The problem also occurs by running RC8/1.1. It even occurs on our 
production system running MCF 1.0.1. I created a test job which crawls 
the same documents, but instead of posting to Solr, it just dumps to 
NullOutput, which means it is not related to the Solr output connector.

We have never discovered such problems before, and the only difference 
is that I have added a couple of new hosts. These hosts do have a lot of 
excludes. We have began to use the "exclude from index" functionality 
which do have some complex regexps, so maybe this might be the source of 
the problem (a possible bug in MCF):
---8<--
# Exclude some journal resources from the index:
https://www\.journals\.uio\.no/index\.php/.*/(search|issue|information|announcement|current|user|login|about|notification).*
https://www\.journals\.uio\.no/index\.php/index/(search|user|login).*
https://www\.journals\.uio\.no/index\.php/.*/user$

# Exclude some DOU resources from the index:
# - where the url contains characters after the digits (.+):
https://www\.duo\.uio\.no/handle/\d{9}/\d+/.+
# - if they are navigation pages:
https://www\.duo\.uio\.no/(community-list|browse|password-login|inn|discover).*
---8<--

This is what the Postgres admins told me today. The idle processes (idle 
in transaction) are:

- - SELECT id,status,checktime FROM jobqueue WHERE dochash=$1 AND
jobid=$2 FOR UPDATE
- - UPDATE hopcount SET deathmark=$1 WHERE jobid=$2 AND deathmark=$3

I have started the crawl again with the following params set to DEBUG. 
In addition I have notified the server admins and asked them to adjust 
Resin's health check to give us even more detailed output. I will get 
back to this issue as soon as I have found out more.
   <property name="org.apache.manifoldcf.lock" value="DEBUG"/>
   <property name="org.apache.manifoldcf.agents" value="DEBUG"/>
   <property name="org.apache.manifoldcf.crawlerthreads" value="DEBUG"/>
   <property name="org.apache.manifoldcf.jobs" value="DEBUG"/>
   <property name="org.apache.manifoldcf.connectors" value="DEBUG"/>

Here's what Resin have detected regarding these locks:

[2013-02-01 11:06:00.019] {resin-421} Thread Dump generated Fri Feb 01 
11:05:59 CET 2013
                                       "Signal Dispatcher" id=4 RUNNABLE

                                       "resin-421" id=421 RUNNABLE
                                         at 
sun.management.ThreadImpl.getThreadInfo1 (ThreadImpl.java) (native)
                                         at 
sun.management.ThreadImpl.getThreadInfo (ThreadImpl.java:156)
                                         at 
com.caucho.util.ThreadDump.getThreadDump (ThreadDump.java:121)
                                         at 
com.caucho.admin.action.ThreadDumpAction.execute (ThreadDumpAction.java:52)
                                         at 
com.caucho.health.action.DumpThreads.doActionImpl (DumpThreads.java:99)
                                         at 
com.caucho.health.action.AbstractHealthAction.doAction 
(AbstractHealthAction.java:90)
                                         at 
com.caucho.env.health.HealthService.executeActions (HealthService.java:722)
                                         at 
com.caucho.env.health.HealthService$ExecuteActionsTask.run 
(HealthService.java:1030)
                                         at 
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
                                         at 
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)

                                       "resin-449" id=449 RUNNABLE
                                         at 
com.caucho.env.health.HealthService.getCurrent (HealthService.java:100)
                                         at 
com.caucho.env.health.ProHealthSystemFacade.fireEventImpl 
(ProHealthSystemFacade.java:61)
                                         at 
com.caucho.env.health.HealthSystemFacade.fireEvent 
(HealthSystemFacade.java:65)
                                         at 
com.caucho.health.analysis.AnomalyAnalyzer.analyze 
(AnomalyAnalyzer.java:178)
                                         at 
com.caucho.server.admin.ProStatSystem.analyze (ProStatSystem.java:628)
                                         at 
com.caucho.server.admin.ProStatSystem.sample (ProStatSystem.java:560)
                                         at 
com.caucho.server.admin.ProStatSystem.handleAlarm (ProStatSystem.java:646)
                                         at 
com.caucho.util.Alarm.handleAlarm (Alarm.java:523)
                                         at com.caucho.util.Alarm.run 
(Alarm.java:495)
                                         at 
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
                                         at 
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)

                                       "resin-port-127.0.0.1:6945-31" 
id=31 RUNNABLE (in native)

                                       "resin-port-127.0.0.1:6945-37" 
id=37 RUNNABLE (in native)
                                         at 
com.caucho.vfs.JniSocketImpl.readNative (JniSocketImpl.java) (native)
                                         at 
com.caucho.vfs.JniSocketImpl.read (JniSocketImpl.java:438)
                                         at 
com.caucho.vfs.JniStream.readTimeout (JniStream.java:102)
                                         at 
com.caucho.vfs.ReadStream.fillWithTimeout (ReadStream.java:1145)
                                         at 
com.caucho.network.listen.TcpSocketLinkListener.keepaliveThreadRead 
(TcpSocketLinkListener.java:1546)
                                         at 
com.caucho.network.listen.TcpSocketLink.processKeepalive 
(TcpSocketLink.java:1348)
                                         at 
com.caucho.network.listen.TcpSocketLink.handleRequestsImpl 
(TcpSocketLink.java:1233)
                                         at 
com.caucho.network.listen.TcpSocketLink.handleRequests 
(TcpSocketLink.java:1149)
                                         at 
com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl 
(TcpSocketLink.java:950)
                                         at 
com.caucho.network.listen.ConnectionTask.runThread (ConnectionTask.java:117)
                                         at 
com.caucho.network.listen.ConnectionTask.run (ConnectionTask.java:93)
                                         at 
com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks 
(SocketLinkThreadLauncher.java:169)
                                         at 
com.caucho.network.listen.TcpSocketAcceptThread.run 
(TcpSocketAcceptThread.java:61)
                                         at 
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
                                         at 
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)

                                       "resin-port-127.0.0.1:6945-14" 
id=14 RUNNABLE (in native)

                                       "resin-port-127.0.0.1:6945-15" 
id=15 RUNNABLE (in native)

                                       "resin-port-127.0.0.1:6945-22" 
id=22 RUNNABLE (in native)

                                       "resin-port-127.0.0.1:6945-24" 
id=24 RUNNABLE (in native)

                                       "resin-port-127.0.0.1:6945-26" 
id=26 RUNNABLE (in native)

                                       "resin-port-127.0.0.1:6945-30" 
id=30 RUNNABLE (in native)

                                       "resin-port-127.0.0.1:6945-41" 
id=41 RUNNABLE (in native)

                                       "resin-port-127.0.0.1:6945-88" 
id=88 RUNNABLE (in native)

                                       "resin-port-127.0.0.1:6945-90" 
id=90 RUNNABLE (in native)

                                       "resin-port-127.0.0.1:6945-98" 
id=98 RUNNABLE (in native)
                                         at 
com.caucho.vfs.JniSocketImpl.nativeAccept (JniSocketImpl.java) (native)
                                         at 
com.caucho.vfs.JniSocketImpl.accept (JniSocketImpl.java:141)
                                         at 
com.caucho.vfs.JniServerSocketImpl.accept (JniServerSocketImpl.java:254)
                                         at 
com.caucho.network.listen.TcpSocketLinkListener.accept 
(TcpSocketLinkListener.java:1372)
                                         at 
com.caucho.network.listen.TcpSocketLink.accept (TcpSocketLink.java:978)
                                         at 
com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl 
(TcpSocketLink.java:928)
                                         at 
com.caucho.network.listen.TcpSocketLink.handleAcceptTask 
(TcpSocketLink.java:902)
                                         at 
com.caucho.network.listen.ConnectionTask.runThread (ConnectionTask.java:114)
                                         at 
com.caucho.network.listen.ConnectionTask.run (ConnectionTask.java:93)
                                         at 
com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks 
(SocketLinkThreadLauncher.java:169)
                                         at 
com.caucho.network.listen.TcpSocketResumeThread.run 
(TcpSocketResumeThread.java:62)
                                         at 
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
                                         at 
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)

                                       "resin-port-127.0.0.1:6945-13" 
id=13 RUNNABLE (in native)
                                         at 
com.caucho.vfs.JniSocketImpl.readNative (JniSocketImpl.java) (native)
                                         at 
com.caucho.vfs.JniSocketImpl.read (JniSocketImpl.java:438)
                                         at 
com.caucho.vfs.JniStream.readTimeout (JniStream.java:102)
                                         at 
com.caucho.vfs.ReadStream.fillWithTimeout (ReadStream.java:1145)
                                         at 
com.caucho.network.listen.TcpSocketLinkListener.keepaliveThreadRead 
(TcpSocketLinkListener.java:1546)
                                         at 
com.caucho.network.listen.TcpSocketLink.processKeepalive 
(TcpSocketLink.java:1348)
                                         at 
com.caucho.network.listen.TcpSocketLink.handleRequestsImpl 
(TcpSocketLink.java:1233)
                                         at 
com.caucho.network.listen.TcpSocketLink.handleRequests 
(TcpSocketLink.java:1149)
                                         at 
com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl 
(TcpSocketLink.java:950)
                                         at 
com.caucho.network.listen.TcpSocketLink.handleAcceptTask 
(TcpSocketLink.java:902)
                                         at 
com.caucho.network.listen.ConnectionTask.runThread (ConnectionTask.java:114)
                                         at 
com.caucho.network.listen.ConnectionTask.run (ConnectionTask.java:93)
                                         at 
com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks 
(SocketLinkThreadLauncher.java:169)
                                         at 
com.caucho.network.listen.TcpSocketResumeThread.run 
(TcpSocketResumeThread.java:62)
                                         at 
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
                                         at 
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)

                                       "resin-main-link" id=33 RUNNABLE 
(in native)
                                         at 
java.net.SocketInputStream.socketRead0 (SocketInputStream.java) (native)
                                         at 
java.net.SocketInputStream.read (SocketInputStream.java:129)
                                         at 
java.net.SocketInputStream.read (SocketInputStream.java:182)
                                         at 
com.caucho.remote.websocket.UnmaskedFrameInputStream.readFrameHeaderImpl 
(UnmaskedFrameInputStream.java:142)
                                         at 
com.caucho.remote.websocket.FrameInputStream.readFrameHeader 
(FrameInputStream.java:97)
                                         at 
com.caucho.remote.websocket.WebSocketInputStream.startBinaryMessage 
(WebSocketInputStream.java:78)
                                         at 
com.caucho.hmtp.HmtpWebSocketReader.readPacket (HmtpWebSocketReader.java:70)
                                         at 
com.caucho.hmtp.HmtpLinkWorker.run (HmtpLinkWorker.java:114)
                                         at 
com.caucho.server.resin.ResinLink.run (ResinLink.java:71)
                                         at 
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
                                         at 
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)

                                       "resin-select-manager-0" id=36 
RUNNABLE (in native)
                                         at 
com.caucho.network.listen.JniSelectManager.selectNative 
(JniSelectManager.java) (native)
                                         at 
com.caucho.network.listen.JniSelectManager.runImpl 
(JniSelectManager.java:392)
                                         at 
com.caucho.network.listen.JniSelectManager.run (JniSelectManager.java:359)
                                         at java.lang.Thread.run 
(Thread.java:662)

                                       "resin-fail-safe-halt" id=34 WAITING
                                         at sun.misc.Unsafe.park 
(Unsafe.java) (native)
                                         at 
java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
                                         at 
com.caucho.env.shutdown.ShutdownSystem$FailSafeHaltThread.run 
(ShutdownSystem.java:499)

                                       "resin-shutdown" id=35 WAITING
                                         at sun.misc.Unsafe.park 
(Unsafe.java) (native)
                                         at 
java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
                                         at 
com.caucho.env.shutdown.ShutdownSystem$ShutdownThread.run 
(ShutdownSystem.java:446)

                                       "resin-16" id=16 WAITING

                                       "resin-17" id=17 WAITING

                                       "resin-19" id=19 WAITING

                                       "resin-397" id=397 WAITING

                                       "resin-403" id=403 WAITING

                                       "resin-412" id=412 WAITING

                                       "resin-425" id=425 WAITING

                                       "resin-426" id=426 WAITING

                                       "resin-428" id=428 WAITING

                                       "resin-432" id=432 WAITING

                                       "resin-433" id=433 WAITING

                                       "resin-434" id=434 WAITING
[2013-02-01 11:06:00.019] {resin-421}
                                       "resin-435" id=435 WAITING

                                       "resin-436" id=436 WAITING

                                       "resin-438" id=438 WAITING

                                       "resin-439" id=439 WAITING

                                       "resin-440" id=440 WAITING

                                       "resin-441" id=441 WAITING

                                       "resin-442" id=442 WAITING

                                       "resin-444" id=444 WAITING

                                       "resin-451" id=451 WAITING
                                         at sun.misc.Unsafe.park 
(Unsafe.java) (native)
                                         at 
java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
                                         at 
com.caucho.env.thread2.ResinThread2.park (ResinThread2.java:196)
                                         at 
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:147)
                                         at 
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)

                                       "Finalizer" id=3 WAITING
                                           waiting on 
java.lang.ref.ReferenceQueue$Lock@4ecd51ad
                                         at java.lang.Object.wait 
(Object.java) (native)
                                         at 
java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:118)
                                         at 
java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:134)
                                         at 
java.lang.ref.Finalizer$FinalizerThread.run (Finalizer.java:159)

                                       "Reference Handler" id=2 WAITING
                                           waiting on 
java.lang.ref.Reference$Lock@9ad5d46
                                         at java.lang.Object.wait 
(Object.java) (native)
                                         at java.lang.Object.wait 
(Object.java:485)
                                         at 
java.lang.ref.Reference$ReferenceHandler.run (Reference.java:116)

 
"BlockWriter[Table[log_data:7]]-13" id=450 TIMED_WAITING

 
"BlockWriter[Table[stat_data_mcf_1:5]]-11" id=437 TIMED_WAITING

                                       "CoordinatorThread[]-3" id=12 
TIMED_WAITING

                                       "FlushWorker[]-5" id=448 
TIMED_WAITING

                                       "HealthSystemHealthCheckWorker" 
id=28 TIMED_WAITING

 
"resin-port-127.0.0.1:6945-launcher" id=447 TIMED_WAITING
                                         at sun.misc.Unsafe.park 
(Unsafe.java) (native)
                                         at 
java.util.concurrent.locks.LockSupport.parkUntil (LockSupport.java:351)
                                         at 
com.caucho.env.thread2.AbstractTaskWorker2.run 
(AbstractTaskWorker2.java:216)
                                         at 
com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
                                         at 
com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)

                                       "main" id=1 TIMED_WAITING
                                           waiting on 
com.caucho.server.resin.ResinWaitForExitService@7d2f117
                                         at java.lang.Object.wait 
(Object.java) (native)
                                         at 
com.caucho.server.resin.ResinWaitForExitService.waitForExit 
(ResinWaitForExitService.java:135)
                                         at 
com.caucho.server.resin.Resin.waitForExit (Resin.java:1190)
                                         at 
com.caucho.server.resin.Resin.main (Resin.java:1239)

                                       "resin-timer" id=9 TIMED_WAITING
                                         at java.lang.Thread.sleep 
(Thread.java) (native)
                                         at 
com.caucho.util.CurrentTime$CurrentTimeThread.run (CurrentTime.java:228)

 
"ThreadLauncher2[ThreadPool[system]]-1" id=10 TIMED_WAITING

                                       "ThreadScheduleWorker[]-2" id=29 
TIMED_WAITING
                                         at sun.misc.Unsafe.park 
(Unsafe.java) (native)
                                         at 
java.util.concurrent.locks.LockSupport.parkUntil (LockSupport.java:351)
                                         at 
com.caucho.env.thread2.AbstractTaskWorker2.run 
(AbstractTaskWorker2.java:216)
                                         at java.lang.Thread.run 
(Thread.java:662)

                                       "Connection pool reaper" id=49 
TIMED_WAITING
                                         at java.lang.Thread.sleep 
(Thread.java) (native)
                                         at 
org.apache.manifoldcf.core.jdbcpool.ConnectionPoolManager$ConnectionCloserThread.run 
(ConnectionPoolManager.java:130)


Erlend

-- 
Erlend GarĂ¥sen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Mime
View raw message