manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: max_pred_locks_per_transaction
Date Mon, 04 Feb 2013 13:40:27 GMT
There's code now committed to trunk which might help diagnose this
problem.  It's part of CONNECTORS-638 and is meant to find database
connection handle leaks.

Karl

On Fri, Feb 1, 2013 at 8:27 AM, Karl Wright <daddywri@gmail.com> wrote:
> Hi Erlend,
>
> The WAITING in the thread dump has nothing to do with PostgreSQL
> locks.  In fact, these thread dumps show absolutely no database
> activity EXCEPT for one thread which is trying to close idle
> connections.
>
> If you are using the multiprocess model, you should get a thread dump
> of the agents process instead; that would be far more meaningful.  It
> would help us figure out why this:
>
>>>>>>>
> This is what the Postgres admins told me today. The idle processes
> (idle in transaction) are:
>
> - - SELECT id,status,checktime FROM jobqueue WHERE dochash=$1 AND
> jobid=$2 FOR UPDATE
> - - UPDATE hopcount SET deathmark=$1 WHERE jobid=$2 AND deathmark=$3
> <<<<<<
>
> ... is occurring.  I can think of no reason why the above should
> deadlock, which would be one possibility.  The other possibility is
> that Postgresql operations are simply timing out on the server side
> and the JDBC client is unaware of that.
>
> In any case, the agents thread dump will help enormously.
>
> Karl
>
>
> On Fri, Feb 1, 2013 at 7:01 AM, Erlend GarĂ¥sen <e.f.garasen@usit.uio.no> wrote:
>>
>> Sorry, wrong thread dump. This is the correct one. The previous one was
>> generated after I stopped the services. I guess entries like this one is
>> related to the locks?
>> "resin-118" id=118 WAITING
>>
>> ----8<----
>>
>> [2013-01-31 19:00:00.014] {resin-18} Thread Dump generated Thu Jan 31
>> 18:59:59 CET 2013
>>                                      "Signal Dispatcher" id=4 RUNNABLE
>>
>>                                      "resin-18" id=18 RUNNABLE
>>
>>                                        at
>> sun.management.ThreadImpl.getThreadInfo1 (ThreadImpl.java) (native)
>>                                        at
>> sun.management.ThreadImpl.getThreadInfo (ThreadImpl.java:156)
>>                                        at
>> com.caucho.util.ThreadDump.getThreadDump (ThreadDump.java:121)
>>                                        at
>> com.caucho.admin.action.ThreadDumpAction.execute (ThreadDumpAction.java:52)
>>                                        at
>> com.caucho.health.action.DumpThreads.doActionImpl (DumpThreads.java:99)
>>                                        at
>> com.caucho.health.action.AbstractHealthAction.doAction
>> (AbstractHealthAction.java:90)
>>                                        at
>> com.caucho.env.health.HealthService.executeActions (HealthService.java:722)
>>                                        at
>> com.caucho.env.health.HealthService$ExecuteActionsTask.run
>> (HealthService.java:1030)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>>                                      "resin-148" id=148 RUNNABLE
>>                                        at
>> java.lang.Throwable.fillInStackTrace (Throwable.java) (native)
>>                                        at java.lang.Throwable.<init>
>> (Throwable.java:196)
>>                                        at java.lang.Exception.<init>
>> (Exception.java:41)
>>                                        at
>> javax.management.JMException.<init> (JMException.java:35)
>>                                        at
>> javax.management.OperationsException.<init> (OperationsException.java:36)
>>                                        at
>> javax.management.InstanceNotFoundException.<init>
>> (InstanceNotFoundException.java:34)
>>                                        at
>> com.caucho.jmx.AbstractMBeanServer.getAttribute
>> (AbstractMBeanServer.java:594)
>>                                        at
>> com.caucho.env.meter.JmxExpr$JmxDelta.sample (JmxExpr.java:211)
>>                                        at
>> com.caucho.env.meter.JmxExpr$JmxContainerExpr.sample (JmxExpr.java:94)
>>                                        at
>> com.caucho.env.meter.JmxExpr$JmxContainerExpr.sample (JmxExpr.java:94)
>>                                        at
>> com.caucho.env.meter.JmxCalculationMeterImpl.sample
>> (JmxCalculationMeterImpl.java:62)
>>                                        at
>> com.caucho.server.admin.StatSystem$Sample.sample (StatSystem.java:244)
>>                                        at
>> com.caucho.server.admin.ProStatSystem.sampleData (ProStatSystem.java:582)
>>                                        at
>> com.caucho.server.admin.ProStatSystem.sample (ProStatSystem.java:558)
>>
>>                                        at
>> com.caucho.server.admin.ProStatSystem.handleAlarm (ProStatSystem.java:646)
>>                                        at com.caucho.util.Alarm.handleAlarm
>> (Alarm.java:523)
>>                                        at com.caucho.util.Alarm.run
>> (Alarm.java:495)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>>                                      "resin-125" id=125 RUNNABLE
>>                                        at
>> com.caucho.vfs.JniMemoryMappedFile.nativeRead (JniMemoryMappedFile.java)
>> (native)
>>                                        at
>> com.caucho.vfs.JniMemoryMappedFile.read (JniMemoryMappedFile.java:136)
>>                                        at
>> com.caucho.db.block.BlockReadWrite.readBlockImpl (BlockReadWrite.java:210)
>>                                        at
>> com.caucho.db.block.BlockReadWrite.readBlock (BlockReadWrite.java:177)
>>                                        at com.caucho.db.block.Block.read
>> (Block.java:259)
>>                                        at
>> com.caucho.db.block.BlockStore.readBlock (BlockStore.java:569)
>>                                        at
>> com.caucho.db.xa.DbTransaction.readBlock (DbTransaction.java:332)
>>                                        at
>> com.caucho.db.table.TableIterator.nextBlock (TableIterator.java:276)
>>                                        at
>> com.caucho.db.sql.RowIterateExpr.nextBlock (RowIterateExpr.java:87)
>>                                        at com.caucho.db.sql.Query.nextBlock
>> (Query.java:788)
>>                                        at com.caucho.db.sql.Query.start
>> (Query.java:726)
>>                                        at
>> com.caucho.db.sql.DeleteQuery.execute (DeleteQuery.java:74)
>>                                        at
>> com.caucho.db.jdbc.PreparedStatementImpl.execute
>> (PreparedStatementImpl.java:365)
>>                                        at
>> com.caucho.db.jdbc.PreparedStatementImpl.executeUpdate
>> (PreparedStatementImpl.java:335)
>>                                        at
>> com.caucho.server.admin.StatDatabase$ReaperTask.handleAlarm
>> (StatDatabase.java:941)
>>
>>                                        at com.caucho.util.Alarm.handleAlarm
>> (Alarm.java:523)
>>                                        at com.caucho.util.Alarm.run
>> (Alarm.java:495)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>>                                      "resin-port-127.0.0.1:6945-16" id=16
>> RUNNABLE (in native)
>>
>>                                      "resin-port-127.0.0.1:6945-17" id=17
>> RUNNABLE (in native)
>>
>>                                      "resin-port-127.0.0.1:6945-19" id=19
>> RUNNABLE (in native)
>>
>>
>>                                      "resin-port-127.0.0.1:6945-31" id=31
>> RUNNABLE (in native)
>>
>>                                      "resin-port-127.0.0.1:6945-37" id=37
>> RUNNABLE (in native)
>>                                        at
>> com.caucho.vfs.JniSocketImpl.nativeAccept (JniSocketImpl.java) (native)
>>                                        at
>> com.caucho.vfs.JniSocketImpl.accept (JniSocketImpl.java:141)
>>                                        at
>> com.caucho.vfs.JniServerSocketImpl.accept (JniServerSocketImpl.java:254)
>>                                        at
>> com.caucho.network.listen.TcpSocketLinkListener.accept
>> (TcpSocketLinkListener.java:1372)
>>                                        at
>> com.caucho.network.listen.TcpSocketLink.accept (TcpSocketLink.java:978)
>>                                        at
>> com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl
>> (TcpSocketLink.java:928)
>>                                        at
>> com.caucho.network.listen.ConnectionTask.runThread (ConnectionTask.java:117)
>>                                        at
>> com.caucho.network.listen.ConnectionTask.run (ConnectionTask.java:93)
>>                                        at
>> com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks
>> (SocketLinkThreadLauncher.java:169)
>>                                        at
>> com.caucho.network.listen.TcpSocketAcceptThread.run
>> (TcpSocketAcceptThread.java:61)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>>                                      "resin-port-127.0.0.1:6945-13" id=13
>> RUNNABLE (in native)
>>
>>                                      "resin-port-127.0.0.1:6945-14" id=14
>> RUNNABLE (in native)
>>
>>                                      "resin-port-127.0.0.1:6945-15" id=15
>> RUNNABLE (in native)
>>
>>                                      "resin-port-127.0.0.1:6945-22" id=22
>> RUNNABLE (in native)
>>
>>                                      "resin-port-127.0.0.1:6945-24" id=24
>> RUNNABLE (in native)
>>
>>                                      "resin-port-127.0.0.1:6945-26" id=26
>> RUNNABLE (in native)
>>
>>                                      "resin-port-127.0.0.1:6945-30" id=30
>> RUNNABLE (in native)
>>
>>                                      "resin-port-127.0.0.1:6945-41" id=41
>> RUNNABLE (in native)
>>
>>                                      "resin-port-127.0.0.1:6945-88" id=88
>> RUNNABLE (in native)
>>
>>                                      "resin-port-127.0.0.1:6945-90" id=90
>> RUNNABLE (in native)
>>
>>                                      "resin-port-127.0.0.1:6945-98" id=98
>> RUNNABLE (in native)
>>                                        at
>> com.caucho.vfs.JniSocketImpl.nativeAccept (JniSocketImpl.java) (native)
>>                                        at
>> com.caucho.vfs.JniSocketImpl.accept (JniSocketImpl.java:141)
>>                                        at
>> com.caucho.vfs.JniServerSocketImpl.accept (JniServerSocketImpl.java:254)
>>                                        at
>> com.caucho.network.listen.TcpSocketLinkListener.accept
>> (TcpSocketLinkListener.java:1372)
>>                                        at
>> com.caucho.network.listen.TcpSocketLink.accept (TcpSocketLink.java:978)
>>                                        at
>> com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl
>> (TcpSocketLink.java:928)
>>                                        at
>> com.caucho.network.listen.TcpSocketLink.handleAcceptTask
>> (TcpSocketLink.java:902)
>>                                        at
>> com.caucho.network.listen.ConnectionTask.runThread (ConnectionTask.java:114)
>>                                        at
>> com.caucho.network.listen.ConnectionTask.run (ConnectionTask.java:93)
>>                                        at
>> com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks
>> (SocketLinkThreadLauncher.java:169)
>>                                        at
>> com.caucho.network.listen.TcpSocketResumeThread.run
>> (TcpSocketResumeThread.java:62)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>>                                      "resin-main-link" id=33 RUNNABLE (in
>> native)
>>                                        at
>> java.net.SocketInputStream.socketRead0 (SocketInputStream.java) (native)
>>                                        at java.net.SocketInputStream.read
>> (SocketInputStream.java:129)
>>                                        at java.net.SocketInputStream.read
>> (SocketInputStream.java:182)
>>                                        at
>> com.caucho.remote.websocket.UnmaskedFrameInputStream.readFrameHeaderImpl
>> (UnmaskedFrameInputStream.java:142)
>>                                        at
>> com.caucho.remote.websocket.FrameInputStream.readFrameHeader
>> (FrameInputStream.java:97)
>>                                        at
>> com.caucho.remote.websocket.WebSocketInputStream.startBinaryMessage
>> (WebSocketInputStream.java:78)
>>                                        at
>> com.caucho.hmtp.HmtpWebSocketReader.readPacket (HmtpWebSocketReader.java:70)
>>                                        at com.caucho.hmtp.HmtpLinkWorker.run
>> (HmtpLinkWorker.java:114)
>>                                        at
>> com.caucho.server.resin.ResinLink.run (ResinLink.java:71)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>>                                      "resin-select-manager-0" id=36 RUNNABLE
>> (in native)
>>                                        at
>> com.caucho.network.listen.JniSelectManager.selectNative
>> (JniSelectManager.java) (native)
>>                                        at
>> com.caucho.network.listen.JniSelectManager.runImpl
>> (JniSelectManager.java:392)
>>                                        at
>> com.caucho.network.listen.JniSelectManager.run (JniSelectManager.java:359)
>>                                        at java.lang.Thread.run
>> (Thread.java:662)
>>
>>                                      "resin-fail-safe-halt" id=34 WAITING
>>                                        at sun.misc.Unsafe.park (Unsafe.java)
>> (native)
>>                                        at
>> java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
>>                                        at
>> com.caucho.env.shutdown.ShutdownSystem$FailSafeHaltThread.run
>> (ShutdownSystem.java:499)
>>
>>                                      "resin-shutdown" id=35 WAITING
>> [2013-01-31 19:00:00.014] {resin-18}   at sun.misc.Unsafe.park (Unsafe.java)
>> (native)
>>
>>                                        at
>> java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
>>                                        at
>> com.caucho.env.shutdown.ShutdownSystem$ShutdownThread.run
>> (ShutdownSystem.java:446)
>>
>>                                      "resin-118" id=118 WAITING
>>
>>                                      "resin-134" id=134 WAITING
>>
>>                                      "resin-141" id=141 WAITING
>>
>>                                      "resin-144" id=144 WAITING
>>
>>                                      "resin-154" id=154 WAITING
>>
>>                                      "resin-156" id=156 WAITING
>>
>>                                      "resin-163" id=163 WAITING
>>
>>                                      "resin-167" id=167 WAITING
>>
>>                                      "resin-169" id=169 WAITING
>>
>>                                      "resin-170" id=170 WAITING
>>
>>                                      "resin-171" id=171 WAITING
>>
>>                                      "resin-174" id=174 WAITING
>>
>>                                      "resin-175" id=175 WAITING
>>
>>                                      "resin-176" id=176 WAITING
>>
>>                                      "resin-177" id=177 WAITING
>>
>>                                      "resin-178" id=178 WAITING
>>
>>                                      "resin-179" id=179 WAITING
>>
>>                                      "resin-47" id=47 WAITING
>>
>>                                        at sun.misc.Unsafe.park (Unsafe.java)
>> (native)
>>                                        at
>> java.util.concurrent.locks.LockSupport.park (LockSupport.java:283)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.park (ResinThread2.java:196)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:147)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>>                                      "Finalizer" id=3 WAITING
>>                                          waiting on
>> java.lang.ref.ReferenceQueue$Lock@4ecd51ad
>>                                        at java.lang.Object.wait
>> (Object.java) (native)
>>                                        at
>> java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:118)
>>                                        at
>> java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:134)
>>                                        at
>> java.lang.ref.Finalizer$FinalizerThread.run (Finalizer.java:159)
>>
>>                                      "Reference Handler" id=2 WAITING
>>                                          waiting on
>> java.lang.ref.Reference$Lock@9ad5d46
>>                                        at java.lang.Object.wait
>> (Object.java) (native)
>>                                        at java.lang.Object.wait
>> (Object.java:485)
>>                                        at
>> java.lang.ref.Reference$ReferenceHandler.run (Reference.java:116)
>>
>>
>> "BlockWriter[Table[log_data:7]]-13" id=38 TIMED_WAITING
>>
>>
>> "BlockWriter[Table[stat_data_mcf_1:5]]-11" id=136 TIMED_WAITING
>>
>>                                      "CoordinatorThread[]-3" id=12
>> TIMED_WAITING
>>
>>                                      "FlushWorker[]-5" id=168 TIMED_WAITING
>>
>>                                      "HealthSystemHealthCheckWorker" id=28
>> TIMED_WAITING
>>
>>
>> "resin-port-127.0.0.1:6945-launcher" id=172 TIMED_WAITING
>>
>>                                        at sun.misc.Unsafe.park (Unsafe.java)
>> (native)
>>                                        at
>> java.util.concurrent.locks.LockSupport.parkUntil (LockSupport.java:351)
>>                                        at
>> com.caucho.env.thread2.AbstractTaskWorker2.run
>> (AbstractTaskWorker2.java:216)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.runTasks (ResinThread2.java:173)
>>                                        at
>> com.caucho.env.thread2.ResinThread2.run (ResinThread2.java:118)
>>
>>                                      "main" id=1 TIMED_WAITING
>>                                          waiting on
>> com.caucho.server.resin.ResinWaitForExitService@7d2f117
>>                                        at java.lang.Object.wait
>> (Object.java) (native)
>>                                        at
>> com.caucho.server.resin.ResinWaitForExitService.waitForExit
>> (ResinWaitForExitService.java:135)
>>                                        at
>> com.caucho.server.resin.Resin.waitForExit (Resin.java:1190)
>>                                        at com.caucho.server.resin.Resin.main
>> (Resin.java:1239)
>>
>>                                      "resin-timer" id=9 TIMED_WAITING
>>                                        at java.lang.Thread.sleep
>> (Thread.java) (native)
>>                                        at
>> com.caucho.util.CurrentTime$CurrentTimeThread.run (CurrentTime.java:228)
>>
>>
>> "ThreadLauncher2[ThreadPool[system]]-1" id=10 TIMED_WAITING
>>
>>                                      "ThreadScheduleWorker[]-2" id=29
>> TIMED_WAITING
>>                                        at sun.misc.Unsafe.park (Unsafe.java)
>> (native)
>>                                        at
>> java.util.concurrent.locks.LockSupport.parkUntil (LockSupport.java:351)
>>                                        at
>> com.caucho.env.thread2.AbstractTaskWorker2.run
>> (AbstractTaskWorker2.java:216)
>>                                        at java.lang.Thread.run
>> (Thread.java:662)
>>
>>                                      "Connection pool reaper" id=49
>> TIMED_WAITING
>>                                        at java.lang.Thread.sleep
>> (Thread.java) (native)
>>                                        at
>> org.apache.manifoldcf.core.jdbcpool.ConnectionPoolManager$ConnectionCloserThread.run
>> (ConnectionPoolManager.java:130)
>>
>> [2013-01-31 19:03:00.003] {resin-154} AnomalyAnalyzer JVM|Thread|JVM
>> Runnable Count WARNING
>>                                        3.000 sample is 147.49% of 2.034 avg,
>> 5.327 std deviations (std=0.181, n=1029.0)
>> [2013-01-31 19:09:00.002] {resin-168} AnomalyAnalyzer JVM|Thread|JVM
>> Runnable Count WARNING
>>                                        3.000 sample is 147.44% of 2.035 avg,
>> 5.269 std deviations (std=0.183, n=1065.0)
>> [2013-01-31 19:14:00.002] {resin-118} AnomalyAnalyzer JVM|Thread|JVM
>> Runnable Count WARNING
>>                                        3.000 sample is 147.43% of 2.035 avg,
>> 5.257 std deviations (std=0.184, n=1089.0)
>> [2013-02-01 11:05:00.003] {resin-449} AnomalyAnalyzer OS|Process|File
>> Descriptor Count WARNING
>>                                        229.000 sample is 101.32% of 226.010
>> avg, 8.674 std deviations (std=0.345, n=8554.0)
>> [2013-02-01 11:06:00.005] {resin-449} AnomalyAnalyzer JVM|Thread|JVM Waiting
>> Count WARNING
>>                                        37.000 sample is 114.35% of 32.357
>> avg, 5.171 std deviations (std=0.898, n=8559.0)
>> [2013-02-01 11:06:00.005] {resin-449} AnomalyAnalyzer Resin|Thread|Thread
>> Active Count WARNING
>>                                        18.000 sample is 81.99% of 21.953
>> avg, 6.191 std deviations (std=0.639, n=8559.0)
>> [2013-02-01 11:06:00.005] {resin-421} Health action DumpThreads[] executing
>> [2013-02-01 11:06:00.008] {resin-425} Health action DumpThreads[] executing

Mime
View raw message