manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Wright (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CONNECTORS-1395) Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
Date Wed, 08 Mar 2017 13:04:37 GMT

    [ https://issues.apache.org/jira/browse/CONNECTORS-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901174#comment-15901174 ] 

Karl Wright edited comment on CONNECTORS-1395 at 3/8/17 1:04 PM:
-----------------------------------------------------------------

Thread dump is consistent with a lock that was taken but hasn't been freed, and was dropped on the floor.  Probably this happened when the "FATAL" log entry was created; during the Idle Cleanup thread a lock is getting freed that was never grabbed.

{code}
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.31-b07 mixed mode):

"Set priority thread-EventThread" #30657286 daemon prio=10 os_prio=2 tid=0x000000006e79b800 nid=0x149c waiting on condition [0x0000000054cee000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f710cc40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Set priority thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30657285 daemon prio=10 os_prio=2 tid=0x000000006e79d000 nid=0xd4c runnable [0x000000005830f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f710fdc0> (a sun.nio.ch.Util$2)
        - locked <0x00000000f710fdb0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f710fc88> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Agents thread-EventThread" #30657284 daemon prio=5 os_prio=0 tid=0x0000000056dbb800 nid=0xab4 waiting on condition [0x0000000057dff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f7112c40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Agents thread-SendThread(127.0.0.1:8349)" #30657283 daemon prio=5 os_prio=0 tid=0x000000006b2e3800 nid=0x11e8 runnable [0x000000005f96f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f710df30> (a sun.nio.ch.Util$2)
        - locked <0x00000000f710df20> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f710dea0> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Idle cleanup thread-EventThread" #30657282 daemon prio=5 os_prio=0 tid=0x000000006c1b8000 nid=0x928 waiting on condition [0x000000005f24f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f7113470> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Idle cleanup thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30657281 daemon prio=5 os_prio=0 tid=0x000000006c1b7000 nid=0x1040 runnable [0x0000000055f8e000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f71144d0> (a sun.nio.ch.Util$2)
        - locked <0x00000000f71144c0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f7113f90> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Finisher thread-EventThread" #30650263 daemon prio=5 os_prio=0 tid=0x000000005768f800 nid=0x163c waiting on condition [0x000000005ef2f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f70e6bb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Finisher thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30650262 daemon prio=5 os_prio=0 tid=0x000000005768e800 nid=0x1158 runnable [0x000000005f02e000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f70e9d30> (a sun.nio.ch.Util$2)
        - locked <0x00000000f70e9d20> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f70e9bf8> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Job delete thread-EventThread" #30650034 daemon prio=5 os_prio=0 tid=0x000000006b2e4800 nid=0x10e8 waiting on condition [0x000000005f82f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f7098c48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Job delete thread-SendThread(127.0.0.1:8349)" #30650033 daemon prio=5 os_prio=0 tid=0x000000006c1b8800 nid=0x8fc runnable [0x00000000566af000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f709ba78> (a sun.nio.ch.Util$2)
        - locked <0x00000000f709ba68> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f709b9e8> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Job reset thread-EventThread" #30650028 daemon prio=5 os_prio=0 tid=0x000000006e79c800 nid=0x17a8 waiting on condition [0x000000005c89e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f709dcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Job reset thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30650027 daemon prio=5 os_prio=0 tid=0x000000006e79b000 nid=0xcec runnable [0x000000005c4df000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f709e9d8> (a sun.nio.ch.Util$2)
        - locked <0x00000000f709e9c8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f709e530> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Job reset thread-EventThread" #30649986 daemon prio=5 os_prio=0 tid=0x0000000056f19800 nid=0x1490 waiting on condition [0x000000005f42f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f70a6aa8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Job reset thread-SendThread(127.0.0.1:8349)" #30649985 daemon prio=5 os_prio=0 tid=0x0000000056f12800 nid=0xacc runnable [0x000000005619e000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f70aab20> (a sun.nio.ch.Util$2)
        - locked <0x00000000f70aab10> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f70aaa90> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Agents thread-EventThread" #30640967 daemon prio=5 os_prio=0 tid=0x000000006c1b6800 nid=0x1120 waiting on condition [0x000000005fafe000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f6fe3c30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Agents thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30640966 daemon prio=5 os_prio=0 tid=0x000000006c1b5800 nid=0x12e0 runnable [0x000000005d28f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f6fe6db0> (a sun.nio.ch.Util$2)
        - locked <0x00000000f6fe6da0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f6fe6c78> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Set priority thread-EventThread" #30595106 daemon prio=10 os_prio=2 tid=0x0000000056c28800 nid=0x15f8 waiting on condition [0x000000005fc0f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f63607e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Set priority thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30595105 daemon prio=10 os_prio=2 tid=0x0000000056c28000 nid=0x12f0 runnable [0x000000005fd8f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f6362598> (a sun.nio.ch.Util$2)
        - locked <0x00000000f6362588> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f6362318> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Idle cleanup thread-EventThread" #30594300 daemon prio=5 os_prio=0 tid=0x000000006b2e1800 nid=0x10c8 waiting on condition [0x000000005fedf000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f6367fa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Idle cleanup thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30594299 daemon prio=5 os_prio=0 tid=0x000000006b2e2000 nid=0x1540 runnable [0x000000005f68f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f6368a58> (a sun.nio.ch.Util$2)
        - locked <0x00000000f6368a48> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f6368740> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Expire stuffer thread-EventThread" #30594028 daemon prio=6 os_prio=0 tid=0x00000000577bb000 nid=0x157c waiting on condition [0x000000005eddf000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f6369b98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Expire stuffer thread-SendThread(127.0.0.1:8349)" #30594027 daemon prio=6 os_prio=0 tid=0x00000000550c9000 nid=0x118c runnable [0x000000005804f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f63786f0> (a sun.nio.ch.Util$2)
        - locked <0x00000000f63786e0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f6378398> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Document cleanup stuffer thread-EventThread" #30594022 daemon prio=5 os_prio=0 tid=0x0000000056f13000 nid=0x1634 waiting on condition [0x000000005ec0e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f637e0a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Document cleanup stuffer thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30594021 daemon prio=5 os_prio=0 tid=0x0000000056f18800 nid=0x1488 runnable [0x000000005646e000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f637b120> (a sun.nio.ch.Util$2)
        - locked <0x00000000f637b110> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f637b090> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Connection pool reaper" #2098506 daemon prio=5 os_prio=0 tid=0x000000006e79a000 nid=0xf58 waiting on condition [0x000000006cf3f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.manifoldcf.core.jdbcpool.ConnectionPoolManager$ConnectionCloserThread.run(ConnectionPoolManager.java:152)

"Assessment thread" #138 daemon prio=5 os_prio=0 tid=0x0000000056fa6800 nid=0xd90 in Object.wait() [0x000000005e2ef000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff522940> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.AssessmentThread.run(AssessmentThread.java:68)

"Idle cleanup thread" #137 daemon prio=5 os_prio=0 tid=0x0000000056fa5800 nid=0xc94 in Object.wait() [0x000000005e91f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff575968> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.IdleCleanupThread.run(IdleCleanupThread.java:74)

"Seeding thread" #136 daemon prio=5 os_prio=0 tid=0x0000000056fa5000 nid=0xd94 in Object.wait() [0x000000005e6ee000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000feea03e0> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.SeedingThread.run(SeedingThread.java:104)

"Job reset thread" #135 daemon prio=5 os_prio=0 tid=0x0000000056fa4000 nid=0x810 in Object.wait() [0x000000005dedf000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff3cd530> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.JobResetThread.run(JobResetThread.java:126)

"Document delete thread '9'" #123 daemon prio=5 os_prio=0 tid=0x0000000056fa3800 nid=0x91c in Object.wait() [0x000000005e5df000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '8'" #122 daemon prio=5 os_prio=0 tid=0x0000000056fa2800 nid=0xd74 in Object.wait() [0x000000005e42f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '7'" #121 daemon prio=5 os_prio=0 tid=0x0000000056fa2000 nid=0xd80 in Object.wait() [0x000000005856f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '6'" #120 daemon prio=5 os_prio=0 tid=0x0000000056fa1000 nid=0xe4c in Object.wait() [0x000000005e1cf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '5'" #119 daemon prio=5 os_prio=0 tid=0x0000000056fa0800 nid=0x988 in Object.wait() [0x000000005dfde000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '4'" #118 daemon prio=5 os_prio=0 tid=0x0000000056f92000 nid=0x4e0 in Object.wait() [0x000000005ddbe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '3'" #117 daemon prio=5 os_prio=0 tid=0x0000000056f91800 nid=0xf9c in Object.wait() [0x000000005d08f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '2'" #116 daemon prio=5 os_prio=0 tid=0x0000000056f90800 nid=0xf88 in Object.wait() [0x000000005dc6f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '1'" #115 daemon prio=5 os_prio=0 tid=0x0000000056f90000 nid=0x2dc in Object.wait() [0x000000005da7f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '0'" #114 daemon prio=5 os_prio=0 tid=0x0000000056f8f000 nid=0xc74 in Object.wait() [0x000000005d94e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete stuffer thread" #113 daemon prio=5 os_prio=0 tid=0x0000000056f8e800 nid=0xc48 in Object.wait() [0x000000005ae8f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.core.lockmanager.LockGate.waitForPermission(LockGate.java:136)
        - locked <0x00000000f627b588> (a org.apache.manifoldcf.core.lockmanager.LockGate)
        at org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:258)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1522)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:280)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
        at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
        at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:862)
        at org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:236)
        at org.apache.manifoldcf.crawler.jobs.Jobs.deletingJobsPresent(Jobs.java:3133)
        at org.apache.manifoldcf.crawler.jobs.JobManager.getNextDeletableDocuments(JobManager.java:1862)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteStufferThread.run(DocumentDeleteStufferThread.java:108)

"Document cleanup thread '9'" #134 daemon prio=5 os_prio=0 tid=0x0000000056f8d800 nid=0x8dc in Object.wait() [0x000000005d7ef000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '8'" #133 daemon prio=5 os_prio=0 tid=0x0000000056f8c800 nid=0xe04 in Object.wait() [0x000000005d6cf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '7'" #132 daemon prio=5 os_prio=0 tid=0x0000000056f8c000 nid=0x4b4 in Object.wait() [0x000000005d4fe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '6'" #131 daemon prio=5 os_prio=0 tid=0x0000000056f8b000 nid=0xd38 in Object.wait() [0x000000005d39e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '5'" #130 daemon prio=5 os_prio=0 tid=0x0000000056f3d800 nid=0xec8 in Object.wait() [0x000000005d18f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '4'" #129 daemon prio=5 os_prio=0 tid=0x0000000056f3d000 nid=0xee8 in Object.wait() [0x000000005cf1f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '3'" #128 daemon prio=5 os_prio=0 tid=0x0000000056f3c000 nid=0x9d8 in Object.wait() [0x000000005cd8e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '2'" #127 daemon prio=5 os_prio=0 tid=0x0000000056f3b800 nid=0x2b8 in Object.wait() [0x000000005b7ce000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '1'" #126 daemon prio=5 os_prio=0 tid=0x0000000056f3a800 nid=0xf84 in Object.wait() [0x000000005cbef000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '0'" #125 daemon prio=5 os_prio=0 tid=0x0000000056f3a000 nid=0xf4c in Object.wait() [0x000000005c78f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup stuffer thread" #124 daemon prio=5 os_prio=0 tid=0x0000000056f39000 nid=0xf98 in Object.wait() [0x000000005ca6e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.core.lockmanager.LockGate.waitForPermission(LockGate.java:136)
        - locked <0x00000000f627b588> (a org.apache.manifoldcf.core.lockmanager.LockGate)
        at org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:258)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1522)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:280)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
        at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
        at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:862)
        at org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:236)
        at org.apache.manifoldcf.crawler.jobs.Jobs.cleaningJobsPresent(Jobs.java:3150)
        at org.apache.manifoldcf.crawler.jobs.JobManager.getNextCleanableDocuments(JobManager.java:1618)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupStufferThread.run(DocumentCleanupStufferThread.java:108)

"Expiration thread '9'" #112 daemon prio=5 os_prio=0 tid=0x0000000056f37000 nid=0xc0c in Object.wait() [0x000000005b9de000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '8'" #111 daemon prio=5 os_prio=0 tid=0x0000000056c72000 nid=0x134 in Object.wait() [0x000000005c64e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '7'" #110 daemon prio=5 os_prio=0 tid=0x0000000056c71800 nid=0xff8 in Object.wait() [0x000000005c06f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '6'" #109 daemon prio=5 os_prio=0 tid=0x0000000056c70800 nid=0x9f0 in Object.wait() [0x000000005c3ce000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '5'" #108 daemon prio=5 os_prio=0 tid=0x0000000056c70000 nid=0x498 in Object.wait() [0x000000005c2cf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '4'" #107 daemon prio=5 os_prio=0 tid=0x0000000056c6f000 nid=0x568 in Object.wait() [0x000000005c1bf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '3'" #106 daemon prio=5 os_prio=0 tid=0x0000000056c6e800 nid=0xd84 in Object.wait() [0x000000005bf4f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '2'" #105 daemon prio=5 os_prio=0 tid=0x0000000056c6d800 nid=0x244 in Object.wait() [0x000000005be0f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '1'" #104 daemon prio=5 os_prio=0 tid=0x0000000056c6d000 nid=0xe38 in Object.wait() [0x000000005bc3f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '0'" #103 daemon prio=5 os_prio=0 tid=0x0000000056c6c000 nid=0xff4 in Object.wait() [0x000000005bade000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Worker thread '29'" #102 daemon prio=5 os_prio=0 tid=0x0000000056c6b800 nid=0xf48 in Object.wait() [0x000000005b8df000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '28'" #101 daemon prio=5 os_prio=0 tid=0x0000000056bf0000 nid=0x9b4 in Object.wait() [0x000000005b6cf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '27'" #100 daemon prio=5 os_prio=0 tid=0x0000000056bef000 nid=0x5fc in Object.wait() [0x0000000059d8f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '26'" #99 daemon prio=5 os_prio=0 tid=0x0000000056bee800 nid=0x2a4 in Object.wait() [0x000000005b4fe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '25'" #98 daemon prio=5 os_prio=0 tid=0x0000000056bed800 nid=0xa20 in Object.wait() [0x000000005b3be000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '24'" #97 daemon prio=5 os_prio=0 tid=0x0000000056bed000 nid=0xa90 in Object.wait() [0x000000005b28e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '23'" #96 daemon prio=5 os_prio=0 tid=0x0000000056bec000 nid=0xe90 in Object.wait() [0x000000005b12f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '22'" #95 daemon prio=5 os_prio=0 tid=0x0000000056beb800 nid=0x998 in Object.wait() [0x000000005af8f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '21'" #94 daemon prio=5 os_prio=0 tid=0x0000000056bea800 nid=0x80c in Object.wait() [0x000000005ad8e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '20'" #93 daemon prio=5 os_prio=0 tid=0x0000000056bea000 nid=0x2e0 in Object.wait() [0x000000005ac0e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '19'" #92 daemon prio=5 os_prio=0 tid=0x0000000056be9000 nid=0x2ec in Object.wait() [0x000000005a1bf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '18'" #91 daemon prio=5 os_prio=0 tid=0x0000000056bde000 nid=0xd54 in Object.wait() [0x000000005ab0f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '17'" #90 daemon prio=5 os_prio=0 tid=0x0000000056bdd000 nid=0x5a4 in Object.wait() [0x000000005a95f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '16'" #89 daemon prio=5 os_prio=0 tid=0x0000000056bdc800 nid=0xc98 in Object.wait() [0x000000005a79f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '15'" #88 daemon prio=5 os_prio=0 tid=0x0000000056bdb800 nid=0xb18 in Object.wait() [0x000000005a68e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '14'" #87 daemon prio=5 os_prio=0 tid=0x0000000056bdb000 nid=0x8bc in Object.wait() [0x000000005a4fe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '13'" #86 daemon prio=5 os_prio=0 tid=0x0000000056bda000 nid=0x8d8 in Object.wait() [0x0000000058b3e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '12'" #85 daemon prio=5 os_prio=0 tid=0x0000000056bd9800 nid=0x668 in Object.wait() [0x000000005a30e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '11'" #84 daemon prio=5 os_prio=0 tid=0x0000000056bd8800 nid=0xfcc in Object.wait() [0x000000005a0be000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '10'" #83 daemon prio=5 os_prio=0 tid=0x0000000056bd8000 nid=0xef0 in Object.wait() [0x0000000059fae000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '9'" #82 daemon prio=5 os_prio=0 tid=0x0000000056bd7000 nid=0x3f0 in Object.wait() [0x0000000059eae000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '8'" #81 daemon prio=5 os_prio=0 tid=0x0000000056b3f800 nid=0x290 in Object.wait() [0x0000000059c6f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '7'" #80 daemon prio=5 os_prio=0 tid=0x0000000056b3e800 nid=0x39c in Object.wait() [0x0000000057f3e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '6'" #79 daemon prio=5 os_prio=0 tid=0x0000000056b3e000 nid=0x924 in Object.wait() [0x0000000059a9f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '5'" #78 daemon prio=5 os_prio=0 tid=0x0000000056b3d000 nid=0xd60 in Object.wait() [0x00000000598cf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '4'" #77 daemon prio=5 os_prio=0 tid=0x0000000056b3c000 nid=0x938 in Object.wait() [0x00000000597cf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '3'" #76 daemon prio=5 os_prio=0 tid=0x0000000056b3b800 nid=0xe7c in Object.wait() [0x00000000596af000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '2'" #75 daemon prio=5 os_prio=0 tid=0x0000000056b3a800 nid=0xf30 in Object.wait() [0x000000005952e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '1'" #74 daemon prio=5 os_prio=0 tid=0x0000000056b3a000 nid=0x12c in Object.wait() [0x000000005935f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '0'" #73 daemon prio=5 os_prio=0 tid=0x0000000056b39000 nid=0x4f4 in Object.wait() [0x000000005919f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"History cleanup thread" #72 daemon prio=5 os_prio=0 tid=0x0000000056b38800 nid=0xec4 in Object.wait() [0x00000000588ff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ffe18000> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.HistoryCleanupThread.run(HistoryCleanupThread.java:78)

"Set priority thread" #71 daemon prio=10 os_prio=2 tid=0x0000000056abb000 nid=0x6bc in Object.wait() [0x0000000058fce000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff4abfe8> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.SetPriorityThread.run(SetPriorityThread.java:148)

"Expire stuffer thread" #70 daemon prio=6 os_prio=0 tid=0x0000000056aba800 nid=0xdf8 in Object.wait() [0x0000000058ece000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.core.lockmanager.LockGate.waitForPermission(LockGate.java:136)
        - locked <0x00000000f627b588> (a org.apache.manifoldcf.core.lockmanager.LockGate)
        at org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:258)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1522)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:280)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
        at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
        at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:862)
        at org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:236)
        at org.apache.manifoldcf.crawler.jobs.Jobs.activeJobsPresent(Jobs.java:3171)
        at org.apache.manifoldcf.crawler.jobs.JobManager.getExpiredDocuments(JobManager.java:2441)
        at org.apache.manifoldcf.crawler.system.ExpireStufferThread.run(ExpireStufferThread.java:121)

"Stuffer thread" #69 daemon prio=6 os_prio=0 tid=0x0000000056ab9800 nid=0x990 in Object.wait() [0x0000000058d6e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.core.lockmanager.LockGate.waitForPermission(LockGate.java:136)
        - locked <0x00000000f627b588> (a org.apache.manifoldcf.core.lockmanager.LockGate)
        at org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:258)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1522)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:280)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
        at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
        at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:862)
        at org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:236)
        at org.apache.manifoldcf.crawler.jobs.Jobs.activeJobsPresent(Jobs.java:3171)
        at org.apache.manifoldcf.crawler.jobs.JobManager.getNextDocuments(JobManager.java:2698)
        at org.apache.manifoldcf.crawler.system.StufferThread.run(StufferThread.java:186)

"Job delete thread" #68 daemon prio=5 os_prio=0 tid=0x0000000056ab9000 nid=0x270 in Object.wait() [0x0000000058c5f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff3f2de0> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.JobDeleteThread.run(JobDeleteThread.java:78)

"Job notification thread" #67 daemon prio=5 os_prio=0 tid=0x0000000056ab8000 nid=0x820 in Object.wait() [0x0000000058a3f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff5d45b8> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.JobNotificationThread.run(JobNotificationThread.java:431)

"Finisher thread" #66 daemon prio=5 os_prio=0 tid=0x0000000056ab7800 nid=0xda4 in Object.wait() [0x00000000587ff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff451868> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.FinisherThread.run(FinisherThread.java:69)

"Delete startup thread" #65 daemon prio=5 os_prio=0 tid=0x0000000056ab6800 nid=0x580 in Object.wait() [0x000000005867f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff50fe98> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.StartDeleteThread.run(StartDeleteThread.java:93)

"Startup thread" #64 daemon prio=5 os_prio=0 tid=0x0000000056ab6000 nid=0xda0 in Object.wait() [0x000000005846e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.core.lockmanager.LockObject.enterWriteLock(LockObject.java:80)
        - locked <0x00000000f627b5f8> (a org.apache.manifoldcf.core.lockmanager.ZooKeeperLockObject)
        at org.apache.manifoldcf.core.lockmanager.LockGate.enterWriteLock(LockGate.java:177)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1473)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:299)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
        at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
        at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:661)
        at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performUpdate(DBInterfacePostgreSQL.java:254)
        at org.apache.manifoldcf.core.database.BaseTable.performUpdate(BaseTable.java:80)
        at org.apache.manifoldcf.crawler.jobs.Jobs.noteJobStarted(Jobs.java:2322)
        at org.apache.manifoldcf.crawler.jobs.JobManager.noteJobStarted(JobManager.java:6508)
        at org.apache.manifoldcf.crawler.system.StartupThread.run(StartupThread.java:207)

"Job start thread" #63 daemon prio=5 os_prio=0 tid=0x0000000056ab5000 nid=0xc60 in Object.wait() [0x00000000581df000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff639398> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.JobStartThread.run(JobStartThread.java:96)

"Connection pool reaper" #17 daemon prio=5 os_prio=0 tid=0x0000000056aa0000 nid=0xa94 waiting on condition [0x00000000578df000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.manifoldcf.core.jdbcpool.ConnectionPoolManager$ConnectionCloserThread.run(ConnectionPoolManager.java:152)

"Agents thread" #16 daemon prio=5 os_prio=0 tid=0x0000000056bd2800 nid=0xf3c in Object.wait() [0x0000000057aff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff4ddce8> (a java.lang.Integer)
        at org.apache.manifoldcf.agents.system.AgentsDaemon$AgentsThread.run(AgentsDaemon.java:210)

"Idle cleanup thread" #15 daemon prio=5 os_prio=0 tid=0x0000000056bd1000 nid=0xe0c in Object.wait() [0x00000000579df000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff58b430> (a java.lang.Integer)
        at org.apache.manifoldcf.agents.system.IdleCleanupThread.run(IdleCleanupThread.java:96)

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x000000005506c800 nid=0xd2c runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #8 daemon prio=9 os_prio=2 tid=0x0000000054fe8000 nid=0xcbc waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x0000000054fe0800 nid=0x90c waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x000000005352a000 nid=0xdb8 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x0000000053529000 nid=0xc20 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000000053526000 nid=0xdc4 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x00000000534c3800 nid=0xef4 in Object.wait() [0x00000000549bf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        - locked <0x00000000f0056410> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x00000000534ba000 nid=0xd68 in Object.wait() [0x000000005484f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
        - locked <0x00000000f0171988> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x000000000231f000 nid=0xcf8 in Object.wait() [0x000000000253f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff556a88> (a java.lang.Integer)
        at org.apache.manifoldcf.agents.system.AgentsDaemon.runAgents(AgentsDaemon.java:115)
        at org.apache.manifoldcf.agents.AgentRun.doExecute(AgentRun.java:64)
        at org.apache.manifoldcf.agents.BaseAgentsInitializationCommand.execute(BaseAgentsInitializationCommand.java:37)
        at org.apache.manifoldcf.agents.AgentRun.main(AgentRun.java:93)

"VM Thread" os_prio=2 tid=0x00000000534b6000 nid=0xc44 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x000000000218d000 nid=0x1e8 runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x000000000218e800 nid=0xdb0 runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000000002190000 nid=0x900 runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000000002193800 nid=0xca0 runnable

"VM Periodic Task Thread" os_prio=2 tid=0x000000005506f800 nid=0xe74 waiting on condition

JNI global references: 269

Heap
 PSYoungGen      total 86016K, used 77935K [0x00000000fab00000, 0x0000000100000000, 0x0000000100000000)
  eden space 84992K, 91% used [0x00000000fab00000,0x00000000ff6a5b20,0x00000000ffe00000)
  from space 1024K, 62% used [0x00000000ffe00000,0x00000000ffea0000,0x00000000fff00000)
  to   space 1024K, 0% used [0x00000000fff00000,0x00000000fff00000,0x0000000100000000)
 ParOldGen       total 175104K, used 116435K [0x00000000f0000000, 0x00000000fab00000, 0x00000000fab00000)
  object space 175104K, 66% used [0x00000000f0000000,0x00000000f71b4c90,0x00000000fab00000)
 Metaspace       used 19235K, capacity 19524K, committed 20096K, reserved 1067008K
  class space    used 1939K, capacity 2034K, committed 2176K, reserved 1048576K

{code}




was (Author: kwright@metacarta.com):
Thread dump is consistent with a lock that hasn't been freed.

{code}
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.31-b07 mixed mode):

"Set priority thread-EventThread" #30657286 daemon prio=10 os_prio=2 tid=0x000000006e79b800 nid=0x149c waiting on condition [0x0000000054cee000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f710cc40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Set priority thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30657285 daemon prio=10 os_prio=2 tid=0x000000006e79d000 nid=0xd4c runnable [0x000000005830f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f710fdc0> (a sun.nio.ch.Util$2)
        - locked <0x00000000f710fdb0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f710fc88> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Agents thread-EventThread" #30657284 daemon prio=5 os_prio=0 tid=0x0000000056dbb800 nid=0xab4 waiting on condition [0x0000000057dff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f7112c40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Agents thread-SendThread(127.0.0.1:8349)" #30657283 daemon prio=5 os_prio=0 tid=0x000000006b2e3800 nid=0x11e8 runnable [0x000000005f96f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f710df30> (a sun.nio.ch.Util$2)
        - locked <0x00000000f710df20> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f710dea0> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Idle cleanup thread-EventThread" #30657282 daemon prio=5 os_prio=0 tid=0x000000006c1b8000 nid=0x928 waiting on condition [0x000000005f24f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f7113470> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Idle cleanup thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30657281 daemon prio=5 os_prio=0 tid=0x000000006c1b7000 nid=0x1040 runnable [0x0000000055f8e000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f71144d0> (a sun.nio.ch.Util$2)
        - locked <0x00000000f71144c0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f7113f90> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Finisher thread-EventThread" #30650263 daemon prio=5 os_prio=0 tid=0x000000005768f800 nid=0x163c waiting on condition [0x000000005ef2f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f70e6bb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Finisher thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30650262 daemon prio=5 os_prio=0 tid=0x000000005768e800 nid=0x1158 runnable [0x000000005f02e000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f70e9d30> (a sun.nio.ch.Util$2)
        - locked <0x00000000f70e9d20> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f70e9bf8> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Job delete thread-EventThread" #30650034 daemon prio=5 os_prio=0 tid=0x000000006b2e4800 nid=0x10e8 waiting on condition [0x000000005f82f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f7098c48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Job delete thread-SendThread(127.0.0.1:8349)" #30650033 daemon prio=5 os_prio=0 tid=0x000000006c1b8800 nid=0x8fc runnable [0x00000000566af000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f709ba78> (a sun.nio.ch.Util$2)
        - locked <0x00000000f709ba68> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f709b9e8> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Job reset thread-EventThread" #30650028 daemon prio=5 os_prio=0 tid=0x000000006e79c800 nid=0x17a8 waiting on condition [0x000000005c89e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f709dcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Job reset thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30650027 daemon prio=5 os_prio=0 tid=0x000000006e79b000 nid=0xcec runnable [0x000000005c4df000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f709e9d8> (a sun.nio.ch.Util$2)
        - locked <0x00000000f709e9c8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f709e530> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Job reset thread-EventThread" #30649986 daemon prio=5 os_prio=0 tid=0x0000000056f19800 nid=0x1490 waiting on condition [0x000000005f42f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f70a6aa8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Job reset thread-SendThread(127.0.0.1:8349)" #30649985 daemon prio=5 os_prio=0 tid=0x0000000056f12800 nid=0xacc runnable [0x000000005619e000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f70aab20> (a sun.nio.ch.Util$2)
        - locked <0x00000000f70aab10> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f70aaa90> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Agents thread-EventThread" #30640967 daemon prio=5 os_prio=0 tid=0x000000006c1b6800 nid=0x1120 waiting on condition [0x000000005fafe000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f6fe3c30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Agents thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30640966 daemon prio=5 os_prio=0 tid=0x000000006c1b5800 nid=0x12e0 runnable [0x000000005d28f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f6fe6db0> (a sun.nio.ch.Util$2)
        - locked <0x00000000f6fe6da0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f6fe6c78> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Set priority thread-EventThread" #30595106 daemon prio=10 os_prio=2 tid=0x0000000056c28800 nid=0x15f8 waiting on condition [0x000000005fc0f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f63607e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Set priority thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30595105 daemon prio=10 os_prio=2 tid=0x0000000056c28000 nid=0x12f0 runnable [0x000000005fd8f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f6362598> (a sun.nio.ch.Util$2)
        - locked <0x00000000f6362588> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f6362318> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Idle cleanup thread-EventThread" #30594300 daemon prio=5 os_prio=0 tid=0x000000006b2e1800 nid=0x10c8 waiting on condition [0x000000005fedf000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f6367fa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Idle cleanup thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30594299 daemon prio=5 os_prio=0 tid=0x000000006b2e2000 nid=0x1540 runnable [0x000000005f68f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f6368a58> (a sun.nio.ch.Util$2)
        - locked <0x00000000f6368a48> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f6368740> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Expire stuffer thread-EventThread" #30594028 daemon prio=6 os_prio=0 tid=0x00000000577bb000 nid=0x157c waiting on condition [0x000000005eddf000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f6369b98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Expire stuffer thread-SendThread(127.0.0.1:8349)" #30594027 daemon prio=6 os_prio=0 tid=0x00000000550c9000 nid=0x118c runnable [0x000000005804f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f63786f0> (a sun.nio.ch.Util$2)
        - locked <0x00000000f63786e0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f6378398> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Document cleanup stuffer thread-EventThread" #30594022 daemon prio=5 os_prio=0 tid=0x0000000056f13000 nid=0x1634 waiting on condition [0x000000005ec0e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f637e0a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"Document cleanup stuffer thread-SendThread(0:0:0:0:0:0:0:1:8349)" #30594021 daemon prio=5 os_prio=0 tid=0x0000000056f18800 nid=0x1488 runnable [0x000000005646e000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000000f637b120> (a sun.nio.ch.Util$2)
        - locked <0x00000000f637b110> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f637b090> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

"Connection pool reaper" #2098506 daemon prio=5 os_prio=0 tid=0x000000006e79a000 nid=0xf58 waiting on condition [0x000000006cf3f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.manifoldcf.core.jdbcpool.ConnectionPoolManager$ConnectionCloserThread.run(ConnectionPoolManager.java:152)

"Assessment thread" #138 daemon prio=5 os_prio=0 tid=0x0000000056fa6800 nid=0xd90 in Object.wait() [0x000000005e2ef000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff522940> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.AssessmentThread.run(AssessmentThread.java:68)

"Idle cleanup thread" #137 daemon prio=5 os_prio=0 tid=0x0000000056fa5800 nid=0xc94 in Object.wait() [0x000000005e91f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff575968> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.IdleCleanupThread.run(IdleCleanupThread.java:74)

"Seeding thread" #136 daemon prio=5 os_prio=0 tid=0x0000000056fa5000 nid=0xd94 in Object.wait() [0x000000005e6ee000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000feea03e0> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.SeedingThread.run(SeedingThread.java:104)

"Job reset thread" #135 daemon prio=5 os_prio=0 tid=0x0000000056fa4000 nid=0x810 in Object.wait() [0x000000005dedf000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff3cd530> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.JobResetThread.run(JobResetThread.java:126)

"Document delete thread '9'" #123 daemon prio=5 os_prio=0 tid=0x0000000056fa3800 nid=0x91c in Object.wait() [0x000000005e5df000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '8'" #122 daemon prio=5 os_prio=0 tid=0x0000000056fa2800 nid=0xd74 in Object.wait() [0x000000005e42f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '7'" #121 daemon prio=5 os_prio=0 tid=0x0000000056fa2000 nid=0xd80 in Object.wait() [0x000000005856f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '6'" #120 daemon prio=5 os_prio=0 tid=0x0000000056fa1000 nid=0xe4c in Object.wait() [0x000000005e1cf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '5'" #119 daemon prio=5 os_prio=0 tid=0x0000000056fa0800 nid=0x988 in Object.wait() [0x000000005dfde000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '4'" #118 daemon prio=5 os_prio=0 tid=0x0000000056f92000 nid=0x4e0 in Object.wait() [0x000000005ddbe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '3'" #117 daemon prio=5 os_prio=0 tid=0x0000000056f91800 nid=0xf9c in Object.wait() [0x000000005d08f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '2'" #116 daemon prio=5 os_prio=0 tid=0x0000000056f90800 nid=0xf88 in Object.wait() [0x000000005dc6f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '1'" #115 daemon prio=5 os_prio=0 tid=0x0000000056f90000 nid=0x2dc in Object.wait() [0x000000005da7f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete thread '0'" #114 daemon prio=5 os_prio=0 tid=0x0000000056f8f000 nid=0xc74 in Object.wait() [0x000000005d94e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0216580> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
        - locked <0x00000000f0216580> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:95)

"Document delete stuffer thread" #113 daemon prio=5 os_prio=0 tid=0x0000000056f8e800 nid=0xc48 in Object.wait() [0x000000005ae8f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.core.lockmanager.LockGate.waitForPermission(LockGate.java:136)
        - locked <0x00000000f627b588> (a org.apache.manifoldcf.core.lockmanager.LockGate)
        at org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:258)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1522)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:280)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
        at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
        at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:862)
        at org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:236)
        at org.apache.manifoldcf.crawler.jobs.Jobs.deletingJobsPresent(Jobs.java:3133)
        at org.apache.manifoldcf.crawler.jobs.JobManager.getNextDeletableDocuments(JobManager.java:1862)
        at org.apache.manifoldcf.crawler.system.DocumentDeleteStufferThread.run(DocumentDeleteStufferThread.java:108)

"Document cleanup thread '9'" #134 daemon prio=5 os_prio=0 tid=0x0000000056f8d800 nid=0x8dc in Object.wait() [0x000000005d7ef000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '8'" #133 daemon prio=5 os_prio=0 tid=0x0000000056f8c800 nid=0xe04 in Object.wait() [0x000000005d6cf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '7'" #132 daemon prio=5 os_prio=0 tid=0x0000000056f8c000 nid=0x4b4 in Object.wait() [0x000000005d4fe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '6'" #131 daemon prio=5 os_prio=0 tid=0x0000000056f8b000 nid=0xd38 in Object.wait() [0x000000005d39e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '5'" #130 daemon prio=5 os_prio=0 tid=0x0000000056f3d800 nid=0xec8 in Object.wait() [0x000000005d18f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '4'" #129 daemon prio=5 os_prio=0 tid=0x0000000056f3d000 nid=0xee8 in Object.wait() [0x000000005cf1f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '3'" #128 daemon prio=5 os_prio=0 tid=0x0000000056f3c000 nid=0x9d8 in Object.wait() [0x000000005cd8e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '2'" #127 daemon prio=5 os_prio=0 tid=0x0000000056f3b800 nid=0x2b8 in Object.wait() [0x000000005b7ce000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '1'" #126 daemon prio=5 os_prio=0 tid=0x0000000056f3a800 nid=0xf84 in Object.wait() [0x000000005cbef000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup thread '0'" #125 daemon prio=5 os_prio=0 tid=0x0000000056f3a000 nid=0xf4c in Object.wait() [0x000000005c78f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f025aae8> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f025aae8> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:106)

"Document cleanup stuffer thread" #124 daemon prio=5 os_prio=0 tid=0x0000000056f39000 nid=0xf98 in Object.wait() [0x000000005ca6e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.core.lockmanager.LockGate.waitForPermission(LockGate.java:136)
        - locked <0x00000000f627b588> (a org.apache.manifoldcf.core.lockmanager.LockGate)
        at org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:258)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1522)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:280)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
        at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
        at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:862)
        at org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:236)
        at org.apache.manifoldcf.crawler.jobs.Jobs.cleaningJobsPresent(Jobs.java:3150)
        at org.apache.manifoldcf.crawler.jobs.JobManager.getNextCleanableDocuments(JobManager.java:1618)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupStufferThread.run(DocumentCleanupStufferThread.java:108)

"Expiration thread '9'" #112 daemon prio=5 os_prio=0 tid=0x0000000056f37000 nid=0xc0c in Object.wait() [0x000000005b9de000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '8'" #111 daemon prio=5 os_prio=0 tid=0x0000000056c72000 nid=0x134 in Object.wait() [0x000000005c64e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '7'" #110 daemon prio=5 os_prio=0 tid=0x0000000056c71800 nid=0xff8 in Object.wait() [0x000000005c06f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '6'" #109 daemon prio=5 os_prio=0 tid=0x0000000056c70800 nid=0x9f0 in Object.wait() [0x000000005c3ce000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '5'" #108 daemon prio=5 os_prio=0 tid=0x0000000056c70000 nid=0x498 in Object.wait() [0x000000005c2cf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '4'" #107 daemon prio=5 os_prio=0 tid=0x0000000056c6f000 nid=0x568 in Object.wait() [0x000000005c1bf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '3'" #106 daemon prio=5 os_prio=0 tid=0x0000000056c6e800 nid=0xd84 in Object.wait() [0x000000005bf4f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '2'" #105 daemon prio=5 os_prio=0 tid=0x0000000056c6d800 nid=0x244 in Object.wait() [0x000000005be0f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '1'" #104 daemon prio=5 os_prio=0 tid=0x0000000056c6d000 nid=0xe38 in Object.wait() [0x000000005bc3f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Expiration thread '0'" #103 daemon prio=5 os_prio=0 tid=0x0000000056c6c000 nid=0xff4 in Object.wait() [0x000000005bade000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000f0275528> (a java.util.ArrayList)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:107)
        - locked <0x00000000f0275528> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:95)

"Worker thread '29'" #102 daemon prio=5 os_prio=0 tid=0x0000000056c6b800 nid=0xf48 in Object.wait() [0x000000005b8df000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '28'" #101 daemon prio=5 os_prio=0 tid=0x0000000056bf0000 nid=0x9b4 in Object.wait() [0x000000005b6cf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '27'" #100 daemon prio=5 os_prio=0 tid=0x0000000056bef000 nid=0x5fc in Object.wait() [0x0000000059d8f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '26'" #99 daemon prio=5 os_prio=0 tid=0x0000000056bee800 nid=0x2a4 in Object.wait() [0x000000005b4fe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '25'" #98 daemon prio=5 os_prio=0 tid=0x0000000056bed800 nid=0xa20 in Object.wait() [0x000000005b3be000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '24'" #97 daemon prio=5 os_prio=0 tid=0x0000000056bed000 nid=0xa90 in Object.wait() [0x000000005b28e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '23'" #96 daemon prio=5 os_prio=0 tid=0x0000000056bec000 nid=0xe90 in Object.wait() [0x000000005b12f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '22'" #95 daemon prio=5 os_prio=0 tid=0x0000000056beb800 nid=0x998 in Object.wait() [0x000000005af8f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '21'" #94 daemon prio=5 os_prio=0 tid=0x0000000056bea800 nid=0x80c in Object.wait() [0x000000005ad8e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '20'" #93 daemon prio=5 os_prio=0 tid=0x0000000056bea000 nid=0x2e0 in Object.wait() [0x000000005ac0e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '19'" #92 daemon prio=5 os_prio=0 tid=0x0000000056be9000 nid=0x2ec in Object.wait() [0x000000005a1bf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '18'" #91 daemon prio=5 os_prio=0 tid=0x0000000056bde000 nid=0xd54 in Object.wait() [0x000000005ab0f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '17'" #90 daemon prio=5 os_prio=0 tid=0x0000000056bdd000 nid=0x5a4 in Object.wait() [0x000000005a95f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '16'" #89 daemon prio=5 os_prio=0 tid=0x0000000056bdc800 nid=0xc98 in Object.wait() [0x000000005a79f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '15'" #88 daemon prio=5 os_prio=0 tid=0x0000000056bdb800 nid=0xb18 in Object.wait() [0x000000005a68e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '14'" #87 daemon prio=5 os_prio=0 tid=0x0000000056bdb000 nid=0x8bc in Object.wait() [0x000000005a4fe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '13'" #86 daemon prio=5 os_prio=0 tid=0x0000000056bda000 nid=0x8d8 in Object.wait() [0x0000000058b3e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '12'" #85 daemon prio=5 os_prio=0 tid=0x0000000056bd9800 nid=0x668 in Object.wait() [0x000000005a30e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '11'" #84 daemon prio=5 os_prio=0 tid=0x0000000056bd8800 nid=0xfcc in Object.wait() [0x000000005a0be000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '10'" #83 daemon prio=5 os_prio=0 tid=0x0000000056bd8000 nid=0xef0 in Object.wait() [0x0000000059fae000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '9'" #82 daemon prio=5 os_prio=0 tid=0x0000000056bd7000 nid=0x3f0 in Object.wait() [0x0000000059eae000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '8'" #81 daemon prio=5 os_prio=0 tid=0x0000000056b3f800 nid=0x290 in Object.wait() [0x0000000059c6f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '7'" #80 daemon prio=5 os_prio=0 tid=0x0000000056b3e800 nid=0x39c in Object.wait() [0x0000000057f3e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '6'" #79 daemon prio=5 os_prio=0 tid=0x0000000056b3e000 nid=0x924 in Object.wait() [0x0000000059a9f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '5'" #78 daemon prio=5 os_prio=0 tid=0x0000000056b3d000 nid=0xd60 in Object.wait() [0x00000000598cf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '4'" #77 daemon prio=5 os_prio=0 tid=0x0000000056b3c000 nid=0x938 in Object.wait() [0x00000000597cf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '3'" #76 daemon prio=5 os_prio=0 tid=0x0000000056b3b800 nid=0xe7c in Object.wait() [0x00000000596af000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '2'" #75 daemon prio=5 os_prio=0 tid=0x0000000056b3a800 nid=0xf30 in Object.wait() [0x000000005952e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '1'" #74 daemon prio=5 os_prio=0 tid=0x0000000056b3a000 nid=0x12c in Object.wait() [0x000000005935f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"Worker thread '0'" #73 daemon prio=5 os_prio=0 tid=0x0000000056b39000 nid=0x4f4 in Object.wait() [0x000000005919f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.crawler.system.DocumentQueue.getDocument(DocumentQueue.java:110)
        - locked <0x00000000f027d248> (a java.util.ArrayList)
        at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:121)

"History cleanup thread" #72 daemon prio=5 os_prio=0 tid=0x0000000056b38800 nid=0xec4 in Object.wait() [0x00000000588ff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ffe18000> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.HistoryCleanupThread.run(HistoryCleanupThread.java:78)

"Set priority thread" #71 daemon prio=10 os_prio=2 tid=0x0000000056abb000 nid=0x6bc in Object.wait() [0x0000000058fce000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff4abfe8> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.SetPriorityThread.run(SetPriorityThread.java:148)

"Expire stuffer thread" #70 daemon prio=6 os_prio=0 tid=0x0000000056aba800 nid=0xdf8 in Object.wait() [0x0000000058ece000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.core.lockmanager.LockGate.waitForPermission(LockGate.java:136)
        - locked <0x00000000f627b588> (a org.apache.manifoldcf.core.lockmanager.LockGate)
        at org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:258)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1522)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:280)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
        at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
        at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:862)
        at org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:236)
        at org.apache.manifoldcf.crawler.jobs.Jobs.activeJobsPresent(Jobs.java:3171)
        at org.apache.manifoldcf.crawler.jobs.JobManager.getExpiredDocuments(JobManager.java:2441)
        at org.apache.manifoldcf.crawler.system.ExpireStufferThread.run(ExpireStufferThread.java:121)

"Stuffer thread" #69 daemon prio=6 os_prio=0 tid=0x0000000056ab9800 nid=0x990 in Object.wait() [0x0000000058d6e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.core.lockmanager.LockGate.waitForPermission(LockGate.java:136)
        - locked <0x00000000f627b588> (a org.apache.manifoldcf.core.lockmanager.LockGate)
        at org.apache.manifoldcf.core.lockmanager.LockGate.enterReadLock(LockGate.java:258)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1522)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:280)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
        at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
        at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:862)
        at org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:236)
        at org.apache.manifoldcf.crawler.jobs.Jobs.activeJobsPresent(Jobs.java:3171)
        at org.apache.manifoldcf.crawler.jobs.JobManager.getNextDocuments(JobManager.java:2698)
        at org.apache.manifoldcf.crawler.system.StufferThread.run(StufferThread.java:186)

"Job delete thread" #68 daemon prio=5 os_prio=0 tid=0x0000000056ab9000 nid=0x270 in Object.wait() [0x0000000058c5f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff3f2de0> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.JobDeleteThread.run(JobDeleteThread.java:78)

"Job notification thread" #67 daemon prio=5 os_prio=0 tid=0x0000000056ab8000 nid=0x820 in Object.wait() [0x0000000058a3f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff5d45b8> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.JobNotificationThread.run(JobNotificationThread.java:431)

"Finisher thread" #66 daemon prio=5 os_prio=0 tid=0x0000000056ab7800 nid=0xda4 in Object.wait() [0x00000000587ff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff451868> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.FinisherThread.run(FinisherThread.java:69)

"Delete startup thread" #65 daemon prio=5 os_prio=0 tid=0x0000000056ab6800 nid=0x580 in Object.wait() [0x000000005867f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff50fe98> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.StartDeleteThread.run(StartDeleteThread.java:93)

"Startup thread" #64 daemon prio=5 os_prio=0 tid=0x0000000056ab6000 nid=0xda0 in Object.wait() [0x000000005846e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at org.apache.manifoldcf.core.lockmanager.LockObject.enterWriteLock(LockObject.java:80)
        - locked <0x00000000f627b5f8> (a org.apache.manifoldcf.core.lockmanager.ZooKeeperLockObject)
        at org.apache.manifoldcf.core.lockmanager.LockGate.enterWriteLock(LockGate.java:177)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enter(BaseLockManager.java:1473)
        at org.apache.manifoldcf.core.lockmanager.BaseLockManager.enterLocks(BaseLockManager.java:803)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCache(CacheManager.java:299)
        at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:100)
        at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204)
        at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:661)
        at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performUpdate(DBInterfacePostgreSQL.java:254)
        at org.apache.manifoldcf.core.database.BaseTable.performUpdate(BaseTable.java:80)
        at org.apache.manifoldcf.crawler.jobs.Jobs.noteJobStarted(Jobs.java:2322)
        at org.apache.manifoldcf.crawler.jobs.JobManager.noteJobStarted(JobManager.java:6508)
        at org.apache.manifoldcf.crawler.system.StartupThread.run(StartupThread.java:207)

"Job start thread" #63 daemon prio=5 os_prio=0 tid=0x0000000056ab5000 nid=0xc60 in Object.wait() [0x00000000581df000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff639398> (a java.lang.Integer)
        at org.apache.manifoldcf.crawler.system.JobStartThread.run(JobStartThread.java:96)

"Connection pool reaper" #17 daemon prio=5 os_prio=0 tid=0x0000000056aa0000 nid=0xa94 waiting on condition [0x00000000578df000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.manifoldcf.core.jdbcpool.ConnectionPoolManager$ConnectionCloserThread.run(ConnectionPoolManager.java:152)

"Agents thread" #16 daemon prio=5 os_prio=0 tid=0x0000000056bd2800 nid=0xf3c in Object.wait() [0x0000000057aff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff4ddce8> (a java.lang.Integer)
        at org.apache.manifoldcf.agents.system.AgentsDaemon$AgentsThread.run(AgentsDaemon.java:210)

"Idle cleanup thread" #15 daemon prio=5 os_prio=0 tid=0x0000000056bd1000 nid=0xe0c in Object.wait() [0x00000000579df000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff58b430> (a java.lang.Integer)
        at org.apache.manifoldcf.agents.system.IdleCleanupThread.run(IdleCleanupThread.java:96)

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x000000005506c800 nid=0xd2c runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #8 daemon prio=9 os_prio=2 tid=0x0000000054fe8000 nid=0xcbc waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x0000000054fe0800 nid=0x90c waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x000000005352a000 nid=0xdb8 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x0000000053529000 nid=0xc20 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000000053526000 nid=0xdc4 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x00000000534c3800 nid=0xef4 in Object.wait() [0x00000000549bf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        - locked <0x00000000f0056410> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x00000000534ba000 nid=0xd68 in Object.wait() [0x000000005484f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Unknown Source)
        at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
        - locked <0x00000000f0171988> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x000000000231f000 nid=0xcf8 in Object.wait() [0x000000000253f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:1233)
        - locked <0x00000000ff556a88> (a java.lang.Integer)
        at org.apache.manifoldcf.agents.system.AgentsDaemon.runAgents(AgentsDaemon.java:115)
        at org.apache.manifoldcf.agents.AgentRun.doExecute(AgentRun.java:64)
        at org.apache.manifoldcf.agents.BaseAgentsInitializationCommand.execute(BaseAgentsInitializationCommand.java:37)
        at org.apache.manifoldcf.agents.AgentRun.main(AgentRun.java:93)

"VM Thread" os_prio=2 tid=0x00000000534b6000 nid=0xc44 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x000000000218d000 nid=0x1e8 runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x000000000218e800 nid=0xdb0 runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000000002190000 nid=0x900 runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000000002193800 nid=0xca0 runnable

"VM Periodic Task Thread" os_prio=2 tid=0x000000005506f800 nid=0xe74 waiting on condition

JNI global references: 269

Heap
 PSYoungGen      total 86016K, used 77935K [0x00000000fab00000, 0x0000000100000000, 0x0000000100000000)
  eden space 84992K, 91% used [0x00000000fab00000,0x00000000ff6a5b20,0x00000000ffe00000)
  from space 1024K, 62% used [0x00000000ffe00000,0x00000000ffea0000,0x00000000fff00000)
  to   space 1024K, 0% used [0x00000000fff00000,0x00000000fff00000,0x0000000100000000)
 ParOldGen       total 175104K, used 116435K [0x00000000f0000000, 0x00000000fab00000, 0x00000000fab00000)
  object space 175104K, 66% used [0x00000000f0000000,0x00000000f71b4c90,0x00000000fab00000)
 Metaspace       used 19235K, capacity 19524K, committed 20096K, reserved 1067008K
  class space    used 1939K, capacity 2034K, committed 2176K, reserved 1048576K

{code}



> Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
> ------------------------------------------------------------------------------------
>
>                 Key: CONNECTORS-1395
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1395
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Framework core
>    Affects Versions: ManifoldCF 2.6
>            Reporter: Karl Wright
>            Assignee: Karl Wright
>             Fix For: ManifoldCF 2.7
>
>
> User saw this in the log, after which the system hung:
> {code}
> ERROR 2017-03-08 00:25:30,433 (Worker thread '14') - Exception tossed: Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
>                 at org.apache.manifoldcf.crawler.jobs.JobQueue.updateCompletedRecord(JobQueue.java:1019)
>                 at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentCompletedMultiple(JobManager.java:3271)
>                 at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:710)
> WARN 2017-03-08 00:25:30,449 (Worker thread '23') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:30,449 (Worker thread '24') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:30,464 (Worker thread '9') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:30,464 (Worker thread '0') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:31,900 (Worker thread '11') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:31,900 (Worker thread '29') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:32,867 (Worker thread '10') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:32,867 (Worker thread '2') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:33,335 (Worker thread '8') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:36,642 (Worker thread '20') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:37,422 (Worker thread '21') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,280 (Worker thread '22') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,280 (Worker thread '3') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,280 (Worker thread '5') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,826 (Worker thread '28') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:39,045 (Worker thread '13') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:45,425 (Worker thread '4') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:45,425 (Worker thread '15') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:45,425 (Worker thread '17') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:46,392 (Worker thread '25') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:46,392 (Worker thread '27') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:11,043 (Worker thread '1') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:35,817 (Worker thread '19') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:35,817 (Worker thread '26') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:36,753 (Worker thread '7') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:39,248 (Worker thread '6') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:39,248 (Worker thread '18') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:43,129 (Worker thread '16') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
> FATAL 2017-03-08 00:32:24,819 (Idle cleanup thread) - Error tossed: Can't release lock we don't hold
> java.lang.IllegalStateException: Can't release lock we don't hold”
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message