manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: max_pred_locks_per_transaction
Date Tue, 05 Feb 2013 13:10:06 GMT
Ok, it is clear from this that most of your threads are waiting to get
a connection, and there are no connections to be found.  This is
exactly the problem that Maciej reported, which I created the
CONNECTORS-638 ticket for.  There has to be a connection leak
somewhere.  Obviously it is not a common situation, or the problem
would arise almost right away; it probably occurs as a result of some
error condition or pathway that is relatively uncommon.

The diagnostic code that is now checked into trunk should work as follows:

(1) First, checkout and build trunk.  Since there are schema changes
in trunk vs. older versions of ManifoldCF, you cannot "go backwards"
and run an older version on a particular database instance once you've
run trunk.  Keep that in mind.

(2) Add a line to the properties.xml file, as follows:

<property name="org.apache.manifoldcf.database.connectiontracking"
value="true"/>

(3) Start the system up and let it run.

(4) When it fails, you should start to see dumps in the log like this:

            Logging.db.warn("Out of db connections, list of
outstanding ones follows.");
            for (WrappedConnection c : outstandingConnections)
            {
              Logging.db.warn("Found a possibly leaked db
connection",c.getInstantiationException());
            }

... which will dump where all the offending connections were
allocated.  Hopefully this will point us at what the problem is.  If
there seems to be no consistency here, I'll have to explore the
possibility that there are bugs in the connection allocation/free
code, but we'll see.

Karl



On Tue, Feb 5, 2013 at 7:38 AM, Erlend GarĂ¥sen <e.f.garasen@usit.uio.no> wrote:
> On 04.02.13 14.40, Karl Wright wrote:
>>
>> There's code now committed to trunk which might help diagnose this
>> problem.  It's part of CONNECTORS-638 and is meant to find database
>> connection handle leaks.
>
>
> Thanks! I will deploy trunk on our test server and start a new crawl later
> today.
>
> BTW, here's the thread dump from the Agent - generaed by usint kill -QUIT
> <pid>:
>
> "Thread-5386784" daemon prio=10 tid=0xdbb67000 nid=0x318f runnable
> [0xdc6fe000]
>    java.lang.Thread.State: RUNNABLE
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at java.net.SocketInputStream.read(SocketInputStream.java:129)
>         at
> com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
>         at
> com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
>         at
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
>         - locked <0xe41eeea0> (a java.lang.Object)
>         at
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
>         at
> com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
>         - locked <0xe41f0a48> (a
> com.sun.net.ssl.internal.ssl.AppInputStream)
>         at
> org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145)
>         at
> org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114)
>         at
> org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
>         at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274)
>         at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660)
>         at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>         - locked <0xe41efe00> (a org.postgresql.core.v3.QueryExecutorImpl)
>         at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
>         at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
>         at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
>         at
> org.apache.manifoldcf.core.database.Database.execute(Database.java:826)
>         at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641)
>
> "Thread-5386763" daemon prio=10 tid=0xd8202800 nid=0x3171 runnable
> [0xdc369000]
>    java.lang.Thread.State: RUNNABLE
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at java.net.SocketInputStream.read(SocketInputStream.java:129)
>         at
> com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
>         at
> com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
>         at
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
>         - locked <0xe41dbac8> (a java.lang.Object)
>         at
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
>         at
> com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
>         - locked <0xe41dcd78> (a
> com.sun.net.ssl.internal.ssl.AppInputStream)
>         at
> org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:145)
>         at
> org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114)
>         at
> org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
>         at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:274)
>         at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660)
>         at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>         - locked <0xe41dd280> (a org.postgresql.core.v3.QueryExecutorImpl)
>         at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
>         at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
>         at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
>         at
> org.apache.manifoldcf.core.database.Database.execute(Database.java:840)
>         at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:641)
>
> "MultiThreadedHttpConnectionManager cleanup" daemon prio=10 tid=0xddc92800
> nid=0x1ca in Object.wait() [0xdccad000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>         - locked <0xe418ff40> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>         at
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(Unknown
> Source)
>
> "Idle cleanup thread" daemon prio=10 tid=0xde729800 nid=0x1f56 in
> Object.wait() [0xdd3e8000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at
> org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:919)
>         - locked <0xf272fe90> (a java.lang.Integer)
>         at
> org.apache.manifoldcf.crawler.system.IdleCleanupThread.run(IdleCleanupThread.java:70)
>
> "Seeding thread" daemon prio=10 tid=0xde728000 nid=0x1f55 in Object.wait()
> [0xdcfcb000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:250)
>         at
> org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:237)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1382)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
>         at
> org.apache.manifoldcf.crawler.jobs.JobManager.getJobsReadyForSeeding(JobManager.java:5706)
>         at
> org.apache.manifoldcf.crawler.system.SeedingThread.run(SeedingThread.java:94)
>
> "Job reset thread" daemon prio=10 tid=0xde726800 nid=0x1f53 in Object.wait()
> [0xdd06d000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
>         at
> org.apache.manifoldcf.crawler.jobs.JobManager.finishJobStops(JobManager.java:6673)
>         at
> org.apache.manifoldcf.crawler.system.JobResetThread.run(JobResetThread.java:68)
>
> "Document delete thread '9'" daemon prio=10 tid=0xde725000 nid=0x1f52 in
> Object.wait() [0xdd0be000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb3068> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
>         - locked <0xe3fb3068> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '8'" daemon prio=10 tid=0xde724000 nid=0x1f51 in
> Object.wait() [0xdd2f5000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb3068> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
>         - locked <0xe3fb3068> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '7'" daemon prio=10 tid=0xde723400 nid=0x1f50 in
> Object.wait() [0xdd10f000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb3068> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
>         - locked <0xe3fb3068> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '6'" daemon prio=10 tid=0xde721c00 nid=0x1f4f in
> Object.wait() [0xdd160000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb3068> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
>         - locked <0xe3fb3068> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '5'" daemon prio=10 tid=0xde720400 nid=0x1f4e in
> Object.wait() [0xdd1b1000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb3068> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
>         - locked <0xe3fb3068> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '4'" daemon prio=10 tid=0xde71ec00 nid=0x1f4d in
> Object.wait() [0xdd439000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb3068> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
>         - locked <0xe3fb3068> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '3'" daemon prio=10 tid=0xde71e000 nid=0x1f4c in
> Object.wait() [0xdd202000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb3068> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
>         - locked <0xe3fb3068> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '2'" daemon prio=10 tid=0xde71c800 nid=0x1f4b in
> Object.wait() [0xdd253000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb3068> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
>         - locked <0xe3fb3068> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '1'" daemon prio=10 tid=0xde71b000 nid=0x1f48 in
> Object.wait() [0xdd346000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb3068> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
>         - locked <0xe3fb3068> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete thread '0'" daemon prio=10 tid=0xde719800 nid=0x1f47 in
> Object.wait() [0xdd397000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb3068> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteQueue.getDocuments(DocumentDeleteQueue.java:100)
>         - locked <0xe3fb3068> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteThread.run(DocumentDeleteThread.java:89)
>
> "Document delete stuffer thread" daemon prio=10 tid=0xde718400 nid=0x1f44 in
> Object.wait() [0xdd48a000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at
> org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:919)
>         - locked <0xf2706a80> (a java.lang.Integer)
>         at
> org.apache.manifoldcf.crawler.system.DocumentDeleteStufferThread.run(DocumentDeleteStufferThread.java:112)
>
> "Document cleanup thread '9'" daemon prio=10 tid=0xde717000 nid=0x1f43 in
> Object.wait() [0xdd4db000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb7778> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fb7778> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '8'" daemon prio=10 tid=0xde715c00 nid=0x1f42 in
> Object.wait() [0xdd52c000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb7778> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fb7778> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '7'" daemon prio=10 tid=0xde715400 nid=0x1f41 in
> Object.wait() [0xdde22000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb7778> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fb7778> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '6'" daemon prio=10 tid=0xde713c00 nid=0x1f40 in
> Object.wait() [0xdd57d000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb7778> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fb7778> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '5'" daemon prio=10 tid=0xde712400 nid=0x1f3f in
> Object.wait() [0xdd5ce000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb7778> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fb7778> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '4'" daemon prio=10 tid=0xde711000 nid=0x1f3e in
> Object.wait() [0xdd61f000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb7778> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fb7778> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '3'" daemon prio=10 tid=0xde70f800 nid=0x1f3d in
> Object.wait() [0xdd670000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb7778> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fb7778> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '2'" daemon prio=10 tid=0xde70e000 nid=0x1f3c in
> Object.wait() [0xdd6c1000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb7778> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fb7778> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '1'" daemon prio=10 tid=0xde70c800 nid=0x1f3b in
> Object.wait() [0xdd712000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb7778> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fb7778> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup thread '0'" daemon prio=10 tid=0xde70b000 nid=0x1f3a in
> Object.wait() [0xdd763000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fb7778> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fb7778> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupThread.run(DocumentCleanupThread.java:99)
>
> "Document cleanup stuffer thread" daemon prio=10 tid=0xddc0a400 nid=0x1f39
> in Object.wait() [0xdd7b4000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at
> org.apache.manifoldcf.core.system.ManifoldCF.sleep(ManifoldCF.java:919)
>         - locked <0xf2640178> (a java.lang.Integer)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupStufferThread.run(DocumentCleanupStufferThread.java:114)
>
> "Expiration thread '9'" daemon prio=10 tid=0xddc09400 nid=0x1f38 in
> Object.wait() [0xdd805000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fc7060> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fc7060> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '8'" daemon prio=10 tid=0xddc07c00 nid=0x1f37 in
> Object.wait() [0xdd856000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fc7060> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fc7060> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '7'" daemon prio=10 tid=0xddc06400 nid=0x1f36 in
> Object.wait() [0xdd8a7000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fc7060> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fc7060> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '6'" daemon prio=10 tid=0xddc05000 nid=0x1f35 in
> Object.wait() [0xdd8f8000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fc7060> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fc7060> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '5'" daemon prio=10 tid=0xddc03c00 nid=0x1f34 in
> Object.wait() [0xdd949000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fc7060> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fc7060> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '4'" daemon prio=10 tid=0xddc02000 nid=0x1f33 in
> Object.wait() [0xdd99a000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fc7060> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fc7060> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '3'" daemon prio=10 tid=0x0a496400 nid=0x1f32 in
> Object.wait() [0xdd9eb000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fc7060> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fc7060> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '2'" daemon prio=10 tid=0x0a494c00 nid=0x1f31 in
> Object.wait() [0xdda3c000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fc7060> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fc7060> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '1'" daemon prio=10 tid=0x0a493800 nid=0x1f30 in
> Object.wait() [0xdda8d000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fc7060> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fc7060> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Expiration thread '0'" daemon prio=10 tid=0x0a492400 nid=0x1f2f in
> Object.wait() [0xddade000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0xe3fc7060> (a java.util.ArrayList)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.crawler.system.DocumentCleanupQueue.getDocuments(DocumentCleanupQueue.java:100)
>         - locked <0xe3fc7060> (a java.util.ArrayList)
>         at
> org.apache.manifoldcf.crawler.system.ExpireThread.run(ExpireThread.java:89)
>
> "Worker thread '24'" daemon prio=10 tid=0x0a56e000 nid=0x1f2e in
> Object.wait() [0xddb2f000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '23'" daemon prio=10 tid=0x0a56c400 nid=0x1f2d in
> Object.wait() [0xddb80000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '22'" daemon prio=10 tid=0x0a56ac00 nid=0x1f2c in
> Object.wait() [0xddd80000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '21'" daemon prio=10 tid=0x0a569400 nid=0x1f2b in
> Object.wait() [0xdddd1000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '20'" daemon prio=10 tid=0xde709400 nid=0x1f2a in
> Object.wait() [0xddbd1000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '19'" daemon prio=10 tid=0x0a567400 nid=0x1f28 in
> Object.wait() [0xdde73000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '18'" daemon prio=10 tid=0x0a5a5c00 nid=0x1f27 in
> Object.wait() [0xde5ba000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '17'" daemon prio=10 tid=0xde704800 nid=0x1f26 in
> Object.wait() [0xddec4000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '16'" daemon prio=10 tid=0x0a5a4800 nid=0x1f25 in
> Object.wait() [0xddf15000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '15'" daemon prio=10 tid=0x0a5a3000 nid=0x1f24 in
> Object.wait() [0xddf66000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '14'" daemon prio=10 tid=0x0a5a1800 nid=0x1f23 in
> Object.wait() [0xddfb7000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '13'" daemon prio=10 tid=0x0a59fc00 nid=0x1f22 in
> Object.wait() [0xde008000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '12'" daemon prio=10 tid=0xde703000 nid=0x1f20 in
> Object.wait() [0xde059000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '11'" daemon prio=10 tid=0x0a25c400 nid=0x1f1f in
> Object.wait() [0xde0aa000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '10'" daemon prio=10 tid=0x0a180800 nid=0x1f1e in
> Object.wait() [0xde0fb000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '9'" daemon prio=10 tid=0xde701400 nid=0x1f1d in
> Object.wait() [0xde14c000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '8'" daemon prio=10 tid=0xdecfec00 nid=0x1f1c in
> Object.wait() [0xde19d000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Thread.join(Thread.java:1186)
>         - locked <0xea62bb98> (a
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
>         at java.lang.Thread.join(Thread.java:1239)
>         at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:673)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:803)
>         at
> org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4089)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.addDocumentReference(WorkerThread.java:1487)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityLinkHandler.noteDiscoveredLink(WebcrawlerConnector.java:6049)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityHTMLHandler.noteAHREF(WebcrawlerConnector.java:6159)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.LinkParseState.noteNonscriptTag(LinkParseState.java:44)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.FormParseState.noteNonscriptTag(FormParseState.java:52)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState.noteTag(ScriptParseState.java:50)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.BasicParseState.dealWithCharacter(BasicParseState.java:225)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.handleHTML(WebcrawlerConnector.java:7047)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.extractLinks(WebcrawlerConnector.java:6011)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.processDocuments(WebcrawlerConnector.java:1282)
>         at
> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:551)
>
> "Worker thread '7'" daemon prio=10 tid=0xdecfd400 nid=0x1f1b in
> Object.wait() [0xde1ee000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Thread.join(Thread.java:1186)
>         - locked <0xea62cb98> (a
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
>         at java.lang.Thread.join(Thread.java:1239)
>         at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:673)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:709)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.jobs.IntrinsicLink.recordReferences(IntrinsicLink.java:263)
>         at
> org.apache.manifoldcf.crawler.jobs.HopCount.doRecord(HopCount.java:409)
>         at
> org.apache.manifoldcf.crawler.jobs.HopCount.recordReferences(HopCount.java:380)
>         at
> org.apache.manifoldcf.crawler.jobs.JobManager.addDocuments(JobManager.java:4119)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.processDocumentReferences(WorkerThread.java:1932)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.addDocumentReference(WorkerThread.java:1487)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityLinkHandler.noteDiscoveredLink(WebcrawlerConnector.java:6049)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector$ProcessActivityHTMLHandler.noteAHREF(WebcrawlerConnector.java:6159)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.LinkParseState.noteNonscriptTag(LinkParseState.java:44)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.FormParseState.noteNonscriptTag(FormParseState.java:52)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ScriptParseState.noteTag(ScriptParseState.java:50)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.BasicParseState.dealWithCharacter(BasicParseState.java:225)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.handleHTML(WebcrawlerConnector.java:7047)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.extractLinks(WebcrawlerConnector.java:6011)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.processDocuments(WebcrawlerConnector.java:1282)
>         at
> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:551)
>
> "Worker thread '6'" daemon prio=10 tid=0xdecfbc00 nid=0x1f1a in
> Object.wait() [0xde23f000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '5'" daemon prio=10 tid=0xdecfa400 nid=0x1f19 in
> Object.wait() [0xde290000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector.getDocumentVersions(WebcrawlerConnector.java:802)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:321)
>
> "Worker thread '4'" daemon prio=10 tid=0xdecf6c00 nid=0x1f18 in
> Object.wait() [0xde2e1000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:80)
>         - locked <0xe3fe96e8> (a
> org.apache.manifoldcf.core.jdbcpool.ConnectionPool)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:125)
>         at
> org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:96)
>         at
> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:722)
>         at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1394)
>         at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>         at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:186)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>         at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>         at
> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryHistoryManager.addRow(RepositoryHistoryManager.java:192)
>         at
> org.apache.manifoldcf.crawler.repository.RepositoryConnectionManager.recordHistory(RepositoryConnectionManager.java:650)
>         at
> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity.recordActivity(WorkerThread.java:1263)
>         at
> org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.doneFetch(ThrottledFetcher.java:1844)

Mime
View raw message