manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Wright (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CONNECTORS-1395) Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
Date Fri, 10 Mar 2017 18:49:04 GMT

    [ https://issues.apache.org/jira/browse/CONNECTORS-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905542#comment-15905542
] 

Karl Wright edited comment on CONNECTORS-1395 at 3/10/17 6:48 PM:
------------------------------------------------------------------

[~guystanden]: The zookeeper log is only a few lines long and cannot be complete.

For what it is worth, the errors in it are the kinds of errors that you should expect to see
using ManifoldCF in normal operation under Zookeeper.  Those errors are not harmful in any
way.

The forensics for the "unexpected jobqueue status" error indicate that yes indeed there is
a transactional integrity issue.  This could be caused either by Zookeeper dropping connections
or by a Postgresql bug.  Since we cannot tell which it was from this, I strongly suggest increasing
maxClientCnxns from 500 to 1000 to see whether you continue to get this problem, if it takes
longer, etc.  There are also ManifoldCF settings that serve to limit the number of potential
connections, but let's explore the behavior first.

http://stackoverflow.com/questions/9213537/error-hbase-zookeeper-too-many-connections

Logging.ini allows you to directly configure the logging for ManifoldCF using the log4j conventions
you prefer.  I can really say no more useful about this, except that there is online documentation
which should help you.




was (Author: kwright@metacarta.com):
[~guystanden]: The zookeeper log is only a few lines long and cannot be complete.

For what it is worth, the errors in it are the kinds of errors that you should expect to see
using ManifoldCF under Zookeeper.

The forensics for the "unexpected jobqueue status" error indicate that yes indeed there is
a transactional integrity issue.  This could be caused either by Zookeeper dropping connections
or by a Postgresql bug.  Since we cannot tell which it was from this, I strongly suggest increasing
maxClientCnxns from 500 to 1000 to see whether you continue to get this problem, if it takes
longer, etc.  There are also ManifoldCF settings that serve to limit the number of potential
connections, but let's explore the behavior first.

http://stackoverflow.com/questions/9213537/error-hbase-zookeeper-too-many-connections

Logging.ini allows you to directly configure the logging for ManifoldCF using the log4j conventions
you prefer.  I can really say no more useful about this, except that there is online documentation
which should help you.



> Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
> ------------------------------------------------------------------------------------
>
>                 Key: CONNECTORS-1395
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1395
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Framework core
>    Affects Versions: ManifoldCF 2.6
>            Reporter: Karl Wright
>            Assignee: Karl Wright
>             Fix For: ManifoldCF 2.7
>
>         Attachments: MCF Log 2.txt, ZK Console 2.txt, ZK Console Output.txt
>
>
> User saw this in the log, after which the system hung:
> {code}
> ERROR 2017-03-08 00:25:30,433 (Worker thread '14') - Exception tossed: Unexpected jobqueue
status - record id 1488898668325, expecting active status, saw 4
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected jobqueue status
- record id 1488898668325, expecting active status, saw 4
>                 at org.apache.manifoldcf.crawler.jobs.JobQueue.updateCompletedRecord(JobQueue.java:1019)
>                 at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentCompletedMultiple(JobManager.java:3271)
>                 at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:710)
> WARN 2017-03-08 00:25:30,449 (Worker thread '23') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:30,449 (Worker thread '24') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:30,464 (Worker thread '9') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:30,464 (Worker thread '0') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:31,900 (Worker thread '11') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:31,900 (Worker thread '29') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:32,867 (Worker thread '10') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:32,867 (Worker thread '2') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:33,335 (Worker thread '8') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:36,642 (Worker thread '20') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:37,422 (Worker thread '21') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,280 (Worker thread '22') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,280 (Worker thread '3') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,280 (Worker thread '5') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,826 (Worker thread '28') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:39,045 (Worker thread '13') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:45,425 (Worker thread '4') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:45,425 (Worker thread '15') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:45,425 (Worker thread '17') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:46,392 (Worker thread '25') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:46,392 (Worker thread '27') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:11,043 (Worker thread '1') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:35,817 (Worker thread '19') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:35,817 (Worker thread '26') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:36,753 (Worker thread '7') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:39,248 (Worker thread '6') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:39,248 (Worker thread '18') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:43,129 (Worker thread '16') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> FATAL 2017-03-08 00:32:24,819 (Idle cleanup thread) - Error tossed: Can't release lock
we don't hold
> java.lang.IllegalStateException: Can't release lock we don't hold”
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message