manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aeham Abushwashi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CONNECTORS-1395) Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
Date Wed, 08 Mar 2017 17:07:37 GMT

    [ https://issues.apache.org/jira/browse/CONNECTORS-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901563#comment-15901563
] 

Aeham Abushwashi commented on CONNECTORS-1395:
----------------------------------------------

Not sure if you've looked at this but some of ZK's default settings are fairly conservative
and don't play well with connection-hungry applications. I'd suggest setting/increasing ZK's
maxClientCnxns (in zoo.cfg). I tend to go with a value of 500 but it really is down to how
beefy the node is.
It's also worth increasing the aggressiveness of ZK's autopurge. Again in zoo.cfg, set 'autopurge.snapRetainCount'
and 'autopurge.purgeInterval' to, say, 10 and 2, respectively. This helps reduce the memory
pressure on ZK and reduces the likelihood of forcible termination of connections under load.

> Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
> ------------------------------------------------------------------------------------
>
>                 Key: CONNECTORS-1395
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1395
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Framework core
>    Affects Versions: ManifoldCF 2.6
>            Reporter: Karl Wright
>            Assignee: Karl Wright
>             Fix For: ManifoldCF 2.7
>
>         Attachments: ZK Console Output.txt
>
>
> User saw this in the log, after which the system hung:
> {code}
> ERROR 2017-03-08 00:25:30,433 (Worker thread '14') - Exception tossed: Unexpected jobqueue
status - record id 1488898668325, expecting active status, saw 4
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected jobqueue status
- record id 1488898668325, expecting active status, saw 4
>                 at org.apache.manifoldcf.crawler.jobs.JobQueue.updateCompletedRecord(JobQueue.java:1019)
>                 at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentCompletedMultiple(JobManager.java:3271)
>                 at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:710)
> WARN 2017-03-08 00:25:30,449 (Worker thread '23') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:30,449 (Worker thread '24') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:30,464 (Worker thread '9') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:30,464 (Worker thread '0') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:31,900 (Worker thread '11') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:31,900 (Worker thread '29') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:32,867 (Worker thread '10') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:32,867 (Worker thread '2') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:33,335 (Worker thread '8') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:36,642 (Worker thread '20') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:37,422 (Worker thread '21') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,280 (Worker thread '22') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,280 (Worker thread '3') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,280 (Worker thread '5') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:38,826 (Worker thread '28') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:39,045 (Worker thread '13') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:45,425 (Worker thread '4') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:45,425 (Worker thread '15') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:45,425 (Worker thread '17') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:46,392 (Worker thread '25') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:25:46,392 (Worker thread '27') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:11,043 (Worker thread '1') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:35,817 (Worker thread '19') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:35,817 (Worker thread '26') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:36,753 (Worker thread '7') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:39,248 (Worker thread '6') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:39,248 (Worker thread '18') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> WARN 2017-03-08 00:26:43,129 (Worker thread '16') - Service interruption reported for
job 1488898090224 connection 'web': Job no longer active
> FATAL 2017-03-08 00:32:24,819 (Idle cleanup thread) - Error tossed: Can't release lock
we don't hold
> java.lang.IllegalStateException: Can't release lock we don't hold”
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message