Return-Path: X-Original-To: apmail-manifoldcf-dev-archive@www.apache.org Delivered-To: apmail-manifoldcf-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4484817B2C for ; Tue, 4 Nov 2014 08:39:34 +0000 (UTC) Received: (qmail 48176 invoked by uid 500); 4 Nov 2014 08:39:34 -0000 Delivered-To: apmail-manifoldcf-dev-archive@manifoldcf.apache.org Received: (qmail 48118 invoked by uid 500); 4 Nov 2014 08:39:34 -0000 Mailing-List: contact dev-help@manifoldcf.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@manifoldcf.apache.org Delivered-To: mailing list dev@manifoldcf.apache.org Received: (qmail 48106 invoked by uid 99); 4 Nov 2014 08:39:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2014 08:39:34 +0000 Date: Tue, 4 Nov 2014 08:39:34 +0000 (UTC) From: "Karl Wright (JIRA)" To: dev@manifoldcf.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CONNECTORS-1093) ManifoldCF document reprioritization bottleneck MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CONNECTORS-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright updated CONNECTORS-1093: ------------------------------------ Priority: Blocker (was: Major) > ManifoldCF document reprioritization bottleneck > ----------------------------------------------- > > Key: CONNECTORS-1093 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1093 > Project: ManifoldCF > Issue Type: Bug > Components: Framework agents process > Affects Versions: ManifoldCF 1.7.2, ManifoldCF 1.8, ManifoldCF 2.0 > Reporter: Karl Wright > Assignee: Karl Wright > Priority: Blocker > Fix For: ManifoldCF 1.7.2, ManifoldCF 1.8, ManifoldCF 2.0 > > > Starting a job with 200K+ documents now takes many minutes. The reason seems to be document reprioritization, which has a significant bottleneck. A thread dump shows: > {code} > at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.finishUp(Database.java:694) > at org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:728) > at org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:762) > at org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1435) > at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146) > at org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:191) > at org.apache.manifoldcf.core.database.DBInterfaceHSQLDB.performModification(DBInterfaceHSQLDB.java:750) > at org.apache.manifoldcf.core.database.DBInterfaceHSQLDB.performUpdate(DBInterfaceHSQLDB.java:296) > at org.apache.manifoldcf.core.database.BaseTable.performUpdate(BaseTable.java:80) > at org.apache.manifoldcf.crawler.bins.BinManager.getIncrementBinValues(BinManager.java:158) > at org.apache.manifoldcf.crawler.reprioritizationtracker.ReprioritizationTracker.getIncrementBinValue(ReprioritizationTracker.java:328) > at org.apache.manifoldcf.crawler.system.PriorityCalculator.getDocumentPriority(PriorityCalculator.java:145) > at org.apache.manifoldcf.crawler.jobs.JobQueue.writeDocPriority(JobQueue.java:874) > at org.apache.manifoldcf.crawler.jobs.JobManager.writeDocumentPriorities(JobManager.java:2142) > at org.apache.manifoldcf.crawler.system.ManifoldCF.writeDocumentPriorities(ManifoldCF.java:1121) > at org.apache.manifoldcf.crawler.system.ManifoldCF.resetAllDocumentPriorities(ManifoldCF.java:1054) > at org.apache.manifoldcf.crawler.system.StartupThread.run(StartupThread.java:141) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)