Mailing-List: contact dev-help@manifoldcf.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@manifoldcf.apache.org
Date: Thu, 11 Dec 2014 12:08:14 +0000 (UTC)
From: "Aeham Abushwashi (JIRA)" <jira@apache.org>
To: dev@manifoldcf.apache.org
Message-ID: <JIRA.12760461.1418138437000.325.1418299694612@Atlassian.JIRA>
In-Reply-To: <JIRA.12760461.1418138437000@Atlassian.JIRA>
References: <JIRA.12760461.1418138437000@Atlassian.JIRA>
 <JIRA.12760461.1418138437784@arcas>
Subject: [jira] [Commented] (CONNECTORS-1118) Documents processed by the
 shared drive connector incur an unnecessary synchronisation hit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CONNECTORS-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14242450#comment-14242450 ] 

Aeham Abushwashi commented on CONNECTORS-1118:
----------------------------------------------

The profiler trace looks much better now. Thank you!

> Documents processed by the shared drive connector incur an unnecessary synchronisation hit
> ------------------------------------------------------------------------------------------
>
>                 Key: CONNECTORS-1118
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1118
>             Project: ManifoldCF
>          Issue Type: Improvement
>          Components: Framework core
>    Affects Versions: ManifoldCF 1.7.2
>            Reporter: Aeham Abushwashi
>            Assignee: Karl Wright
>             Fix For: ManifoldCF 1.8, ManifoldCF 2.0
>
>
> Each document processed by the shared drive connector is passed through SharedDriveConnector#checkInclude to verify whether the document is eligible for ingestion. The calls made here to WorkerThread$ProcessActivity#checkMimeTypeIndexable and WorkerThread$ProcessActivity#checkLengthIndexable are unnecessarily costly as they each create a fresh instance of IncrementalIngester$PipelineConnections on every call. The constructor of IncrementalIngester$PipelineConnections can be very expensive due to the loading of output connection objects, which in turn requires some locking (via ZK - in a distrubuted environment).
> The other area of inefficiency is in WorkerThread$ProcessActivity#processDocumentReferences. This method creates new instances of PriorityCalculator using the less-efficient 3-arg constructor. This can be addressed using the same pattern implemented for CONNECTORS-1094
> To highlight the impact of the above calls, I profiled an active worker thread for 40 minutes. During that window, it spent ~23 minutes in SharedDriveConnector#checkInclude and its callees + 9 minutes creating instances of PriorityCalculator.
> I've seen the above issues when using the shared drive connector but I think other connectors too could be impacted - depending on how they're implemented.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)