manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Wright (JIRA)" <>
Subject [jira] [Commented] (CONNECTORS-1100) Improve ManifoldCF scalability by adopting dynamic reprioritization
Date Tue, 11 Nov 2014 17:46:33 GMT


Karl Wright commented on CONNECTORS-1100:

I've gotten things working fairly well and committed them to the branch.

Note that the branch is based off of MCF 2.0 code, but other than that, this is probably ready
to try out.

> Improve ManifoldCF scalability by adopting dynamic reprioritization
> -------------------------------------------------------------------
>                 Key: CONNECTORS-1100
>                 URL:
>             Project: ManifoldCF
>          Issue Type: Improvement
>          Components: Framework crawler agent
>    Affects Versions: ManifoldCF 1.8, ManifoldCF 2.0
>            Reporter: Karl Wright
>            Assignee: Karl Wright
>             Fix For: ManifoldCF 1.8, ManifoldCF 2.0
> The single greatest impediment to MCF scalability at this point is the time that is required
to reprioritize documents.  This can be approved by adopting dynamic reprioritization.
> This process involves the following changes to the reprioritization process:
> - At the time of reprioritization, wipe the docpriority field to its null value. (Q:
How long does this take?)
> - A background thread picks up documents on the queue with null priorities, and prioritizes
them based on current conditions, doing 1000 or 10000 at a time
> - We need a new field to properly manage this; a simple boolean field which I'll call
"needpriority", and an index based on it too.
> - ReprioritizationTracker currently has a notion of "reprioritization cycles".  Cycles
are interesting insofar as the minimum depth gets reset at the beginning of them.  We will
need to redefine what a cycle is, if we wish to maintain the current logic.  Specifically,
a "cycle" looks like this:
> (1) Remove all current document priorities (set to nullDocumentPriority), while also
setting "needpriority" field to "true"
> (2) Reset priority calculation values (e.g. minimum depth)
> (3) Done with *first part* of cycle
> (4) Reprioritize via thread over time
> (5) When no more to do, done with *second part* of cycle
> - But, the whole "priorityset" logic is not needed anymore, if there's a persistent thread
and a "needpriority" index.  So we only will need the first part of the cycle, above, and
no priorityset field is needed anymore at all.  We just need a global write lock to coordinate
the priority setting threads cross-cluster.  These also have to synchronize, though, with
any "reprioritization" cycles.    
> - We can simply change the contract of IReprioritizationTracker to include entrance and
exit for the reprioritization threads, and manage any locking  internally.  This would be
PROVIDED the threads exercised good cleanup hygiene.

This message was sent by Atlassian JIRA

View raw message