manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Steenbeke (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CONNECTORS-1562) Documents unreachable due to hopcount are not considered unreachable on cleanup pass
Date Tue, 18 Dec 2018 07:49:00 GMT

    [ https://issues.apache.org/jira/browse/CONNECTORS-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16723782#comment-16723782
] 

Tim Steenbeke edited comment on CONNECTORS-1562 at 12/18/18 7:48 AM:
---------------------------------------------------------------------

[~kwright@metacarta.com]
If we update to manifold 2.12 can we than use the seedmap as originaly intended by us ?

so we create a job with X seeds, ES output, web input and HopCount 0 for links and redirect:
 # Put X seeds in seedmap
 # run job
 # X documents get pushed to ES
 # update job to have X minus 20 seeds
 wait till scheduled time

 
 # run job
 # 20 documents get deleted from ES
 # X minus 20 documents get updated
 # wait till scheduled time
 # ...

Will it work like this ?


was (Author: steenti):
If we update to manifold 2.12 can we than use the seedmap as originaly intended by us ?

so we create a job with X seeds, ES output, web input and HopCount 0 for links and redirect:

 # Put X seeds in seedmap
 # run job
 # X documents get pushed to ES
 # update job to have X minus 20 seeds
wait till scheduled time

 
 # run job
 # 20 documents get deleted from ES
 # X minus 20 documents get updated
 # wait till scheduled time
 # ...

Will it work like this ?

> Documents unreachable due to hopcount are not considered unreachable on cleanup pass
> ------------------------------------------------------------------------------------
>
>                 Key: CONNECTORS-1562
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1562
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Elastic Search connector, Web connector
>    Affects Versions: ManifoldCF 2.11
>         Environment: Manifoldcf 2.11
> Elasticsearch 6.3.2
> Web inputconnector
> elastic outputconnecotr
> Job crawls website input and outputs content to elastic
>            Reporter: Tim Steenbeke
>            Assignee: Karl Wright
>            Priority: Critical
>              Labels: starter
>             Fix For: ManifoldCF 2.12
>
>         Attachments: manifoldcf.log.cleanup, manifoldcf.log.init, manifoldcf.log.reduced
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> My documents aren't removed from ElasticSearch index after rerunning the changed seeds
> I update my job to change the seedmap and rerun it or use the schedualer to keep it runneng
even after updating it.
> After the rerun the unreachable documents don't get deleted.
> It only adds doucments when they can be reached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message