manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Wright (JIRA)" <>
Subject [jira] [Commented] (CONNECTORS-1562) Documents unreachable due to hopcount are not considered unreachable on cleanup pass
Date Thu, 13 Dec 2018 09:02:00 GMT


Karl Wright commented on CONNECTORS-1562:

I went through the invalidation logic, which was last changed in 2012 as part of ticket CONNECTORS-501.
 The fix for that ticket did not seem related to the current problem at least.

I improved logging methods in this area, and documentation, but did not yet find any logical
errors.  The next step is therefore to repeat the experiment with database debugging also
enabled so I can see the queries too.  I can probably start by doing this only during the
reduce phase, but if those queries look good, I'd have to also do the initial phase.

> Documents unreachable due to hopcount are not considered unreachable on cleanup pass
> ------------------------------------------------------------------------------------
>                 Key: CONNECTORS-1562
>                 URL:
>             Project: ManifoldCF
>          Issue Type: Bug
>          Components: Elastic Search connector, Web connector
>    Affects Versions: ManifoldCF 2.11
>         Environment: Manifoldcf 2.11
> Elasticsearch 6.3.2
> Web inputconnector
> elastic outputconnecotr
> Job crawls website input and outputs content to elastic
>            Reporter: Tim Steenbeke
>            Assignee: Karl Wright
>            Priority: Critical
>              Labels: starter
>         Attachments: manifoldcf.log.cleanup, manifoldcf.log.init, manifoldcf.log.reduced
>   Original Estimate: 4h
>  Remaining Estimate: 4h
> My documents aren't removed from ElasticSearch index after rerunning the changed seeds
> I update my job to change the seedmap and rerun it or use the schedualer to keep it runneng
even after updating it.
> After the rerun the unreachable documents don't get deleted.
> It only adds doucments when they can be reached.

This message was sent by Atlassian JIRA

View raw message