manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shinichiro Abe <>
Subject Resume mechanism
Date Thu, 16 Jun 2011 23:09:50 GMT
Please let me know about resume mechanism.

For example, when job is executing, the following things happen. 
MCF services stop, Solr shutdown, repository servers shutdown.
The job can not connect eace connectors by shutdown, it stops to ingest documents.
But when the above things are recovered, the job starts to resume ingesting, it can keep crawling
What manages it? Does jobqueue manage this resume mechanism? 

If so, are there cases that job can not keep crawling consistency?
e.g. the following cases.
 a)Postgresql stops before inserting all into jobqueue, jobqueue data is short and inconsistent.
 b)Though it needs to crawl a lot of documents, MCF stops before inserting all into jobqueue.
As a result, jobqueue data is short and inconsistent.
 c)Any other cases.
I want to know the possibility that data is inconsistent by halfway interrupting when crawling.

Also I want to read Part4 MCF architecture on ManifoldCFinAction.
Shinichiro Abe
View raw message