lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Zhu <t...@mintel.com>
Subject Do delta import will loss some documents, when the documents added in the duration of delta import.
Date Tue, 06 Sep 2016 01:23:30 GMT
hello guys,
       I met a problem when i using the solrcloud mode. When the solr
instance run delta-import, it may take
some time to be finished( my data source is mysql database). So during this
time, the new added documents
will loss, the deltaQuery, i use SUBDATE(${dih.last_index_time}, INTERVAL 2
MINUTE),
let it run the delta-import 2 mins earlier than the last_index_time, if the
delta-import's duration is 5 mins, it will loss the records at the first 3
mins.
      Our servers doesn't use solr cloud mode before, we deal with this
issue is trying to rewrite dataimport.properties file,
query the max(sys_time_stamp), which will help to record the max time
stamp, and let the solr can run delta import standing
by the time found in the file, of course, it will never miss documents.
      But now, we use solrcloud, the dataimport.properties is on the
zookeeper, and we may have multiple collections for the
same core.how can i update the dataimport.properties file now in collection
now? Do you have any solution to help record
the max(sys_time_stamp) in dataimport.properties, rather than using the
time of delta-import start to run? Appreciate for any suggestions.

Thank you !!!

Cheers

-- 




Mintel Group Ltd | London | Chicago | New York | Sydney | Shanghai | Tokyo | Mumbai | Singapore
| Kuala Lumpur | Sao Paulo | Belfast

Contact details for our offices can be found at http://www.mintel.com/office-locations.

This email and any attachments may include content that is confidential, privileged, or otherwise
protected
under applicable law. Unauthorised disclosure, copying, distribution, or use of the contents
is prohibited 
and may be unlawful. If you have received this email in error, including without appropriate
authorisation, 
then please reply to the sender about the error and delete this email and any attachments.


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message