lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SolrUser1543 <osta...@gmail.com>
Subject Performing DIH on predefined list of IDS
Date Fri, 20 Feb 2015 10:32:24 GMT
Relatively frequently (about a once a month) we need to reindex the data, by
using DIH and copying the data from one index to another.
Because of the fact that we have a large index, it could take from 12 to 24
hours to complete. At the same time the old index is being queried by users. 
Sometimes DIH could be interrupted at the middle, because of some unexpected
exception caused by OutOfMemory or something else (many times it failed when
more than 90 % was completed). 
More than this, almost every time, some items are missing at new the  index.
It is very complicated to find them. 
At this stage I can't be sure about what documents exactly were missed and I
have to do it again and waiting for many hours. At the same time the old
index constantly receives new items. 

I want to suggest the following way to solve the problem: 
•	Get list of all item ids ( call LUCINE API , like CLUE does for example ) 
•	Start DIH    , which will iterate over those ids and each time make a
query for n items.
1.	Of course original DIH class should be changed to support it. 
•	This will give the following advantages : 
1.	I will know exactly what items were failed.
2.	I can restart the process from any point and in case of DIH failure
restart it from the point of failure.


so the main difference will be that now DIH running on *:* query and I
suggest to run it list of IDS 

for example if I have 1000 docs and want that this new DIH will take each
time 100 docs , so it will do 10 queries , each one will have 100 IDS . (
like id:(1 2 3 ... 100) then id:(101 102 ... 200) etc... ) 

The question is what do you think about it? Or all of this could be done
another way and I am trying to reinvent the wheel?




--
View this message in context: http://lucene.472066.n3.nabble.com/Performing-DIH-on-predefined-list-of-IDS-tp4187589.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message