lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrzej Bialecki>
Subject Re: Merging "orphaned" segments into a composite index
Date Sat, 16 Sep 2006 13:08:52 GMT
Rob Staveley (Tom) wrote:
> It looks like my segments file only contains information for the .cfs
> segments. So this approach doesn't work. I was wondering if I could use
> IndexWriter.addIndexes(IndexReader[]) instead. Can I open an IndexReader
> without a corresponding segments file? In notice that
> always operates on directories, which I guess means that it uses the
> segments file. 
> Is a segments file something that can be easily bodged for a bunch of index
> files which aren't referenced by the segments file?
> This probably all seems like a foolish errand, but my two indexes are > 300G
> each and regenerating them is something I'd like to avoid.

1. do a backup first !!!

3. separate the segment data which is accounted for in the current 
"segments" file from all other data, and move that to a "healthy" index. 
Move the rest of the info to a "corrupt" index. Make sure that the 
"healthy" index is still healthy after this operation .. ;)

2. look at the source of and 
write(Directory). You can see how the new "segments" file is created 
based on the SegmentInfo information. So, the only challenge is to 
create a bunch of SegmentInfo instances corresponding to your segment 
names in the "corrupt" index, and write them out to a new "segments" 
file according to this format.

3. you can easily discover the number of documents in each segment. This 
is equivalent to the length (in bytes) of each <segment_name>.f<number> 
file, which are storing lengthNorm info per document and per field.

Once you have written the new "segments" file, try to open it in 
IndexReader and iterate over documents to see if the index is basically 
"healthy". You could also do an IndexWriter.optimize() to make sure that 
all parts of the index correctly fit together.

A note: if a cfs file is corrupted, you can try to explode it into 
individual files - you can use 
org.apache.lucene.index.CompoundFileReader, iterate over file names, 
open streams and save them to regular files; and then do the trick with 
re-generating the segment file.

Then you may consider this "corrupt" index to be restored. Now you can 
merge your "healthy" and "restored" indexes using IndexWriter.addIndexes().

BTW. I strongly recommend doing frequent backups at different stages. 
Also, I recommend using BeanShell - you will save a lot of time you 
would otherwise spend on editing and compilation of these steps.

Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration  Contact: info at sigram dot com

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message