lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rob Staveley (Tom)" <rstave...@seseit.com>
Subject RE: Merging "orphaned" segments into a composite index
Date Sat, 16 Sep 2006 19:07:07 GMT
Thanks for the advice, Andrzej, including using BeanShell for this.
SegmentInfos and CompoundFileReader here I come.

-----Original Message-----
From: Andrzej Bialecki [mailto:ab@getopt.org] 
Sent: 16 September 2006 14:09
To: java-user@lucene.apache.org
Subject: Re: Merging "orphaned" segments into a composite index

Rob Staveley (Tom) wrote:
> It looks like my segments file only contains information for the .cfs 
> segments. So this approach doesn't work. I was wondering if I could 
> use
> IndexWriter.addIndexes(IndexReader[]) instead. Can I open an 
> IndexReader without a corresponding segments file? In notice that 
> IndexReader.open(...) always operates on directories, which I guess 
> means that it uses the segments file.
>
> Is a segments file something that can be easily bodged for a bunch of 
> index files which aren't referenced by the segments file?
>
> This probably all seems like a foolish errand, but my two indexes are 
> > 300G each and regenerating them is something I'd like to avoid.
>   


1. do a backup first !!!

3. separate the segment data which is accounted for in the current
"segments" file from all other data, and move that to a "healthy" index. 
Move the rest of the info to a "corrupt" index. Make sure that the "healthy"
index is still healthy after this operation .. ;)

2. look at the source of
org.apache.lucene.index.SegmentInfos.read(Directory) and write(Directory).
You can see how the new "segments" file is created based on the SegmentInfo
information. So, the only challenge is to create a bunch of SegmentInfo
instances corresponding to your segment names in the "corrupt" index, and
write them out to a new "segments" 
file according to this format.

3. you can easily discover the number of documents in each segment. This is
equivalent to the length (in bytes) of each <segment_name>.f<number> file,
which are storing lengthNorm info per document and per field.

Once you have written the new "segments" file, try to open it in IndexReader
and iterate over documents to see if the index is basically "healthy". You
could also do an IndexWriter.optimize() to make sure that all parts of the
index correctly fit together.

A note: if a cfs file is corrupted, you can try to explode it into
individual files - you can use org.apache.lucene.index.CompoundFileReader,
iterate over file names, open streams and save them to regular files; and
then do the trick with re-generating the segment file.

Then you may consider this "corrupt" index to be restored. Now you can merge
your "healthy" and "restored" indexes using IndexWriter.addIndexes().

BTW. I strongly recommend doing frequent backups at different stages. 
Also, I recommend using BeanShell - you will save a lot of time you would
otherwise spend on editing and compilation of these steps.

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web ___|||__||  \|
||  |  Embedded Unix, System Integration http://www.sigram.com  Contact:
info at sigram dot com



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Mime
View raw message