lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pulkit Singhal <pulkitsing...@gmail.com>
Subject Re: Checksum and transactional safety for lucene indexes
Date Fri, 24 Sep 2010 14:57:27 GMT
In order to determine the integrity of an index file, I found that the
easiest way was to use IndexReader.open(directory) and if there were
any problems with the data then catch the exceptions and make a new
one.

I also see that the API offers IndexReader.indexExists() ... would
that be a better alternative?
Or would I miss out on being able to check if the index is corrupt if
I limit myself to that invocaction only?

Thanks!

- Pulkit

On Tue, Sep 21, 2010 at 12:53 AM, Lance Norskog <goksron@gmail.com> wrote:
> If an index file is not completely written to disk, it never become
> available. Lucene has a file describing the current active index segments.
> It writes all new files to the disk, and changes the description file
> (segments.gen) only after that.
>
> If the index files are corrupted, all bets are off. Usually the data
> structures are damaged and Lucene throws CorruptIndexExceptions, NPE or
> array out-of-bounds exceptions. There is no checksumming of the index files.
>
> Lance
>
> Pulkit Singhal wrote:
>>
>> Hello Everyone,
>>
>> What happens if:
>> a) lucene index gets written half-way to the disk and then something goes
>> wrong?
>> b) the index gets corrupted on the file system?
>>
>> When we open that directory location again using FSDirectory
>> implementations:
>> a) Is there any provision for the code to clean out the previous file
>> and start a new index file because the older one was corrupted and
>> didn't match the checksum?
>> b) Or can we check that the # of documents that can be found in the
>> underlying index are now ZERO because they can't be parsed properly?
>> How can we do this?
>>
>> - Pulkit
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message