lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adrien Grand (JIRA)" <j...@apache.org>
Subject [jira] [Created] (LUCENE-5580) Always verify stored fields' checksum on merge
Date Tue, 08 Apr 2014 09:18:16 GMT
Adrien Grand created LUCENE-5580:
------------------------------------

             Summary: Always verify stored fields' checksum on merge
                 Key: LUCENE-5580
                 URL: https://issues.apache.org/jira/browse/LUCENE-5580
             Project: Lucene - Core
          Issue Type: Improvement
            Reporter: Adrien Grand
            Assignee: Adrien Grand
             Fix For: 4.8


I have seen a couple of index corruptions over the last months, and most of them happened
on stored fields. The explanation might just be that since stored fields are usually most
of the index size, they are just more likely to be corrupted due to a hardware/operating-system
failure, but it might be as well a sneaky bug on our side.

Lucene recently added checksums to index files, and you can enable integrity verification
upon merge, but this comes with a cost since you need to read all index files twice instead
of once. If you are merging a very large segment and your merges are I/O-bound, this might
be noticeable.

I would like to implement integrity checks for stored fields on merges on the fly, so that
the stored fields files need to be read only once.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message