lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adrien Grand (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (LUCENE-5580) Always verify stored fields' checksum on merge
Date Tue, 08 Apr 2014 09:22:14 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-5580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Adrien Grand updated LUCENE-5580:
---------------------------------

    Attachment: LUCENE-5580.patch

Here is a patch that verifies checksums on stored fields when doing bulk merges.

> Always verify stored fields' checksum on merge
> ----------------------------------------------
>
>                 Key: LUCENE-5580
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5580
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Adrien Grand
>            Assignee: Adrien Grand
>             Fix For: 4.8
>
>         Attachments: LUCENE-5580.patch
>
>
> I have seen a couple of index corruptions over the last months, and most of them happened
on stored fields. The explanation might just be that since stored fields are usually most
of the index size, they are just more likely to be corrupted due to a hardware/operating-system
failure, but it might be as well a sneaky bug on our side.
> Lucene recently added checksums to index files, and you can enable integrity verification
upon merge, but this comes with a cost since you need to read all index files twice instead
of once. If you are merging a very large segment and your merges are I/O-bound, this might
be noticeable.
> I would like to implement integrity checks for stored fields on merges on the fly, so
that the stored fields files need to be read only once.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message