lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (Commented) (JIRA)" <>
Subject [jira] [Commented] (LUCENE-3627) CorruptIndexException on indexing after a failure occurs after segments file creation but before any bytes are written
Date Thu, 08 Dec 2011 19:26:40 GMT


Michael McCandless commented on LUCENE-3627:

bq. Would you ever have the case there only one "segements_N" file is corrupt 'i.e. 0 size'?

No, unless something terrible is going on, such as your IO system
disregards fsync, you have hardware problems, or something external is
mucking with the index files.

When Lucene commits it writes the next segments_N, fsyncs it (and all
index file it references) and then removes old commits; so the old
commits are not removed until the new one is on durable storage.

I think the conditions for this bug to occur are very rare, ie, lucene
tried to commit, hit a transient IO problem in creating the file (so
that a 0 byte file is created), your app caught this and called
IW.close, this time the transient IO problem is gone and the close
succeeds in writing another segments_N file.  At that point you'd hit
this bug when you next tried to open an IndexWriter on the index.

The more common failure would be if the segments_N fails to write (0
byte file) and then when you close the IW it also fails, ie, because
something suddenly is wrong w/ your IO system.  In this case your
"latest" segments_N is unreadable, and Lucene handles that fine (falls
back to segments_(N-1), which should be ok because it was fsync'd).

> CorruptIndexException on indexing after a failure occurs after segments file creation
but before any bytes are written
> ----------------------------------------------------------------------------------------------------------------------
>                 Key: LUCENE-3627
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 3.5
>         Environment: lucene-3.5.0, src download from GA release
> Mac OS X 10.6.5, running tests in Eclipse Build id: 20100218-1602, 
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07-334-10M3326)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02-334, mixed mode)
>            Reporter: Ken McCracken
>            Assignee: Michael McCandless
>            Priority: Critical
>         Attachments: LUCENE-3627.patch, LUCENE-3627_initial_proposal.txt,
>   Original Estimate: 48h
>  Remaining Estimate: 48h
> FSDirectory.createOutput(..) uses a RandomAccessFile to do its work.  On my system the
default creates an NIOFSDirectory.  If createOutput is called on a segments_*
file and a crash occurs between RandomAccessFile creation (file system shows a segments_*
file exists but has zero bytes) but before any bytes are written to the file, subsequent IndexWriters
cannot proceed.  The difficulty is that it does not know how to clear the empty segments_*
file.  None of the file deletions will happen on such a segment file because the opening bytes
cannot not be read to determine format and version.
> An initial proposed patch file is attached below.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message