lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chuck Williams <ch...@manawiz.com>
Subject After kill -9 index was corrupt
Date Mon, 11 Sep 2006 00:24:04 GMT
Hi All,

An application of ours under development had a memory link that caused
it to slow interminably.  On linux, the application did not response to
kill -15 in a reasonable time, so kill -9 was used to forcibly terminate
it.  After this the segments file contained a reference to a segment
whose index files were not present.  I.e., the index was corrupt and
Lucene could not open it.

A thread dump at the time of the kill -9 shows that Lucene was merging
segments inside IndexWriter.close().  Since segment merging only commits
(updates the segments file) after the newly merged segment(s) are
complete, I expect this is not the actual problem.

Could a kill -9 prevent data from reaching disk for files that were
previously closed?  If so, then Lucene's index can become corrupt after
kill -9.  In this case, it is possible that a prior merge created new
segment index files, updated the segments file, closed everything, the
segments file made it to disk, but the index data files and/or their
directory entries did not.

If this is the case, it seems to me that flush() and
FileDescriptor.sync() are required on each index file prior to close()
to guarantee no corruption.  Additionally a FileDescriptor.sync() is
also probably required on the index directory to ensure the directory
entries have been persisted.

A power failure or other operating system crash could cause this, not
just kill -9.

Does this seem like a possible explanation and fix for what happened? 
Could the same kind of problem happen on Windows?

If this is the issue, then how would people feel about having Lucene do
sync()'s a) always? or b) as an index configuration option?

I need to fix whatever happened and so would submit a patch to resolve it.

Thanks for advice and suggestions,

Chuck


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message