lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Busch (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (LUCENE-554) Possible index corruption if crashing while replacing segments file
Date Sun, 25 Feb 2007 22:36:06 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael Busch resolved LUCENE-554.
----------------------------------

       Resolution: Fixed
    Fix Version/s: 2.1

With lockless commits there is no file renaming anymore in Lucene, as Mike already commented.

> Possible index corruption if crashing while replacing segments file
> -------------------------------------------------------------------
>
>                 Key: LUCENE-554
>                 URL: https://issues.apache.org/jira/browse/LUCENE-554
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 1.9
>            Reporter: Nadav Har'El
>            Priority: Minor
>             Fix For: 2.1
>
>
> Lucene's indexing is expected to be reasonably tolerant to computer crashes or the indexing
process being killed. By reasonably tolerant, I mean that it is ok to lose a few documents
(those currently buffered in memory), or have to repeat some work (e.g., a long merge that
was in progress) - but it is not ok for the entire index, or large chunks of it, to become
irreversebly corrupt.
> The fact that Lucene works by repeated merging of several small segments into a new larger
segments, solves most of the crash problems, because until the new segment is fully created,
the old segments are still there and fully functional. However, one possibility for corruption
remains in the segment replacement code:
> After a new segment is created, a new segments file is written as a new file "segments.new",
and then this file is renamed to "segments". The problem is that this renaming is done using
Directory.renameFile(), and FSDirectory.renameFile is *NOT* atomic: it first deletes the old
file, and then renames the new file. A crash between these stages (or perhaps during Java's
rename which also isn't guaranteed to be atomic) will potentially leave us without a working
"segments" file.
> I will post here a patch for this bug shortly.
> The patch will also include a change to Directory.renameFile()'s Javadoc. It currently
claims "This replacement should be atomic.", which is false in FSDirectory. Instead it should
make a weaker claim, for example
>    "This replacement does not have to be atomic, but must at least obey a weaker guarantee:
at any time during the replacement, either the "from" file is still available, or the "to"
file is available with either the new or old content."
> (or, we can just drop the guaranteee altogether, like Java's File.renameTo() provides
no atomic-ness guarantees).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message