lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-1044) Behavior on hard power shutdown
Date Tue, 05 Feb 2008 11:00:38 GMT


Michael McCandless commented on LUCENE-1044:

On thinking through the above costs of committing, I now think we
should deprecate autoCommit=true entirely, making autocommit=false the
only choice in 3.0.

With that change, when you use an IndexWriter, its changes are never
visible to a reader until you call commit() or close().  I think this
is how KinoSearch and Ferret work, for example.

Here are some reasons:

  * Commit has now become a costly event, because sync() is costly,
    and is forcing us to use this "syncPause" logic (hack) to game the
    OS, which really is ugly, dependent on OS/IO particulars, etc.

  * Since we make no guarantee on when a commit specifically happens,
    and this fix in particular will reduce its frequency from "every
    flush" to "every merge", autoCommit=true really is not that useful
    for applications (ie, they will have to call commit() on their
    anyway if they need to rely on its frequency).

  * It's always possible to build an autocommit layer above
    IndexWriter by calling commit on your own schedule, to tradeoff
    performance for commit frequency (but not vice/versa).

  * Not autocommitting by default opens up some good future
    optimizations on merging since we don't have to flush real
    segments to disk until commit.  One simple example is we could
    skip building CFS files as we flush, and only merge & build CFS on

What do people think?

If we do this, I would right now deprecate all ctors that take
autoCommit and add comment explaining that in 3.0 autoCommit is wired
to "false".  I would leave the "syncPause" logic in there for now,
because it's such a sizable performance gain on windows, but deprecate
it, stating that with it will be removed when we switch to
autoCommit=false in 3.0.

> Behavior on hard power shutdown
> -------------------------------
>                 Key: LUCENE-1044
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>         Environment: Windows Server 2003, Standard Edition, Sun Hotspot Java 1.5
>            Reporter: venkat rangan
>            Assignee: Michael McCandless
>             Fix For: 2.4
>         Attachments:, LUCENE-1044.patch, LUCENE-1044.take2.patch,
LUCENE-1044.take3.patch, LUCENE-1044.take4.patch, LUCENE-1044.take5.patch, LUCENE-1044.take6.patch
> When indexing a large number of documents, upon a hard power failure  (e.g. pull the
power cord), the index seems to get corrupted. We start a Java application as an Windows Service,
and feed it documents. In some cases (after an index size of 1.7GB, with 30-40 index segment
.cfs files) , the following is observed.
> The 'segments' file contains only zeros. Its size is 265 bytes - all bytes are zeros.
> The 'deleted' file also contains only zeros. Its size is 85 bytes - all bytes are zeros.
> Before corruption, the segments file and deleted file appear to be correct. After this
corruption, the index is corrupted and lost.
> This is a problem observed in Lucene 1.4.3. We are not able to upgrade our customer deployments
to 1.9 or later version, but would be happy to back-port a patch, if the patch is small enough
and if this problem is already solved.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message