Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <17691567.1196264383402.JavaMail.jira@brutus>
Date: Wed, 28 Nov 2007 07:39:43 -0800 (PST)
From: "Michael McCandless (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Commented: (LUCENE-1044) Behavior on hard power shutdown
In-Reply-To: <9290721.1194030470805.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/LUCENE-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546277 ] 

Michael McCandless commented on LUCENE-1044:
--------------------------------------------

{quote}
I'm confused. The semantics of commit should be that all changes prior are made permanent, and no subsequent changes are permanent until the next commit. So syncs, if any, should map 1:1 to commits, no? Folks can make indexing faster by committing/syncing less often.
{quote}

But must every "automatic buffer flush" by IndexWriter really be a
"permanent commit"?  I do agree that when you close an IndexWriter, we
should should do a "permanent commit" (and block until it's done).

Even if we use that policy, the BG sync thread can still fall behind
such that the last few/many flushes are still in-process of being made
permanent (eg I see this happening while a merge is running).  In fact
I'll have to block further flushes if syncing falls "too far" behind,
by some metric.  So, we already won't have any "guarantee" on when a
given flush actually becomes permanent even if we adopt this policy.

I think "merge finished" should be made a "permanent commit" because
otherwise we are tying up potentially alot of disk space,
temporarily.  But for a flush there's only a tiny amount of space (the
old segments_N files) being tied up.

Maybe we could make some flushes permanent but not all, depending on
how far behind the sync thread is.  EG if you do a flush, but, the
sync thread is still trying to make the last flush permanent, don't
force the new flush to be permanent?

In general, I think the longer we can wait after flushing before
forcing the OS to make those writes "permanent", the better the
chances that the OS has in fact already sync'd those files anyway, and
so the sync cost should be lower.  So maybe we could make every flush
permanent, but wait a little while before doing so?

Regardless of what policy we choose here (which commits must be made
"permanent", and, when) I think the approach requires that
IndexFileDeleter query the Directory so that it's only allowed to
delete older commit points once a newer commit point has successfully
become permanent.

I also worry about those applications that are accidentally flushing
too often now.  Say your app now sets maxBufferedDocs=100.  Right now,
that gives you poor performance but not disastrous, but I fear if we
do the "every commit is permanent" policy then performance could
easily become disastrous.  People who upgrade will suddenly get much
worse performance.


> Behavior on hard power shutdown
> -------------------------------
>
>                 Key: LUCENE-1044
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1044
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>         Environment: Windows Server 2003, Standard Edition, Sun Hotspot Java 1.5
>            Reporter: venkat rangan
>            Assignee: Michael McCandless
>             Fix For: 2.3
>
>         Attachments: FSyncPerfTest.java, LUCENE-1044.patch, LUCENE-1044.take2.patch, LUCENE-1044.take3.patch, LUCENE-1044.take4.patch
>
>
> When indexing a large number of documents, upon a hard power failure  (e.g. pull the power cord), the index seems to get corrupted. We start a Java application as an Windows Service, and feed it documents. In some cases (after an index size of 1.7GB, with 30-40 index segment .cfs files) , the following is observed.
> The 'segments' file contains only zeros. Its size is 265 bytes - all bytes are zeros.
> The 'deleted' file also contains only zeros. Its size is 85 bytes - all bytes are zeros.
> Before corruption, the segments file and deleted file appear to be correct. After this corruption, the index is corrupted and lost.
> This is a problem observed in Lucene 1.4.3. We are not able to upgrade our customer deployments to 1.9 or later version, but would be happy to back-port a patch, if the patch is small enough and if this problem is already solved.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org