Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: pass (hermes.apache.org: local policy)
Message-ID: <NZADEBNXHQNTKPIANVOBKNAJLVHTKMLQN3N4AFEH@ziplip.com>
Date: Thu, 26 May 2005 12:21:49 -0700 (PDT)
From: Arvind Srinivasan <lucene@ziplip.com>
Reply-To: Arvind Srinivasan <lucene@ziplip.com>
To: <java-dev@lucene.apache.org>
Subject: Re: Potential Segment corruption
Cc: 
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Thanks for the quick turn around.  


>I think the fix is much simpler.  This is a bug in FSDirectory.
>Directory.createOutput() should always create a new empty file, and
>FSDirectory's implementation does not ensure this.  It should try to
>delete the file before opening it and/or call RandomAccessFile.setLength(0).

Agreed.


>I've attached a patch.  Does this fix things for you?

The patch on the follow up mail does look good. However, I have additional
concerns:

(a) deleteFile call may fail. eg. File is left open from the previous exception.
This makes me believe the ideal scenario is to not to reuse the segment name
once the newSegment call issues one. I strongly recommend this for 2.0.

(b)  We should add a comment on Directory interface, so that people who implement
their own directory do not run into this issue and for that reason, I like 
RandomAccessFile.setLength(0). However, since the code currently calls createFile 
from many locations, we could add a comment something like this:

---
 /** Creates a new, empty file in the directory with the given name.
     Returns a stream writing this file. 
     Ensure the OutputStream points to 0 byte length file.
  */
  public abstract OutputStream createFile(String name)
       throws IOException;
---


A side note: I had the task to recover one such index. Initially, I thought
since the bytes are overwritten, the segment should not be corrupted and
can be recovered.  However, the reader code relies on the file length (FieldsReader)
and so if you do not know the exact length, you cannot recover the index. 
It seems to me that with a few tweak on the read, the index can be made robust to 
simple failures. We already have the ability to discard the corrupted segment and 
allow searches to continue on other segments.  I think this tread into the
 White board type of stuff. I am not sure if I can write to the whiteboard.

   
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org