lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-425) "Cannot rename segments.new to segments"
Date Wed, 09 Aug 2006 13:00:16 GMT
    [ http://issues.apache.org/jira/browse/LUCENE-425?page=comments#action_12426910 ] 
            
Michael McCandless commented on LUCENE-425:
-------------------------------------------


There was a recent thread on java-user that looks very similar to this issue:

  http://www.gossamer-threads.com/lists/lucene/java-user/38669

When Lucene writes a new "segments" or new "deletable" file it first
writes to X.new and then renames X.new --> X.  Apparently that
renameTo can sometimes fail, even though both the X.new and the X file
are "closed" from Java.

The "best guess" from that discussion is that there is a JVM bug on
certain JVMs on Windows, whereby a file handle (at the Windows OS
level) is held open even after from Java it has been "closed".

Ant has a workaround for this issue: force a GC, wait, and then retry.
I don't really like that workaround for Lucene because we should not
suddenly / unexpectedly do a system level action like forcing GC --
it's a "leaky abstraction"
(http://www.joelonsoftware.com/articles/LeakyAbstractions.html).

Which JVM are you using?  I couldn't reproduce this; I made a small
change to the IndexFiles demo command-line tool and then refreshed the
view from Windows Explorer.  On Sun JDK 1.4.2_10 and Sun JDK 1.5.0_07
I couldn't get the IOException.

Issue LUCNE-554 also proposes more robust logic around opening the
.NEW file if it exists.  I think that makes alot of sense.



> "Cannot rename segments.new to segments"
> ----------------------------------------
>
>                 Key: LUCENE-425
>                 URL: http://issues.apache.org/jira/browse/LUCENE-425
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 1.4
>         Environment: Operating System: Windows XP
> Platform: PC
>            Reporter: Mikko Noromaa
>         Assigned To: Lucene Developers
>
> AddDocument periodically fails with "Cannot rename segments.new to segments". 
> Sometimes it happens after 10,000 documents, sometimes after 200,000 documents.
> My index is placed on a FAT partition. My Lucene version is 1.4.3. I am using 
> a single IndexWriter object and doing all AddDocuments calls serially from a 
> single thread.
> I saw some discussion about this problem in the mailing lists, but looks like 
> the problem wasn't completely eliminated yet.
> My suggested solutions are as follows:
> - If renameTo() fails, wait some time and retry. After this, apply the other 
> methods that are now in FSDirectory.java. Retry them as well after waiting for 
> some time.
> - In the IndexWriter constructor, check for 'segments.new' if 'segments' 
> cannot be found. This would allow recovery if FSDirectory failed on a previous 
> run. Now I have to write the recovery code myself outside of Lucene.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message