lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-1658) Absorb NIOFSDirectory into FSDirectory
Date Sun, 31 May 2009 20:17:07 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714921#action_12714921
] 

Michael McCandless commented on LUCENE-1658:
--------------------------------------------

bq. a very lot of tests are failing with MMap.

It looks like all but one (see below) of the failures are "innocent",
eg the test hits an exception while cleaning up (removing the index
directory it had created), or the test is trying to overwrite a file.

I agree this will be a hassle for normal usage of Lucene, so let's
change the default on Win64 to SimpleFSDir (can't be NIOFSDir because
of another Sun bug).  So, for open(), on non-Windows 32 bit we use
NIOFSDir, on non-Windows 64 bit we use MMapDir, and on Windows 32 or
64 we use SimpleFSDir.  Crazy how many challenges there are with IO on
Windows from Java...

bq. This is the problem on windows: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4724038

Sheesh that's alot of votes!

Can you add that bug into the toplevel javadocs of FSDir explaining
why MMapDir is not a great choice on even Win64?

There was one odd failure I noticed, on Win64 when running tests from
a mounted remote (CIFS) drive:

{code}
    [junit] Testcase: testIndexAndMerge(org.apache.lucene.index.TestDoc):	FAILED
    [junit] junit.framework.AssertionFailedError:
    [junit] 	at org.apache.lucene.index.FieldsWriter.addRawDocuments(FieldsWriter.java:249)
    [junit] 	at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:350)
    [junit] 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
    [junit] 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
    [junit] 	at org.apache.lucene.index.TestDoc.merge(TestDoc.java:182)
    [junit] 	at org.apache.lucene.index.TestDoc.testIndexAndMerge(TestDoc.java:117)
    [junit] 	at org.apache.lucene.util.LuceneTestCase.runTest(LuceneTestCase.java:88)
{code}

I dug down and found that this test illegally opens SegmentReaders on
files that IndexWriter still has open for writing, and somehow this
causes problems when using an MMapDir.  I'll open a separate issue and
put details there.


> Absorb NIOFSDirectory into FSDirectory
> --------------------------------------
>
>                 Key: LUCENE-1658
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1658
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Store
>            Reporter: Michael McCandless
>            Assignee: Uwe Schindler
>            Priority: Minor
>             Fix For: 2.9
>
>         Attachments: LUCENE-1658-take2.patch, LUCENE-1658-take2.patch, LUCENE-1658-take3.patch,
LUCENE-1658-take3.patch, LUCENE-1658.patch, LUCENE-1658.patch, LUCENE-1658.patch
>
>
> I think whether one uses java.io.* vs java.nio.* or eventually
> java.nio2.*, or some other means, is an under-the-hood implementation
> detail of FSDirectory and doesn't merit a whole separate class.
> I think FSDirectory should be the core class one uses when one's index
> is in the filesystem.
> So, I'd like to deprecate NIOFSDirectory, absorbing it into
> FSDirectory, and add a setting "useNIO" to FSDirectory.  It should
> default to "true" for non-Windows OSs, because it gives far better
> concurrent performance on all platforms but Windows (due to known Sun
> JRE issue http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message