lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-1658) Absorb NIOFSDirectory into FSDirectory
Date Tue, 26 May 2009 10:41:45 GMT


Michael McCandless commented on LUCENE-1658:

bq. Not completely sure why, but MMap failed for me on Win32 some time ago

I think on 32 bit JRE we shouldn't choose MMap in

bq. It's a one-liner. I can make a patch, but while it works better than vanilla MMapD, I'm
not yet sure it's the best approach. As for writing - okay, I'll tell you if it turns out
to be anything good.

Which way do you think "prime all bytes up front on open" should default?

There are a lot of variables. Just as you said, on 32bit systems you have to take care of
address space. So, nice for small indexes, bad for big ones. But, mmap in Java cannot be explicitly
closed, it is closed in finalizer, so even for a small index if you update really often, you
can hit an OOM even though you have enough memory. MMapD failed for me on windows. But it
is fast. It is absolutely, totally, uber-indespensible, if you have a 64bit system, fat index,
memory to spare and require lots of fast searches.

So, you can probably enable it for non-Win 64bit?

Wait, are you saying Win 64 bit has problems w/ MMapDir?  (I thought you just said Win 32
bit, above).

Is MMapDir fine w/ concurrency?  (I'd assume so).  So, if you had the choice (ie, you're running
in env w/ plenty of virtual mem), MMapDir would be preferred over NIOFSDir?

On a 64 bit env that doesn't have that much RAM, MMapDir should fare no worse than NIOFSDir,
right?  Ie both are competing for the same IO cache.  And so on a 64 bit JRE perhaps the default
should always be MMAPDir.

> Absorb NIOFSDirectory into FSDirectory
> --------------------------------------
>                 Key: LUCENE-1658
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Store
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>             Fix For: 2.9
>         Attachments: LUCENE-1658.patch, LUCENE-1658.patch
> I think whether one uses* vs java.nio.* or eventually
> java.nio2.*, or some other means, is an under-the-hood implementation
> detail of FSDirectory and doesn't merit a whole separate class.
> I think FSDirectory should be the core class one uses when one's index
> is in the filesystem.
> So, I'd like to deprecate NIOFSDirectory, absorbing it into
> FSDirectory, and add a setting "useNIO" to FSDirectory.  It should
> default to "true" for non-Windows OSs, because it gives far better
> concurrent performance on all platforms but Windows (due to known Sun
> JRE issue

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message