lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <>
Subject [jira] Commented: (LUCENE-1658) Absorb NIOFSDirectory into FSDirectory
Date Mon, 01 Jun 2009 10:28:08 GMT


Uwe Schindler commented on LUCENE-1658:

bq. I'm a bit nervous about creating MMapDirectory automatically for any OS, not just Windows.
Imagine a large index which is suddenly mapped to MMB w/o anyone intending to do so.

This would not be a problem, because MMaped files does not "consume" real memory. If a part
of the index is seen in "real memory" still is in the responsibility of the operating system.
I have a mmaped index here that is updated very often. It leads to a lot of assigned address
space in top for the process but does not affect system performance or swapping (sometimes
beyond the physical limit of the machine (mem + swap). Memory mapping files does not consume
RAM, it just consumes address space (this is why I changed the javadocs a little bit to clarify

bq. The "hack" also saves transient disk space, on all systems, right? Just because the file
name link is deleted on Unix, the bytes are still consuming disk space.

Yes, this is why I let the fix enabled for any OS.

bq. Also: I think we should fix the open logic so that on an unknown platform (not unix, not
windows, not 64 bit), return SimpleFSDir?

I would not do this.

NIO2 still has no solution for the problem. But they have a 64 bit buffer, so you could use
it instead of many buffers. On the Amsterdam ApacheCon conference, this was part of the discussion
with the guy from sun, but I forgot what the conclusion was! Maybe Oracle helps here :-)

Maybe we should move this hack to contrib ( a class that extends MMapDirectory by adding a
close method) with a big warning!

> Absorb NIOFSDirectory into FSDirectory
> --------------------------------------
>                 Key: LUCENE-1658
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Store
>            Reporter: Michael McCandless
>            Assignee: Uwe Schindler
>            Priority: Minor
>             Fix For: 2.9
>         Attachments: LUCENE-1658-take2.patch, LUCENE-1658-take2.patch, LUCENE-1658-take3.patch,
LUCENE-1658-take3.patch, LUCENE-1658-take3.patch, LUCENE-1658-take3.patch, LUCENE-1658-take3.patch,
LUCENE-1658.patch, LUCENE-1658.patch, LUCENE-1658.patch
> I think whether one uses* vs java.nio.* or eventually
> java.nio2.*, or some other means, is an under-the-hood implementation
> detail of FSDirectory and doesn't merit a whole separate class.
> I think FSDirectory should be the core class one uses when one's index
> is in the filesystem.
> So, I'd like to deprecate NIOFSDirectory, absorbing it into
> FSDirectory, and add a setting "useNIO" to FSDirectory.  It should
> default to "true" for non-Windows OSs, because it gives far better
> concurrent performance on all platforms but Windows (due to known Sun
> JRE issue

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message