lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <>
Subject [jira] Commented: (LUCENE-1743) MMapDirectory should only mmap large files, small files should be opened using SimpleFS/NIOFS
Date Wed, 15 Jul 2009 19:18:14 GMT


Uwe Schindler commented on LUCENE-1743:

A typical example, where MMap would be the wrong thing are e.g. norms. They are read one time
and then the file is not accessed anymore. It would only be cool, if MMapDir could directly
return the mapped array (but MappedByteBuffer.array() does not work) and use it as norms array.
That would be cool.

My problem was more with all these small files like segments_XXXX and segments.gen or *.del
files. They are small and only used one time. Mapping them is just waste of resources and
completely useless (even slower that opening them directly). This is why I said, some limit
or file extension based mapping would be good.

> MMapDirectory should only mmap large files, small files should be opened using SimpleFS/NIOFS
> ---------------------------------------------------------------------------------------------
>                 Key: LUCENE-1743
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Store
>    Affects Versions: 2.9
>            Reporter: Uwe Schindler
>            Assignee: Uwe Schindler
>             Fix For: 3.1
> This is a followup to LUCENE-1741:
> Javadocs state (in FileChannel#map): "For most operating systems, mapping a file into
memory is more expensive than reading or writing a few tens of kilobytes of data via the usual
read and write methods. From the standpoint of performance it is generally only worth mapping
relatively large files into memory."
> MMapDirectory should get a user-configureable size parameter that is a lower limit for
mmapping files. All files with a size<limit should be opened using a conventional IndexInput
from SimpleFS or NIO (another configuration option for the fallback?).

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message