lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Earwin Burrfoot (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (LUCENE-1743) MMapDirectory should only mmap large files, small files should be opened using SimpleFS/NIOFS
Date Wed, 15 Jul 2009 19:16:14 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731632#action_12731632
] 

Earwin Burrfoot edited comment on LUCENE-1743 at 7/15/09 12:14 PM:
-------------------------------------------------------------------

The initial motive for the issue seems wrong to me.

bq. "For most operating systems, mapping a file into memory is more expensive than reading
or writing a few tens of kilobytes of data via the usual read and write methods. From the
standpoint of performance it is generally only worth mapping relatively large files into memory."
It is probably right if you're doing a single read through the file. If you're opening/mapping
it and do thousands of repeated reads, mmap would be superior, because after initial mapping
it's just a memory access VS system call for file.read().

Add: In case you're not doing repeated reads, and just read these small files once from time
to time, you can totally neglect speed difference between mmap and fopen. At least it doesn't
warrant increased complexity.

      was (Author: earwin):
    The initial motive for the issue seems wrong to me.

bq. "For most operating systems, mapping a file into memory is more expensive than reading
or writing a few tens of kilobytes of data via the usual read and write methods. From the
standpoint of performance it is generally only worth mapping relatively large files into memory."
It is probably right if you're doing a single read through the file. If you're opening/mapping
it and do thousands of repeated reads, mmap would be superior, because after initial mapping
it's just a memory access VS system call for file.read().
  
> MMapDirectory should only mmap large files, small files should be opened using SimpleFS/NIOFS
> ---------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-1743
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1743
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Store
>    Affects Versions: 2.9
>            Reporter: Uwe Schindler
>            Assignee: Uwe Schindler
>             Fix For: 3.1
>
>
> This is a followup to LUCENE-1741:
> Javadocs state (in FileChannel#map): "For most operating systems, mapping a file into
memory is more expensive than reading or writing a few tens of kilobytes of data via the usual
read and write methods. From the standpoint of performance it is generally only worth mapping
relatively large files into memory."
> MMapDirectory should get a user-configureable size parameter that is a lower limit for
mmapping files. All files with a size<limit should be opened using a conventional IndexInput
from SimpleFS or NIO (another configuration option for the fallback?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message