Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <440544082.1247602994971.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 13:23:14 -0700 (PDT)
From: "Uwe Schindler (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Issue Comment Edited: (LUCENE-1743) MMapDirectory should
 only mmap large files, small files should be opened using SimpleFS/NIOFS
In-Reply-To: <1857108067.1247564355510.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/LUCENE-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731101#action_12731101 ] 

Uwe Schindler edited comment on LUCENE-1743 at 7/14/09 1:21 PM:
----------------------------------------------------------------

bq. HighlyConcurentForWindows64WithTermDictionaryInRamAndStoredFieldsOnDiskDirectory just for me  

By the way, for MMapDirectory the MappedByteBuffer.load() method should be somehow accesible/configureable to create this TermDictionaryInRam part (IndexInput would call load() and tells the OS to swap as much as possible of the mmaped file into RAM). Just an idea.

The Hyper FileSwitchDirectory was my idea yesterday, too. As Mike said, at least the getDirectory() should be configureable.

And for some good defaults, a factory could be provided like getUwesBestOfMMapDirectoryFor32BitOSs(File, LockFactory). What I do not like with the current FileSwitchDir is the fact, that you must create instances with the same Dir and LockFactory for each sub-directory (e.g. what happens if you use 2 different LockFactories on the same physical dir inside a FileSwitchDir?). Maybe the FileSwitchDirectory could just get the File and LockFactory once and creates the instances? Many ideas...

      was (Author: thetaphi):
    bq. HighlyConcurentForWindows64WithTermDictionaryInRamAndStoredFieldsOnDiskDirectory just for me  

By the way, for MMapDirectory the MappedByteBuffer.load() method should be somehow accesible/configureable to create this TermDictionaryInRam part (IndexInput would call load() and tells the OS to swap as much as possible of the mmaped file into RAM). Just an idea.

The Hyper FileSwitchDirectory was my idea yesterday, too. As Mike that, at least the getDirectory() should be configureable.

And for some good defaults, a factory could be provided like getUwesBestOfMMapDirectoryFor32BitOSs(File, LockFactory). What I do not like with the current FileSwitchDir is the fact, that you must create instances with the same Dir and LockFactory for each sub-directory (e.g. what happens if you use 2 different LockFactories on the same physical dir inside a FileSwitchDir?). Maybe the FileSwitchDirectory could just get the File and LockFactory once and creates the instances? Many ideas...
  
> MMapDirectory should only mmap large files, small files should be opened using SimpleFS/NIOFS
> ---------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-1743
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1743
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Store
>    Affects Versions: 2.9
>            Reporter: Uwe Schindler
>            Assignee: Uwe Schindler
>             Fix For: 3.1
>
>
> This is a followup to LUCENE-1741:
> Javadocs state (in FileChannel#map): "For most operating systems, mapping a file into memory is more expensive than reading or writing a few tens of kilobytes of data via the usual read and write methods. From the standpoint of performance it is generally only worth mapping relatively large files into memory."
> MMapDirectory should get a user-configureable size parameter that is a lower limit for mmapping files. All files with a size<limit should be opened using a conventional IndexInput from SimpleFS or NIO (another configuration option for the fallback?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org