Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <724079135.1130362255797.JavaMail.jira@ajax.apache.org>
Date: Wed, 26 Oct 2005 23:30:55 +0200 (CEST)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Commented: (LUCENE-414) Java NIO patch against Lucene 1.9
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

    [ http://issues.apache.org/jira/browse/LUCENE-414?page=comments#action_12356015 ] 

Doug Cutting commented on LUCENE-414:
-------------------------------------

The channels should all be opened when the IndexInput is created, as files can subsequently get deleted.

Also, I'm not sure why this uses nio.  Classic io would also permit you to have multiple file handles per file, for more parallel io.  So you could just patch FSDirectory to permit that, no?

Finally, if files are on a single drive, then the concurrency improvements are probably negligible.  This would only really pay off with a RAID, where different parts of a file are stored on different physical devices.  Or am I missing something?

> Java NIO patch against Lucene 1.9
> ---------------------------------
>
>          Key: LUCENE-414
>          URL: http://issues.apache.org/jira/browse/LUCENE-414
>      Project: Lucene - Java
>         Type: Bug
>   Components: Store
>     Versions: unspecified
>  Environment: Operating System: All
> Platform: All
>     Reporter: Chris Lamprecht
>     Assignee: Lucene Developers
>  Attachments: MemoryLRUCache.java, NioFile.java, nio-lucene-1.9.patch
>
> Robert Engels previously submitted a patch against Lucene 1.4 for a Java NIO-
> based Directory implementation.  It also included some changes to FSDirectory 
> to allow better concurrency when searching from multiple threads.  The 
> complete thread is at:
> http://mail-archives.apache.org/mod_mbox/lucene-java-dev/200505.mbox/%
> 3cLMENLAOACIBLMOIILNNNEEOEEPAA.rengels@ix.netcom.com%3e
> This thread ended with Doug Cutting suggesting that someone port Robert's 
> changes to the SVN trunk.  This is what I've done in this patch.
> There are two parts to the patch.  The first part modifies FieldsReader, 
> CompoundFileReader, and SegmentReader, to allow better concurrency when 
> reading an index.  The second part includes the new NioFSDirectory 
> implementation, and makes small changes to FSDirectory and IndexInput to 
> accomodate this change.  I'll put a more detailed outline of the changes to 
> each file in a separate message.
> To use the new NioFSDirectory, set the system property 
> org.apache.lucene.FSDirectory.class to 
> org.apache.lucene.store.NioFSDirectory.  This will cause 
> FSDirectory.getDirectory() to return an NioFSDirectory instance.  By default, 
> NioFile limits the number of concurrent channels to 4, but you can override 
> this by setting the system property org.apache.lucene.nio.channels.  
> I did some performance tests with these patches.  The biggest improvement came 
> from the concurrency improvements.  NioFSDirectory performed about the same as 
> FSDirectory (with the concurrency improvements).  
> I ran my tests under Fedora Core 1; uname -a reports:
> Linux myhost 2.4.22-1.2199.nptlsmp #1 SMP Wed Aug 4 11:48:29 EDT 2004 i686 
> i686 i386 GNU/Linux
> The machine is a dual xeon 2.8GHz with 4GB RAM, and the tests were run against 
> a 9GB compound index file.  The tests were run "hot" -- with everything 
> already cached by linux's filesystem cache.  The numbers are:
> FSDirectory without patch:          13.3 searches per second
> FSDirectory WITH concurrency patch: 14.3 searches per second
> Both tests were run with 6 concurrent threads, which gave the highest numbers 
> in each case.  I suspect that the concurrency improvements would make a bigger 
> difference on a more realistic test where the index isn't all cached in RAM 
> already, since the I/O happens whild holding the sychronized lock.  Patches to 
> follow...
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org