lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzi...@apache.org
Subject DO NOT REPLY [Bug 35838] New: - Java NIO patch against Lucene 1.9
Date Sun, 24 Jul 2005 03:26:07 GMT
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG·
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=35838>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND·
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=35838

           Summary: Java NIO patch against Lucene 1.9
           Product: Lucene
           Version: unspecified
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Store
        AssignedTo: java-dev@lucene.apache.org
        ReportedBy: clamprecht@gmail.com


Robert Engels previously submitted a patch against Lucene 1.4 for a Java NIO-
based Directory implementation.  It also included some changes to FSDirectory 
to allow better concurrency when searching from multiple threads.  The 
complete thread is at:

http://mail-archives.apache.org/mod_mbox/lucene-java-dev/200505.mbox/%
3cLMENLAOACIBLMOIILNNNEEOEEPAA.rengels@ix.netcom.com%3e

This thread ended with Doug Cutting suggesting that someone port Robert's 
changes to the SVN trunk.  This is what I've done in this patch.

There are two parts to the patch.  The first part modifies FieldsReader, 
CompoundFileReader, and SegmentReader, to allow better concurrency when 
reading an index.  The second part includes the new NioFSDirectory 
implementation, and makes small changes to FSDirectory and IndexInput to 
accomodate this change.  I'll put a more detailed outline of the changes to 
each file in a separate message.

To use the new NioFSDirectory, set the system property 
org.apache.lucene.FSDirectory.class to 
org.apache.lucene.store.NioFSDirectory.  This will cause 
FSDirectory.getDirectory() to return an NioFSDirectory instance.  By default, 
NioFile limits the number of concurrent channels to 4, but you can override 
this by setting the system property org.apache.lucene.nio.channels.  

I did some performance tests with these patches.  The biggest improvement came 
from the concurrency improvements.  NioFSDirectory performed about the same as 
FSDirectory (with the concurrency improvements).  

I ran my tests under Fedora Core 1; uname -a reports:
Linux myhost 2.4.22-1.2199.nptlsmp #1 SMP Wed Aug 4 11:48:29 EDT 2004 i686 
i686 i386 GNU/Linux

The machine is a dual xeon 2.8GHz with 4GB RAM, and the tests were run against 
a 9GB compound index file.  The tests were run "hot" -- with everything 
already cached by linux's filesystem cache.  The numbers are:

FSDirectory without patch:          13.3 searches per second
FSDirectory WITH concurrency patch: 14.3 searches per second

Both tests were run with 6 concurrent threads, which gave the highest numbers 
in each case.  I suspect that the concurrency improvements would make a bigger 
difference on a more realistic test where the index isn't all cached in RAM 
already, since the I/O happens whild holding the sychronized lock.  Patches to 
follow...

Thoughts?

-- 
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message