lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Engels" <reng...@ix.netcom.com>
Subject major searching performance improvement
Date Wed, 25 May 2005 20:42:36 GMT
Attached are files that dramatically improve the searching performance (2x
improvement on several hardware configurations!) in a multithreaded, high
concurrency environment.

The change has 3 parts:

1) remove synchronization required in SegmentReader document. This required
changes to FieldsReader to handle concurrent access.

2) change FSDirectory to use a 'nio' to improve concurrency. Changed to use
NioFile. This class has some workaround because under Windows, the
FileChannel is not fully reentrant, and so allocates multiple handles per
physical file - this code can be removed under non-Windows systems. This
also required changes to InputStream to allow for reading at a direct
offset.

3) move disk buffering into the Java layer to avoid the overhead of OS
calls. The buffer percentage can be configured to store the entire index in
memory. Running with as little as a 10% cache, the performance is
dramatically improved. Reading larger blocks also improves the performance
in most cases, but can actually degrade performance if doing very small
reads. Using the cache implies that you have configured the JVM to have as
much heap space available as the percent of index size on the disk. The
NioFile can be easily changed to use a "soft" cache to avoid the potential
of OutOfMemoryExceptions.

I welcome comments/feedback.

Robert Engels


Mime
View raw message