lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <>
Subject Re: potential indexing perormance improvement for compound index - cut IO - have more files though
Date Sat, 16 Dec 2006 21:05:06 GMT
Doug Cutting wrote:
> Yes.  On 32-bit systems with indexes larger than 1GB or so, memory 
> mapping is impractical, so synchronization is required around shared 
> file handles (using Java's classic i/o APIs, w/o pread).  The 
> non-compound format, with more files, has fewer synchronization 
> bottlenecks.  One could of course achieve the same improvements in other 
> ways, e.g., by pooling multiple IndexReaders per index, but in straight 
> A-to-B comparisons, folks see better throughput with non-compound 
> indexes for multi-threaded applications.

On second thought, a good fix for this might be to simply convert 
FSDirectory to use nio's pread support, eliminating file handle 
synchronization even when mmap isn't used.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message