lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: major searching performance improvement
Date Thu, 26 May 2005 16:16:27 GMT
Robert Engels wrote:
> 2. I agree that creating NioFSDirectory rather than modifying FSDirectory. I
> originally felt the memory mapped files would be the fastest, but it also
> requires OS calls, the "caching" code is CONSIDERABLY faster, since it does
> not need to do any JNI, or make OS calls.

On the contrary, mmap, if implemented well, should require fewer system 
calls and fewer buffer copies.  However, in prior, single-threaded 
benchmarks, FSDirectory appeared to still be a bit faster than 
MmapDirectory, as have other nio-based directories that folks have 
tried.  So nio's implementation may not have been optimal, although 
perhaps it is improving.  But in multi-threaded benchmarks like yours, 
an nio-based Directory should shine.

> As for creating a CachingDirectory that can cache any directory that should
> be feasible as well, but I am not sure it would perform as well as the
> direct internal cache version.

I would be surprised if it is measurably slower.  It would add only a 
single additional method call in the case of cache misses.

If separate, this could be layered on: FSDirectory for best 
single-threaded performance; on an MMapDirectory for best multi-threaded 
performance on 64-bit machines or 32-bit machines with indexes less than 
a few hundred megabytes; or on an NioDirectory for best multi-threaded 
performance on a 32-bit machine with a large index.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message