lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Engels" <>
Subject Nio File Caching & Performance Test
Date Fri, 12 May 2006 19:38:55 GMT
I finally got around to making the NioFSDirectory with caching 1.9
compliant. I also produced a performance test case.

Below is the results on my machine:

read random = 586391
read same = 68578
nio read random = 72766
nio max mem = 203292672
nio memory = 102453248
nio hits = 14974713
nio misses = 25039
nio hit rate = 99
nio read same = 22344

The most important statistic is that the reading via the local cache, vs.
going to the OS (where the block is cached) is 3x faster (22344 vs. 68578).
With random reads, when the block may not be in the OS cache, it is 8x
faster (72766 vs. 586391).

Attached are all of the files needed to run the test (The NioFSDirectory is
not needed for the test).

The revised NioFile shares one cache for all Nio files. The revised
MemoryCache uses SoftReferences to allow the cache to grow unbounded, and
let the GC handle cache reductions (it seems to be that for most JVMs,
SoftReferences are reclaimed in a LRU order which helps).

This test only demonstrates improvements in the low-level IO layer, but one
could infer significant performance improvements for common searches and/or
document retrievals.

Is there a standard Lucene search performance I could run both with and
without the NioFSDirectory to demonstrate real world performance
improvements? I have some internal tests that I am collating, but I would
rather use a standard test if possible.

View raw message