lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Engels" <>
Subject RE: Nio File Caching & Performance Test
Date Tue, 16 May 2006 15:43:13 GMT
My tests still hold that the NioFile I submitted is significantly faster
than the standard FSDirectory.

BUT, the memory mapped implementation is significantly faster than NioFile.
I attribute this to the overhead of managing the soft references, and
possible GC interaction.

SO, I would like to use a memory mapped reader, but I encounter OOM errors
when mapping large files, due to running out of address space.

Has anyone found a solution for this? (A 2 gig index is not all that

-----Original Message-----
From: Murat Yakici []
Sent: Tuesday, May 16, 2006 1:55 AM
Subject: Re: Nio File Caching & Performance Test


According to my humble tests, there is no significant improvement
either. NIO has buffer creation time costs compared to other Buffered
IOs. However, a testbed would be ideal for benchmarks.


Doug Cutting wrote:

> Robert Engels wrote:
>> The most important statistic is that the reading via the local cache, vs.
>> going to the OS (where the block is cached) is 3x faster (22344 vs.
>> 68578).
>> With random reads, when the block may not be in the OS cache, it is 8x
>> faster (72766 vs. 586391).
> [ ... ]
>> This test only demonstrates improvements in the low-level IO layer,
>> but one
>> could infer significant performance improvements for common searches
>> and/or
>> document retrievals.
> That is not an inference I would make.  There should be some
> improvement, but whether it is significant is not clear to me.
>> Is there a standard Lucene search performance I could run both with and
>> without the NioFSDirectory to demonstrate real world performance
>> improvements? I have some internal tests that I am collating, but I would
>> rather use a standard test if possible.
> No, we don't have a standard benchmark suite.  Folks have talked about
> developing one, but I don't think one yet exists.
> Report what you have.  Describe the collection, how it is indexed, how
> you've selected queries, and the improvement in average response time.
> Doug
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message