lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Engels" <reng...@ix.netcom.com>
Subject RE: Nio File Caching & Performance Test
Date Tue, 16 May 2006 15:43:13 GMT
My tests still hold that the NioFile I submitted is significantly faster
than the standard FSDirectory.

BUT, the memory mapped implementation is significantly faster than NioFile.
I attribute this to the overhead of managing the soft references, and
possible GC interaction.

SO, I would like to use a memory mapped reader, but I encounter OOM errors
when mapping large files, due to running out of address space.

Has anyone found a solution for this? (A 2 gig index is not all that
large...).

-----Original Message-----
From: Murat Yakici [mailto:Murat.Yakici@cis.strath.ac.uk]
Sent: Tuesday, May 16, 2006 1:55 AM
To: java-dev@lucene.apache.org
Subject: Re: Nio File Caching & Performance Test


Hi,

According to my humble tests, there is no significant improvement
either. NIO has buffer creation time costs compared to other Buffered
IOs. However, a testbed would be ideal for benchmarks.

Murat

Doug Cutting wrote:

> Robert Engels wrote:
>
>> The most important statistic is that the reading via the local cache, vs.
>> going to the OS (where the block is cached) is 3x faster (22344 vs.
>> 68578).
>> With random reads, when the block may not be in the OS cache, it is 8x
>> faster (72766 vs. 586391).
>
> [ ... ]
>
>> This test only demonstrates improvements in the low-level IO layer,
>> but one
>> could infer significant performance improvements for common searches
>> and/or
>> document retrievals.
>
>
> That is not an inference I would make.  There should be some
> improvement, but whether it is significant is not clear to me.
>
>> Is there a standard Lucene search performance I could run both with and
>> without the NioFSDirectory to demonstrate real world performance
>> improvements? I have some internal tests that I am collating, but I would
>> rather use a standard test if possible.
>
>
> No, we don't have a standard benchmark suite.  Folks have talked about
> developing one, but I don't think one yet exists.
>
> Report what you have.  Describe the collection, how it is indexed, how
> you've selected queries, and the improvement in average response time.
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message