Mailing-List: contact lucene-dev-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Lucene Developers List" <lucene-dev@jakarta.apache.org>
Message-ID: <010e01c3461c$e5976a00$c4f21b9d@ARIEL>
From: "Francesco Bellomi \(UNIVR\)" <bellomi@profs.sci.univr.it>
To: "Lucene Developers List" <lucene-dev@jakarta.apache.org>
References: <008801c34400$2a67f290$2700a8c0@ARIEL>
 <004c01c345fb$ad97a410$6a0010ac@teck>
Subject: Re: Directory implementation using NIO (moved from Lucene User List)
Date: Wed, 9 Jul 2003 15:20:41 +0200
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit

Hi Julien,
I just subscribed to lucene-dev.

First of all, thank you for your tests.

> On the first run (with Xms 200M and Xmx300M)
>> FSDirectory :: 241 sec
>> NIODirectory :: 206 sec
>
> I tried a second time (there is surely a caching effect in the OS)
>> FSDirectory :: 223 sec
>> NIODirectory :: 182 sec

Roughly, a 20% speedup

> In my case and with my hardware configuration, the index
> is too big to be loaded in memory with RAMDirectory.
> On a smaller index or with less RAM available FSDirectory should be
> faster?

Memory mapping deals with the low-level virtual memory system, that is,
(simplifying) it caches in RAM the file blocks when they are accessed, and
then discards the least recently used blocks when more space is needed. The
filesystem has a similar approach, but the caching happens at different
levels, including a java-based one. I think NIO is faster simply because
it's more direct and low-level. This advantage should remain also with less
RAM and a smaller index, but only testing will tell.

It would be interesting to test NIO against RAM, with an index smaller
enough to be entirely contained in memory. Cached memory-mapped buffer
should be (theoretically) accessed nearly as fast as RAM.

However, the case I'm really interested in is the one you tested (match
against FS with an index bigger than the memory), and I'm happy to see there
is a speedup.

>
> BTW modifications are to be made in the
> org.apache.lucene.store.InputStream class, not in Directory.

You are absolutely right

>
> Has anybody else tried it? Do you find similar results?
> What does it
> bring on a bigger index?
>
> Cheers
> Julien Nioche

Thanks indeed,
Francesco

-
Francesco Bellomi
"Use truth to show illusion,
and illusion to show truth."


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org