lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Nioche" <Julien.Nio...@lingway.com>
Subject Re: Directory implementation using NIO (moved from Lucene User List)
Date Wed, 09 Jul 2003 09:22:51 GMT
Hello Francesco,

I ran a little test using your implementation of NIODirectory on a 384 MB
index using a few hundred different queries on win 2K with 523M RAM.

On the first run (with Xms 200M and Xmx300M)
> FSDirectory :: 241 sec
> NIODirectory :: 206 sec

I tried a second time (there is surely a caching effect in the OS)
> FSDirectory :: 223 sec
> NIODirectory :: 182 sec

Of course the absolute value of the results is not really interesting. In my
case and with my hardware configuration, the index is too big to be loaded
in memory with RAMDirectory.
On a smaller index or with less RAM available FSDirectory should be faster?

BTW modifications are to be made in the org.apache.lucene.store.InputStream
class, not in Directory.

Has anybody else tried it? Do you find similar results? What does it bring
on a bigger index?

Cheers

-----------------------------------
Julien Nioche
Consultant in Linguistic Engineering
www.lingway.com


----- Original Message -----
From: "Francesco Bellomi" <fbellomi@libero.it>
To: "Lucene Users List" <lucene-user@jakarta.apache.org>
Sent: Sunday, July 06, 2003 10:49 PM
Subject: Directory implementation using NIO


> Hi,
>
> I developed a Directory implementation that accesses an index stored on
the
> filesystem using memory-mapped files (as provided by the NIO API,
introduced
> in Java 1.4).
>
> You can download the complied jar and the source from here:
> www.fran.it/lucene-NIO.zip
>
> Basically there are 3 new classes: NIODirectory, NIOInputStream and
> NIOOutputStream. They are heavily based on FSDirectory, FSInputStream and
> FSOutputStream.
>
> NIOInputStram provides memory-mapped access to files. It does not rely on
> Lucene InputStream's caching feature, since direct access to the
> memory-mapped file should be faster. Also, cloned stream with independent
> positions are implemented using NIO buffer duplication (a buffer duplicate
> holds the same content but has its own position), and so the
implementation
> logic is much simpler than FSInputStream's.
>
> Some methods of Directory have been overridden to replace the caching
> feature. Some of then were final in Directory, so I have used a slightly
> modified version of Directory.java (BTW, I wonder why so many methods in
> Lucene are made final...)
>
> These classes only works with the recently released Java 1.4.2. This is
due
> to the fact that buffers connected with memory-mapped files could not be
> programmatically unmapped in previous releases, (they were unmapped only
> through finalization) and actively mapped files cannot be deleted. These
> issue are partially resolved with 1.4.2.
>
> NIOOutputStream is the same as FSOutputStream; I don't know any way to
take
> advantege of NIO for writing indexes (memory mapped buffers have a static
> size, so they are not useful if your file is growing).
>
> I don't have a benchmarking suite for Lucene, so I can't accurately
evaluate
> the speed of this implementation. I tested it on a small application I am
> developing and it seems to work well, but I think my test are not
> significative. Of course only the searching feature is expected to be
> faster, since the index writing is unchanged.
>
> Francesco
>
> -
> Francesco Bellomi
> "Use truth to show illusion,
> and illusion to show truth."
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message