lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Uwe Schindler <...@thetaphi.de>
Subject Re: RAMDirectory unexpectedly slows
Date Mon, 04 Jun 2012 15:42:52 GMT
Hi,

If you are using MMapDirectory or this ByteBufferDirectory (which is similar to the first)
the used RAM is outside JVM heap, it is in the FS cache of the OS kernel. Giving too much
memory to the JVM penalizes the OS cache, so give only as much as the App needs. Lucene and
the OS kernel will then utilize the remaining memory for caching.

Please read docs of MMapDirectory and inform yourself about mmap in e.g. Wikipedia.

Uwe
--
Uwe Schindler
H.-H.-Meier-Allee 63, 28213 Bremen
http://www.thetaphi.de



Cheng <zhoucheng2008@gmail.com> schrieb:

Please shed more insight into the difference between JVM heap size and the
memory size used by Lucene.

What I am getting at is that no matter however much ram I give my apps,
Lucene can't utilize it. Is that right?

What about the ByteBufferDirectory? Can this specific directory utilize the
2GB memory I grant to the app?

On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
jason.rutherglen@gmail.com> wrote:

> If you want the index to be stored completely in RAM, there is the
> ByteBuffer directory [1]. Though I do not see the point in putting an
> index in RAM, it will be cached in RAM regardless in the OS system IO
> cache.
>
> 1.
> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>
> On Mon, Jun 4, 2012 at 10:55 AM, Cheng <zhoucheng2008@gmail.com> wrote:
> > My indexes are 500MB+. So it seems like that RAMDirectory is not good for
> > that big a size.
> >
> > My challenge, on the other side, is that I need to update the indexes
> very
> > frequently. So, do you think MMapDirectory is the solution?
> >
> > Thanks.
> >
> > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <jack@basetechnology.com
> >wrote:
> >
> >> From the javadoc for RAMDirectory:
> >>
> >> "Warning: This class is not intended to work with huge indexes.
> Everything
> >> beyond several hundred megabytes will waste resources (GC cycles),
> because
> >> it uses an internal buffer size of 1024 bytes, producing millions of
> >> byte[1024] arrays. This class is optimized for small memory-resident
> >> indexes. It also has bad concurrency on multithreaded environments.
> >>
> >> It is recommended to materialize large indexes on disk and use
> >> MMapDirectory, which is a high-performance directory implementation
> working
> >> directly on the file system cache of the operating system, so copying
> data
> >> to Java heap space is not useful."
> >>
> >> -- Jack Krupansky
> >>
> >> -----Original Message----- From: Cheng
> >> Sent: Monday, June 04, 2012 10:08 AM
> >> To: java-user@lucene.apache.org
> >> Subject: RAMDirectory unexpectedly slows
> >>
> >>
> >> Hi,
> >>
> >> My apps need to read from and write to some big indexes frequently. So I
> >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
> >> size.
> >>
> >> I notice that the speed of reading and writing unexpectedly slows as the
> >> size of the indexes increases. Since the usage of RAM is less than 20%,
> I
> >> think by default the RAMDirectory doesn't take advantage of the memory I
> >> assigned to JVM.
> >>
> >> What are the steps to improve the reading and writing speed of
> >> RAMDirectory?
> >>
> >> Thanks!
> >> Jeff
> >>
> >>
>_____________________________________________
**_____________________________________________
**---------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<
> java-user-unsubscribe@lucene.apache.org>
> >> For additional commands, e-mail: java-user-help@lucene.apache.**org<
> java-user-help@lucene.apache.org>
> >>
> >>
>
>_____________________________________________

> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message