lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng <zhoucheng2...@gmail.com>
Subject Re: RAMDirectory unexpectedly slows
Date Mon, 04 Jun 2012 16:09:44 GMT
Can I control the size of ram given to either MMapDirectory or
ByteBufferDirectory?

On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <uwe@thetaphi.de> wrote:

> Hi,
>
> If you are using MMapDirectory or this ByteBufferDirectory (which is
> similar to the first) the used RAM is outside JVM heap, it is in the FS
> cache of the OS kernel. Giving too much memory to the JVM penalizes the OS
> cache, so give only as much as the App needs. Lucene and the OS kernel will
> then utilize the remaining memory for caching.
>
> Please read docs of MMapDirectory and inform yourself about mmap in e.g.
> Wikipedia.
>
> Uwe
> --
> Uwe Schindler
> H.-H.-Meier-Allee 63, 28213 Bremen
> http://www.thetaphi.de
>
>
>
> Cheng <zhoucheng2008@gmail.com> schrieb:
>
> Please shed more insight into the difference between JVM heap size and the
> memory size used by Lucene.
>
> What I am getting at is that no matter however much ram I give my apps,
> Lucene can't utilize it. Is that right?
>
> What about the ByteBufferDirectory? Can this specific directory utilize the
> 2GB memory I grant to the app?
>
> On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
> jason.rutherglen@gmail.com> wrote:
>
> > If you want the index to be stored completely in RAM, there is the
> > ByteBuffer directory [1]. Though I do not see the point in putting an
> > index in RAM, it will be cached in RAM regardless in the OS system IO
> > cache.
> >
> > 1.
> >
> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
> >
> > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <zhoucheng2008@gmail.com> wrote:
> > > My indexes are 500MB+. So it seems like that RAMDirectory is not good
> for
> > > that big a size.
> > >
> > > My challenge, on the other side, is that I need to update the indexes
> > very
> > > frequently. So, do you think MMapDirectory is the solution?
> > >
> > > Thanks.
> > >
> > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
> jack@basetechnology.com
> > >wrote:
> > >
> > >> From the javadoc for RAMDirectory:
> > >>
> > >> "Warning: This class is not intended to work with huge indexes.
> > Everything
> > >> beyond several hundred megabytes will waste resources (GC cycles),
> > because
> > >> it uses an internal buffer size of 1024 bytes, producing millions of
> > >> byte[1024] arrays. This class is optimized for small memory-resident
> > >> indexes. It also has bad concurrency on multithreaded environments.
> > >>
> > >> It is recommended to materialize large indexes on disk and use
> > >> MMapDirectory, which is a high-performance directory implementation
> > working
> > >> directly on the file system cache of the operating system, so copying
> > data
> > >> to Java heap space is not useful."
> > >>
> > >> -- Jack Krupansky
> > >>
> > >> -----Original Message----- From: Cheng
> > >> Sent: Monday, June 04, 2012 10:08 AM
> > >> To: java-user@lucene.apache.org
> > >> Subject: RAMDirectory unexpectedly slows
> > >>
> > >>
> > >> Hi,
> > >>
> > >> My apps need to read from and write to some big indexes frequently.
> So I
> > >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
> > >> size.
> > >>
> > >> I notice that the speed of reading and writing unexpectedly slows as
> the
> > >> size of the indexes increases. Since the usage of RAM is less than
> 20%,
> > I
> > >> think by default the RAMDirectory doesn't take advantage of the
> memory I
> > >> assigned to JVM.
> > >>
> > >> What are the steps to improve the reading and writing speed of
> > >> RAMDirectory?
> > >>
> > >> Thanks!
> > >> Jeff
> > >>
> > >>
> >_____________________________________________
> **_____________________________________________
> **---------
> > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<
> > java-user-unsubscribe@lucene.apache.org>
> > >> For additional commands, e-mail: java-user-help@lucene.apache.**org<
> > java-user-help@lucene.apache.org>
> > >>
> > >>
> >
> >_____________________________________________
>
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message