lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: RAMDirectory unexpectedly slows
Date Mon, 04 Jun 2012 23:53:06 GMT
This is managed by your operating system. In general OS kernels like Linux
or Windows use all free memory to cache disk accesses.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Cheng [mailto:zhoucheng2008@gmail.com]
> Sent: Monday, June 04, 2012 6:10 PM
> To: java-user@lucene.apache.org
> Subject: Re: RAMDirectory unexpectedly slows
> 
> Can I control the size of ram given to either MMapDirectory or
> ByteBufferDirectory?
> 
> On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <uwe@thetaphi.de> wrote:
> 
> > Hi,
> >
> > If you are using MMapDirectory or this ByteBufferDirectory (which is
> > similar to the first) the used RAM is outside JVM heap, it is in the
> > FS cache of the OS kernel. Giving too much memory to the JVM penalizes
> > the OS cache, so give only as much as the App needs. Lucene and the OS
> > kernel will then utilize the remaining memory for caching.
> >
> > Please read docs of MMapDirectory and inform yourself about mmap in e.g.
> > Wikipedia.
> >
> > Uwe
> > --
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, 28213 Bremen
> > http://www.thetaphi.de
> >
> >
> >
> > Cheng <zhoucheng2008@gmail.com> schrieb:
> >
> > Please shed more insight into the difference between JVM heap size and
> > the memory size used by Lucene.
> >
> > What I am getting at is that no matter however much ram I give my
> > apps, Lucene can't utilize it. Is that right?
> >
> > What about the ByteBufferDirectory? Can this specific directory
> > utilize the 2GB memory I grant to the app?
> >
> > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
> > jason.rutherglen@gmail.com> wrote:
> >
> > > If you want the index to be stored completely in RAM, there is the
> > > ByteBuffer directory [1]. Though I do not see the point in putting
> > > an index in RAM, it will be cached in RAM regardless in the OS
> > > system IO cache.
> > >
> > > 1.
> > >
> > https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja
> > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
> > >
> > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <zhoucheng2008@gmail.com>
> wrote:
> > > > My indexes are 500MB+. So it seems like that RAMDirectory is not
> > > > good
> > for
> > > > that big a size.
> > > >
> > > > My challenge, on the other side, is that I need to update the
> > > > indexes
> > > very
> > > > frequently. So, do you think MMapDirectory is the solution?
> > > >
> > > > Thanks.
> > > >
> > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
> > jack@basetechnology.com
> > > >wrote:
> > > >
> > > >> From the javadoc for RAMDirectory:
> > > >>
> > > >> "Warning: This class is not intended to work with huge indexes.
> > > Everything
> > > >> beyond several hundred megabytes will waste resources (GC
> > > >> cycles),
> > > because
> > > >> it uses an internal buffer size of 1024 bytes, producing millions
> > > >> of byte[1024] arrays. This class is optimized for small
> > > >> memory-resident indexes. It also has bad concurrency on
multithreaded
> environments.
> > > >>
> > > >> It is recommended to materialize large indexes on disk and use
> > > >> MMapDirectory, which is a high-performance directory
> > > >> implementation
> > > working
> > > >> directly on the file system cache of the operating system, so
> > > >> copying
> > > data
> > > >> to Java heap space is not useful."
> > > >>
> > > >> -- Jack Krupansky
> > > >>
> > > >> -----Original Message----- From: Cheng
> > > >> Sent: Monday, June 04, 2012 10:08 AM
> > > >> To: java-user@lucene.apache.org
> > > >> Subject: RAMDirectory unexpectedly slows
> > > >>
> > > >>
> > > >> Hi,
> > > >>
> > > >> My apps need to read from and write to some big indexes frequently.
> > So I
> > > >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB
> > > >> memory size.
> > > >>
> > > >> I notice that the speed of reading and writing unexpectedly slows
> > > >> as
> > the
> > > >> size of the indexes increases. Since the usage of RAM is less
> > > >> than
> > 20%,
> > > I
> > > >> think by default the RAMDirectory doesn't take advantage of the
> > memory I
> > > >> assigned to JVM.
> > > >>
> > > >> What are the steps to improve the reading and writing speed of
> > > >> RAMDirectory?
> > > >>
> > > >> Thanks!
> > > >> Jeff
> > > >>
> > > >>
> > >_____________________________________________
> > **_____________________________________________
> > **---------
> > > >> To unsubscribe, e-mail:
> > > >> java-user-unsubscribe@lucene.**apache.org<
> > > java-user-unsubscribe@lucene.apache.org>
> > > >> For additional commands, e-mail:
> > > >> java-user-help@lucene.apache.**org<
> > > java-user-help@lucene.apache.org>
> > > >>
> > > >>
> > >
> > >_____________________________________________
> >
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message