lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: RAMDirectory unexpectedly slows
Date Mon, 18 Jun 2012 11:32:19 GMT
9 fold improvement using RAMDir over MMapDir is much more than I've
seen (~30-40% maybe) in the past.

Can you explain how you are using Lucene?

You may also want to try the CachingRAMDirectory patch on
https://issues.apache.org/jira/browse/LUCENE-4123

Mike McCandless

http://blog.mikemccandless.com

On Sat, Jun 16, 2012 at 7:18 AM, Cheng <zhoucheng2008@gmail.com> wrote:
> After a number of test, the performance of MMapDirectory is not even close
> to that of RAMDirectory, in terms of speed.
>
> My application w/ the former can only deal with 10 tasks per round while it
> could handle over 90 w/ RAMDirectory.
>
> I use the application in Linux.
>
> What can be the reasons?
>
> Thanks.
>
>
> On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <uwe@thetaphi.de> wrote:
>
>> This is managed by your operating system. In general OS kernels like Linux
>> or Windows use all free memory to cache disk accesses.
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>> > -----Original Message-----
>> > From: Cheng [mailto:zhoucheng2008@gmail.com]
>> > Sent: Monday, June 04, 2012 6:10 PM
>> > To: java-user@lucene.apache.org
>> > Subject: Re: RAMDirectory unexpectedly slows
>> >
>> > Can I control the size of ram given to either MMapDirectory or
>> > ByteBufferDirectory?
>> >
>> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <uwe@thetaphi.de> wrote:
>> >
>> > > Hi,
>> > >
>> > > If you are using MMapDirectory or this ByteBufferDirectory (which is
>> > > similar to the first) the used RAM is outside JVM heap, it is in the
>> > > FS cache of the OS kernel. Giving too much memory to the JVM penalizes
>> > > the OS cache, so give only as much as the App needs. Lucene and the OS
>> > > kernel will then utilize the remaining memory for caching.
>> > >
>> > > Please read docs of MMapDirectory and inform yourself about mmap in
>> e.g.
>> > > Wikipedia.
>> > >
>> > > Uwe
>> > > --
>> > > Uwe Schindler
>> > > H.-H.-Meier-Allee 63, 28213 Bremen
>> > > http://www.thetaphi.de
>> > >
>> > >
>> > >
>> > > Cheng <zhoucheng2008@gmail.com> schrieb:
>> > >
>> > > Please shed more insight into the difference between JVM heap size and
>> > > the memory size used by Lucene.
>> > >
>> > > What I am getting at is that no matter however much ram I give my
>> > > apps, Lucene can't utilize it. Is that right?
>> > >
>> > > What about the ByteBufferDirectory? Can this specific directory
>> > > utilize the 2GB memory I grant to the app?
>> > >
>> > > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
>> > > jason.rutherglen@gmail.com> wrote:
>> > >
>> > > > If you want the index to be stored completely in RAM, there is the
>> > > > ByteBuffer directory [1]. Though I do not see the point in putting
>> > > > an index in RAM, it will be cached in RAM regardless in the OS
>> > > > system IO cache.
>> > > >
>> > > > 1.
>> > > >
>> > > https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja
>> > > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>> > > >
>> > > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <zhoucheng2008@gmail.com>
>> > wrote:
>> > > > > My indexes are 500MB+. So it seems like that RAMDirectory is
not
>> > > > > good
>> > > for
>> > > > > that big a size.
>> > > > >
>> > > > > My challenge, on the other side, is that I need to update the
>> > > > > indexes
>> > > > very
>> > > > > frequently. So, do you think MMapDirectory is the solution?
>> > > > >
>> > > > > Thanks.
>> > > > >
>> > > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
>> > > jack@basetechnology.com
>> > > > >wrote:
>> > > > >
>> > > > >> From the javadoc for RAMDirectory:
>> > > > >>
>> > > > >> "Warning: This class is not intended to work with huge indexes.
>> > > > Everything
>> > > > >> beyond several hundred megabytes will waste resources (GC
>> > > > >> cycles),
>> > > > because
>> > > > >> it uses an internal buffer size of 1024 bytes, producing
millions
>> > > > >> of byte[1024] arrays. This class is optimized for small
>> > > > >> memory-resident indexes. It also has bad concurrency on
>> multithreaded
>> > environments.
>> > > > >>
>> > > > >> It is recommended to materialize large indexes on disk and
use
>> > > > >> MMapDirectory, which is a high-performance directory
>> > > > >> implementation
>> > > > working
>> > > > >> directly on the file system cache of the operating system,
so
>> > > > >> copying
>> > > > data
>> > > > >> to Java heap space is not useful."
>> > > > >>
>> > > > >> -- Jack Krupansky
>> > > > >>
>> > > > >> -----Original Message----- From: Cheng
>> > > > >> Sent: Monday, June 04, 2012 10:08 AM
>> > > > >> To: java-user@lucene.apache.org
>> > > > >> Subject: RAMDirectory unexpectedly slows
>> > > > >>
>> > > > >>
>> > > > >> Hi,
>> > > > >>
>> > > > >> My apps need to read from and write to some big indexes
>> frequently.
>> > > So I
>> > > > >> use RAMDirectory instead of FSDirectory, and give JVM about
2GB
>> > > > >> memory size.
>> > > > >>
>> > > > >> I notice that the speed of reading and writing unexpectedly
slows
>> > > > >> as
>> > > the
>> > > > >> size of the indexes increases. Since the usage of RAM is
less
>> > > > >> than
>> > > 20%,
>> > > > I
>> > > > >> think by default the RAMDirectory doesn't take advantage
of the
>> > > memory I
>> > > > >> assigned to JVM.
>> > > > >>
>> > > > >> What are the steps to improve the reading and writing speed
of
>> > > > >> RAMDirectory?
>> > > > >>
>> > > > >> Thanks!
>> > > > >> Jeff
>> > > > >>
>> > > > >>
>> > > >_____________________________________________
>> > > **_____________________________________________
>> > > **---------
>> > > > >> To unsubscribe, e-mail:
>> > > > >> java-user-unsubscribe@lucene.**apache.org<
>> > > > java-user-unsubscribe@lucene.apache.org>
>> > > > >> For additional commands, e-mail:
>> > > > >> java-user-help@lucene.apache.**org<
>> > > > java-user-help@lucene.apache.org>
>> > > > >>
>> > > > >>
>> > > >
>> > > >_____________________________________________
>> > >
>> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
>> > > >
>> > > >
>> > >
>> > >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message