lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: RAMDirectory unexpectedly slows
Date Sat, 30 Jun 2012 10:24:32 GMT
It hasn't been committed yet ... it's a patch on
https://issues.apache.org/jira/browse/LUCENE-4123

Mike McCandless

http://blog.mikemccandless.com


On Sat, Jun 30, 2012 at 4:24 AM, Cheng <zhoucheng2008@gmail.com> wrote:
> Hi,
>
> I can't find the  CachingRAMDirectory  in Lucene 3.6. Is this decaperated?
>
> Thanks
>
> On Mon, Jun 18, 2012 at 7:32 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> 9 fold improvement using RAMDir over MMapDir is much more than I've
>> seen (~30-40% maybe) in the past.
>>
>> Can you explain how you are using Lucene?
>>
>> You may also want to try the CachingRAMDirectory patch on
>> https://issues.apache.org/jira/browse/LUCENE-4123
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Sat, Jun 16, 2012 at 7:18 AM, Cheng <zhoucheng2008@gmail.com> wrote:
>> > After a number of test, the performance of MMapDirectory is not even
>> close
>> > to that of RAMDirectory, in terms of speed.
>> >
>> > My application w/ the former can only deal with 10 tasks per round while
>> it
>> > could handle over 90 w/ RAMDirectory.
>> >
>> > I use the application in Linux.
>> >
>> > What can be the reasons?
>> >
>> > Thanks.
>> >
>> >
>> > On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <uwe@thetaphi.de> wrote:
>> >
>> >> This is managed by your operating system. In general OS kernels like
>> Linux
>> >> or Windows use all free memory to cache disk accesses.
>> >>
>> >> -----
>> >> Uwe Schindler
>> >> H.-H.-Meier-Allee 63, D-28213 Bremen
>> >> http://www.thetaphi.de
>> >> eMail: uwe@thetaphi.de
>> >>
>> >>
>> >> > -----Original Message-----
>> >> > From: Cheng [mailto:zhoucheng2008@gmail.com]
>> >> > Sent: Monday, June 04, 2012 6:10 PM
>> >> > To: java-user@lucene.apache.org
>> >> > Subject: Re: RAMDirectory unexpectedly slows
>> >> >
>> >> > Can I control the size of ram given to either MMapDirectory or
>> >> > ByteBufferDirectory?
>> >> >
>> >> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <uwe@thetaphi.de>
>> wrote:
>> >> >
>> >> > > Hi,
>> >> > >
>> >> > > If you are using MMapDirectory or this ByteBufferDirectory (which
is
>> >> > > similar to the first) the used RAM is outside JVM heap, it is
in the
>> >> > > FS cache of the OS kernel. Giving too much memory to the JVM
>> penalizes
>> >> > > the OS cache, so give only as much as the App needs. Lucene and
the
>> OS
>> >> > > kernel will then utilize the remaining memory for caching.
>> >> > >
>> >> > > Please read docs of MMapDirectory and inform yourself about mmap
in
>> >> e.g.
>> >> > > Wikipedia.
>> >> > >
>> >> > > Uwe
>> >> > > --
>> >> > > Uwe Schindler
>> >> > > H.-H.-Meier-Allee 63, 28213 Bremen
>> >> > > http://www.thetaphi.de
>> >> > >
>> >> > >
>> >> > >
>> >> > > Cheng <zhoucheng2008@gmail.com> schrieb:
>> >> > >
>> >> > > Please shed more insight into the difference between JVM heap
size
>> and
>> >> > > the memory size used by Lucene.
>> >> > >
>> >> > > What I am getting at is that no matter however much ram I give
my
>> >> > > apps, Lucene can't utilize it. Is that right?
>> >> > >
>> >> > > What about the ByteBufferDirectory? Can this specific directory
>> >> > > utilize the 2GB memory I grant to the app?
>> >> > >
>> >> > > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
>> >> > > jason.rutherglen@gmail.com> wrote:
>> >> > >
>> >> > > > If you want the index to be stored completely in RAM, there
is the
>> >> > > > ByteBuffer directory [1]. Though I do not see the point in
putting
>> >> > > > an index in RAM, it will be cached in RAM regardless in the
OS
>> >> > > > system IO cache.
>> >> > > >
>> >> > > > 1.
>> >> > > >
>> >> > >
>> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja
>> >> > > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>> >> > > >
>> >> > > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <zhoucheng2008@gmail.com>
>> >> > wrote:
>> >> > > > > My indexes are 500MB+. So it seems like that RAMDirectory
is not
>> >> > > > > good
>> >> > > for
>> >> > > > > that big a size.
>> >> > > > >
>> >> > > > > My challenge, on the other side, is that I need to update
the
>> >> > > > > indexes
>> >> > > > very
>> >> > > > > frequently. So, do you think MMapDirectory is the solution?
>> >> > > > >
>> >> > > > > Thanks.
>> >> > > > >
>> >> > > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
>> >> > > jack@basetechnology.com
>> >> > > > >wrote:
>> >> > > > >
>> >> > > > >> From the javadoc for RAMDirectory:
>> >> > > > >>
>> >> > > > >> "Warning: This class is not intended to work with
huge indexes.
>> >> > > > Everything
>> >> > > > >> beyond several hundred megabytes will waste resources
(GC
>> >> > > > >> cycles),
>> >> > > > because
>> >> > > > >> it uses an internal buffer size of 1024 bytes, producing
>> millions
>> >> > > > >> of byte[1024] arrays. This class is optimized for
small
>> >> > > > >> memory-resident indexes. It also has bad concurrency
on
>> >> multithreaded
>> >> > environments.
>> >> > > > >>
>> >> > > > >> It is recommended to materialize large indexes on
disk and use
>> >> > > > >> MMapDirectory, which is a high-performance directory
>> >> > > > >> implementation
>> >> > > > working
>> >> > > > >> directly on the file system cache of the operating
system, so
>> >> > > > >> copying
>> >> > > > data
>> >> > > > >> to Java heap space is not useful."
>> >> > > > >>
>> >> > > > >> -- Jack Krupansky
>> >> > > > >>
>> >> > > > >> -----Original Message----- From: Cheng
>> >> > > > >> Sent: Monday, June 04, 2012 10:08 AM
>> >> > > > >> To: java-user@lucene.apache.org
>> >> > > > >> Subject: RAMDirectory unexpectedly slows
>> >> > > > >>
>> >> > > > >>
>> >> > > > >> Hi,
>> >> > > > >>
>> >> > > > >> My apps need to read from and write to some big
indexes
>> >> frequently.
>> >> > > So I
>> >> > > > >> use RAMDirectory instead of FSDirectory, and give
JVM about 2GB
>> >> > > > >> memory size.
>> >> > > > >>
>> >> > > > >> I notice that the speed of reading and writing unexpectedly
>> slows
>> >> > > > >> as
>> >> > > the
>> >> > > > >> size of the indexes increases. Since the usage of
RAM is less
>> >> > > > >> than
>> >> > > 20%,
>> >> > > > I
>> >> > > > >> think by default the RAMDirectory doesn't take advantage
of the
>> >> > > memory I
>> >> > > > >> assigned to JVM.
>> >> > > > >>
>> >> > > > >> What are the steps to improve the reading and writing
speed of
>> >> > > > >> RAMDirectory?
>> >> > > > >>
>> >> > > > >> Thanks!
>> >> > > > >> Jeff
>> >> > > > >>
>> >> > > > >>
>> >> > > >_____________________________________________
>> >> > > **_____________________________________________
>> >> > > **---------
>> >> > > > >> To unsubscribe, e-mail:
>> >> > > > >> java-user-unsubscribe@lucene.**apache.org<
>> >> > > > java-user-unsubscribe@lucene.apache.org>
>> >> > > > >> For additional commands, e-mail:
>> >> > > > >> java-user-help@lucene.apache.**org<
>> >> > > > java-user-help@lucene.apache.org>
>> >> > > > >>
>> >> > > > >>
>> >> > > >
>> >> > > >_____________________________________________
>> >> > >
>> >> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >> > > >
>> >> > > >
>> >> > >
>> >> > >
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message