lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng <zhoucheng2...@gmail.com>
Subject Re: RAMDirectory unexpectedly slows
Date Mon, 04 Jun 2012 15:04:32 GMT
Please shed more insight into the difference between JVM heap size and the
memory size used by Lucene.

What I am getting at is that no matter however much ram I give my apps,
Lucene can't utilize it. Is that right?

What about the ByteBufferDirectory? Can this specific directory utilize the
2GB memory I grant to the app?

On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
jason.rutherglen@gmail.com> wrote:

> If you want the index to be stored completely in RAM, there is the
> ByteBuffer directory [1].  Though I do not see the point in putting an
> index in RAM, it will be cached in RAM regardless in the OS system IO
> cache.
>
> 1.
> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>
> On Mon, Jun 4, 2012 at 10:55 AM, Cheng <zhoucheng2008@gmail.com> wrote:
> > My indexes are 500MB+. So it seems like that RAMDirectory is not good for
> > that big a size.
> >
> > My challenge, on the other side, is that I need to update the indexes
> very
> > frequently. So, do you think  MMapDirectory is the solution?
> >
> > Thanks.
> >
> > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <jack@basetechnology.com
> >wrote:
> >
> >> From the javadoc for RAMDirectory:
> >>
> >> "Warning: This class is not intended to work with huge indexes.
> Everything
> >> beyond several hundred megabytes will waste resources (GC cycles),
> because
> >> it uses an internal buffer size of 1024 bytes, producing millions of
> >> byte[1024] arrays. This class is optimized for small memory-resident
> >> indexes. It also has bad concurrency on multithreaded environments.
> >>
> >> It is recommended to materialize large indexes on disk and use
> >> MMapDirectory, which is a high-performance directory implementation
> working
> >> directly on the file system cache of the operating system, so copying
> data
> >> to Java heap space is not useful."
> >>
> >> -- Jack Krupansky
> >>
> >> -----Original Message----- From: Cheng
> >> Sent: Monday, June 04, 2012 10:08 AM
> >> To: java-user@lucene.apache.org
> >> Subject: RAMDirectory unexpectedly slows
> >>
> >>
> >> Hi,
> >>
> >> My apps need to read from and write to some big indexes frequently. So I
> >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
> >> size.
> >>
> >> I notice that the speed of reading and writing unexpectedly slows as the
> >> size of the indexes increases. Since the usage of RAM is less than 20%,
> I
> >> think by default the RAMDirectory doesn't take advantage of the memory I
> >> assigned to JVM.
> >>
> >> What are the steps to improve the reading and writing speed of
> >> RAMDirectory?
> >>
> >> Thanks!
> >> Jeff
> >>
> >>
> ------------------------------**------------------------------**---------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<
> java-user-unsubscribe@lucene.apache.org>
> >> For additional commands, e-mail: java-user-help@lucene.apache.**org<
> java-user-help@lucene.apache.org>
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message