lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rutherglen <jason.rutherg...@gmail.com>
Subject Re: RAMDirectory unexpectedly slows
Date Mon, 04 Jun 2012 14:58:59 GMT
If you want the index to be stored completely in RAM, there is the
ByteBuffer directory [1].  Though I do not see the point in putting an
index in RAM, it will be cached in RAM regardless in the OS system IO
cache.

1. https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java

On Mon, Jun 4, 2012 at 10:55 AM, Cheng <zhoucheng2008@gmail.com> wrote:
> My indexes are 500MB+. So it seems like that RAMDirectory is not good for
> that big a size.
>
> My challenge, on the other side, is that I need to update the indexes very
> frequently. So, do you think  MMapDirectory is the solution?
>
> Thanks.
>
> On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <jack@basetechnology.com>wrote:
>
>> From the javadoc for RAMDirectory:
>>
>> "Warning: This class is not intended to work with huge indexes. Everything
>> beyond several hundred megabytes will waste resources (GC cycles), because
>> it uses an internal buffer size of 1024 bytes, producing millions of
>> byte[1024] arrays. This class is optimized for small memory-resident
>> indexes. It also has bad concurrency on multithreaded environments.
>>
>> It is recommended to materialize large indexes on disk and use
>> MMapDirectory, which is a high-performance directory implementation working
>> directly on the file system cache of the operating system, so copying data
>> to Java heap space is not useful."
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: Cheng
>> Sent: Monday, June 04, 2012 10:08 AM
>> To: java-user@lucene.apache.org
>> Subject: RAMDirectory unexpectedly slows
>>
>>
>> Hi,
>>
>> My apps need to read from and write to some big indexes frequently. So I
>> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
>> size.
>>
>> I notice that the speed of reading and writing unexpectedly slows as the
>> size of the indexes increases. Since the usage of RAM is less than 20%, I
>> think by default the RAMDirectory doesn't take advantage of the memory I
>> assigned to JVM.
>>
>> What are the steps to improve the reading and writing speed of
>> RAMDirectory?
>>
>> Thanks!
>> Jeff
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<java-user-unsubscribe@lucene.apache.org>
>> For additional commands, e-mail: java-user-help@lucene.apache.**org<java-user-help@lucene.apache.org>
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message