lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Li <fancye...@gmail.com>
Subject Re: what's better for in memory searching?
Date Mon, 11 Jun 2012 08:50:42 GMT
1. this setting is global, I just want my lucene searching program
don't swap. for other less important programs, it can still swap.
2. do I need call MappedByteBuffer.load() explicitly? or I have to
warm up the indexes to guarantee all my files are in physical memory?

On Mon, Jun 11, 2012 at 4:45 PM, Michael Kuhlmann <kuli@solarier.de> wrote:
> Set the swapiness to 0 to avoid memory pages being swapped to disk too
> early.
>
> http://en.wikipedia.org/wiki/Swappiness
>
> -Kuli
>
> Am 11.06.2012 10:38, schrieb Li Li:
>
>> I have roughly read the codes of RAMDirectory. it use a list of 1024
>> byte arrays and many overheads.
>> But as far as I know, using MMapDirectory, I can't prevent the page
>> faults. OS will swap less frequent pages out. Even if I allocate
>> enough memory for JVM, I can guarantee all the files in the directory
>> are in memory. am I understanding right? if it is, then some less
>> frequent queries will be slow.  How can I let them always in memory?
>>
>> On Fri, Jun 8, 2012 at 5:53 PM, Lance Norskog<goksron@gmail.com>  wrote:
>>>
>>> Yes, use MMapDirectory. It is faster and uses memory more efficiently
>>> than RAMDirectory. This sounds wrong, but it is true. With
>>> RAMDirectory, Java has to work harder doing garbage collection.
>>>
>>> On Fri, Jun 8, 2012 at 1:30 AM, Li Li<fancyerii@gmail.com>  wrote:
>>>>
>>>> hi all
>>>>   I want to use lucene 3.6 providing searching service. my data is
>>>> not very large, raw data is less that 1GB and I want to use load all
>>>> indexes into memory. also I need save all indexes into disk
>>>> persistently.
>>>>   I originally want to use RAMDirectory. But when I read its javadoc.
>>>>
>>>>   Warning: This class is not intended to work with huge indexes.
>>>> Everything beyond several hundred megabytes
>>>>  will waste resources (GC cycles), because it uses an internal buffer
>>>> size of 1024 bytes, producing millions of byte
>>>>  [1024] arrays. This class is optimized for small memory-resident
>>>> indexes. It also has bad concurrency on
>>>>  multithreaded environments.
>>>> It is recommended to materialize large indexes on disk and use
>>>> MMapDirectory, which is a high-performance
>>>>  directory implementation working directly on the file system cache of
>>>> the operating system, so copying data to
>>>>  Java heap space is not useful.
>>>>
>>>>    should I use MMapDirectory? it seems another contrib instantiated.
>>>> anyone test it with RAMDirectory?
>>>
>>>
>>>
>>>
>>> --
>>> Lance Norskog
>>> goksron@gmail.com
>
>

Mime
View raw message