lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Willnauer <simon.willna...@gmail.com>
Subject Re: "4.1 consuming more memory than 3.0.2 while Indexing"
Date Mon, 01 Apr 2013 13:39:19 GMT
can you provide some information how much ram you are setting on the
index writer config?

also how many threads are you using for indexing?

simon

On Mon, Apr 1, 2013 at 2:21 PM, Arun Kumar K <arunk786@gmail.com> wrote:
> Hi Adrien,
>
> I have seen memory usage using linux command top for RES memory & i have
> used SimpleFSDirectory for indexing (NOT FSDirectory which i guess will
> take it's default as MMapDir)  in both versions.
>
> Does 4.1 still take MMapDir as default ?
>
>
> Arun
>
>
> On Mon, Apr 1, 2013 at 5:38 PM, Adrien Grand <jpountz@gmail.com> wrote:
>
>> On Mon, Apr 1, 2013 at 1:56 PM, Arun Kumar K <arunk786@gmail.com> wrote:
>> > Hi Guys,
>>
>> Hi,
>>
>> > I have been finding out the heap space requirement for indexing and
>> > searching with 3.0.2 vs 4.1 (with BlockPostings Format).
>> >
>> > I have a 2GB index with 1 million docs with around 42 fields with 40
>> fields
>> > being random strings.
>> >
>> > I have seen that memory for search has reduced by 5X with 4.1 (with
>> > BlockPostings Format) but the memory usage during indexing with 4.1 is
>> > around 800MB~1.7 GB whereas for 3.0.2 it is 300~600MB.
>> > But indexing time is almost same with both versions.
>>
>> How did you measure memory usage ? Operating systems may report high
>> memory usage because MMapDirectory became the new default directory
>> implementation and memory-mapped files are taken into account in the
>> virtual memory of your processes. See
>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>> for more information.
>>
>> --
>> Adrien
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message