lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: IndexDivisor
Date Thu, 03 Dec 2009 16:05:51 GMT
Can you run w/ a memory profiler?  I don't trust that gc is truly running.

Mike

On Thu, Dec 3, 2009 at 10:47 AM, Ganesh <emailgane@yahoo.co.in> wrote:
> I am doing GC before calculating the memory. Even i set my indexdivisor to 10000 but
there seems to be no change.
>
> Below are my stats
>  IndexDivisor Memory
>  -1         7 MB
>  1         486 MB
>  100     180 MB
>  1000     176 MB.
> 10000    176MB
>
> Regards
> Ganesh
>
> ----- Original Message -----
> From: "Danil ŢORIN" <torindan@gmail.com>
> To: <java-user@lucene.apache.org>
> Sent: Thursday, December 03, 2009 8:35 PM
> Subject: Re: IndexDivisor
>
>
> Run System.gc() exactly before measuring memory usage.
>
> On sun jvm it will FORCE gc (unless DisableExplicitGC is used).
>
>
> On Thu, Dec 3, 2009 at 16:30, Ganesh <emailgane@yahoo.co.in> wrote:
>> Thanks mike.
>>
>> I am opening the reader and warming it up and then calculating the memory consumed.
>> long usedMemory = runtime.totalMemory() - runtime.freeMemory();
>>
>> Regards
>> Ganesh
>>
>> ----- Original Message -----
>> From: "Michael McCandless" <lucene@mikemccandless.com>
>> To: <java-user@lucene.apache.org>
>> Sent: Thursday, December 03, 2009 6:22 PM
>> Subject: Re: IndexDivisor
>>
>>
>> On Thu, Dec 3, 2009 at 7:15 AM, Ganesh <emailgane@yahoo.co.in> wrote:
>>
>>> Below are my stats
>>> IndexDivisor Memory
>>> -1 7 MB
>>> 1 486 MB
>>> 100 180 MB
>>> 1000 176 MB.
>>
>> Do you simply create the IndexWriter & IndexReader, but do no
>> searching/indexing?
>>
>> How are you measuring RAM? You should use a RAM profiler (eg YourKit)
>> to get the "real" usage, not counting garbage that's yet to be
>> collected.
>>
>> My guess is the shear process of Lucene scanning your terms index
>> creates lots of garbage and you're measuring the RAM consumed by that
>> garbage. The garbage should be harmless (it'll eventually get GCd &
>> reused).
>>
>>> 1. Whether term vector will consume RAM. I think it should only consume disk
space.
>>
>> No RAM is consumed.
>>
>>> 2. By setting RAM buffer for IW, Does it will allocate memory Or based on the
usage the memory will be allocated / increased.
>>
>> Only a little RAM is allocated up front, I think. It's only as you
>> start indexing docs that the big RAM is really allocated. After a
>> flush the RAM is then reused (ie, not freed) for subsequent segments.
>>
>> Mike
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message