lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: IndexDivisor
Date Sun, 06 Dec 2009 09:44:22 GMT
You really have to use a profiler, to trust the RAM measurements.

Certain queries, eg function queries, can load the field cache.  What
kind of queries are you running?

Mike

On Fri, Dec 4, 2009 at 9:11 AM, Ganesh <emailgane@yahoo.co.in> wrote:
> I didn't run with profiler. I created a test app and run that.. I am opening multiple
database.
>
> IndexReader opened with IndexDivisor: 100 //Open the reader with the divisor value
> TermCount: 7046764                                   //Available unique
terms in the db
> Warmup done:                                             //warming
up
> Memory: UsedMemory: 4 :FreeMemory:788 //Memory consumed after GC.
>
> I couldn't see the memory difference for the value btw 100 to 10000. I will try to run
with profiler and get back to you.
>
> Other than terms what all the other things which consumes memory.  There is no deletions,
no norms and i am not doing sorting
>
> Regards
> Ganesh
>
>
> ----- Original Message -----
> From: "Michael McCandless" <lucene@mikemccandless.com>
> To: <java-user@lucene.apache.org>
> Sent: Friday, December 04, 2009 7:01 PM
> Subject: Re: IndexDivisor
>
>
> I'm confused -- what are these attachments?  Output from a memory profiler?
>
> Can you post the app you created?
>
> Mike
>
> On Fri, Dec 4, 2009 at 12:24 AM, Ganesh <emailgane@yahoo.co.in> wrote:
>> Thanks mike..
>>
>> Please find the attached file. I ran the testing for 1,100,1000,10000 divisor value.
There is difference from 1 to 100 but there is no difference between 100 to 10000.
>>
>> I created a new application, in which i opened all reader and searcher and warmed
up. Sleep for a minute and close all the searcher and reader.
>>
>> There is no deletions, no norms and i am not doing sorting.
>>
>> Regards
>> Ganesh
>>
>>
>>
>> ----- Original Message -----
>> From: "Michael McCandless" <lucene@mikemccandless.com>
>> To: <java-user@lucene.apache.org>
>> Sent: Thursday, December 03, 2009 9:35 PM
>> Subject: Re: IndexDivisor
>>
>>
>> Can you run w/ a memory profiler? I don't trust that gc is truly running.
>>
>> Mike
>>
>> On Thu, Dec 3, 2009 at 10:47 AM, Ganesh <emailgane@yahoo.co.in> wrote:
>>> I am doing GC before calculating the memory. Even i set my indexdivisor to 10000
but there seems to be no change.
>>>
>>> Below are my stats
>>> IndexDivisor Memory
>>> -1 7 MB
>>> 1 486 MB
>>> 100 180 MB
>>> 1000 176 MB.
>>> 10000 176MB
>>>
>>> Regards
>>> Ganesh
>>>
>>> ----- Original Message -----
>>> From: "Danil ŢORIN" <torindan@gmail.com>
>>> To: <java-user@lucene.apache.org>
>>> Sent: Thursday, December 03, 2009 8:35 PM
>>> Subject: Re: IndexDivisor
>>>
>>>
>>> Run System.gc() exactly before measuring memory usage.
>>>
>>> On sun jvm it will FORCE gc (unless DisableExplicitGC is used).
>>>
>>>
>>> On Thu, Dec 3, 2009 at 16:30, Ganesh <emailgane@yahoo.co.in> wrote:
>>>> Thanks mike.
>>>>
>>>> I am opening the reader and warming it up and then calculating the memory
consumed.
>>>> long usedMemory = runtime.totalMemory() - runtime.freeMemory();
>>>>
>>>> Regards
>>>> Ganesh
>>>>
>>>> ----- Original Message -----
>>>> From: "Michael McCandless" <lucene@mikemccandless.com>
>>>> To: <java-user@lucene.apache.org>
>>>> Sent: Thursday, December 03, 2009 6:22 PM
>>>> Subject: Re: IndexDivisor
>>>>
>>>>
>>>> On Thu, Dec 3, 2009 at 7:15 AM, Ganesh <emailgane@yahoo.co.in> wrote:
>>>>
>>>>> Below are my stats
>>>>> IndexDivisor Memory
>>>>> -1 7 MB
>>>>> 1 486 MB
>>>>> 100 180 MB
>>>>> 1000 176 MB.
>>>>
>>>> Do you simply create the IndexWriter & IndexReader, but do no
>>>> searching/indexing?
>>>>
>>>> How are you measuring RAM? You should use a RAM profiler (eg YourKit)
>>>> to get the "real" usage, not counting garbage that's yet to be
>>>> collected.
>>>>
>>>> My guess is the shear process of Lucene scanning your terms index
>>>> creates lots of garbage and you're measuring the RAM consumed by that
>>>> garbage. The garbage should be harmless (it'll eventually get GCd &
>>>> reused).
>>>>
>>>>> 1. Whether term vector will consume RAM. I think it should only consume
disk space.
>>>>
>>>> No RAM is consumed.
>>>>
>>>>> 2. By setting RAM buffer for IW, Does it will allocate memory Or based
on the usage the memory will be allocated / increased.
>>>>
>>>> Only a little RAM is allocated up front, I think. It's only as you
>>>> start indexing docs that the big RAM is really allocated. After a
>>>> flush the RAM is then reused (ie, not freed) for subsequent segments.
>>>>
>>>> Mike
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message