lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: IndexDivisor
Date Tue, 08 Dec 2009 13:46:28 GMT
Visiting all the index terms, which must be done for any divisor !=
-1, generates a good amount of garbage.  So if you're including that
garbage in your memory measurement, that would explain what you're
seeing, and switching to a memory profiler should show the true RAM
usage.

Mike

On Tue, Dec 8, 2009 at 6:29 AM, Ganesh <emailgane@yahoo.co.in> wrote:
> Thanks Mike...
>
> As i explained before, I created a small app, which loads all the db, does term search
(using term query) and calculates the memory consumption. I tried this with divisor value
but after 100 there seems to be no difference.
>
> Just load the database with different divisor value and then perform warming using TermQuery.
I feel there should be some difference when opening with different value. But it didn't. I
will try to perform the same using profiler.
>
> Regards
> Ganesh
>
>
> ----- Original Message -----
> From: "Michael McCandless" <lucene@mikemccandless.com>
> To: <java-user@lucene.apache.org>
> Sent: Sunday, December 06, 2009 3:14 PM
> Subject: Re: IndexDivisor
>
>
> You really have to use a profiler, to trust the RAM measurements.
>
> Certain queries, eg function queries, can load the field cache.  What
> kind of queries are you running?
>
> Mike
>
> On Fri, Dec 4, 2009 at 9:11 AM, Ganesh <emailgane@yahoo.co.in> wrote:
>> I didn't run with profiler. I created a test app and run that.. I am opening multiple
database.
>>
>> IndexReader opened with IndexDivisor: 100 //Open the reader with the divisor value
>> TermCount: 7046764 //Available unique terms in the db
>> Warmup done: //warming up
>> Memory: UsedMemory: 4 :FreeMemory:788 //Memory consumed after GC.
>>
>> I couldn't see the memory difference for the value btw 100 to 10000. I will try to
run with profiler and get back to you.
>>
>> Other than terms what all the other things which consumes memory. There is no deletions,
no norms and i am not doing sorting
>>
>> Regards
>> Ganesh
>>
>>
>> ----- Original Message -----
>> From: "Michael McCandless" <lucene@mikemccandless.com>
>> To: <java-user@lucene.apache.org>
>> Sent: Friday, December 04, 2009 7:01 PM
>> Subject: Re: IndexDivisor
>>
>>
>> I'm confused -- what are these attachments? Output from a memory profiler?
>>
>> Can you post the app you created?
>>
>> Mike
>>
>> On Fri, Dec 4, 2009 at 12:24 AM, Ganesh <emailgane@yahoo.co.in> wrote:
>>> Thanks mike..
>>>
>>> Please find the attached file. I ran the testing for 1,100,1000,10000 divisor
value. There is difference from 1 to 100 but there is no difference between 100 to 10000.
>>>
>>> I created a new application, in which i opened all reader and searcher and warmed
up. Sleep for a minute and close all the searcher and reader.
>>>
>>> There is no deletions, no norms and i am not doing sorting.
>>>
>>> Regards
>>> Ganesh
>>>
>>>
>>>
>>> ----- Original Message -----
>>> From: "Michael McCandless" <lucene@mikemccandless.com>
>>> To: <java-user@lucene.apache.org>
>>> Sent: Thursday, December 03, 2009 9:35 PM
>>> Subject: Re: IndexDivisor
>>>
>>>
>>> Can you run w/ a memory profiler? I don't trust that gc is truly running.
>>>
>>> Mike
>>>
>>> On Thu, Dec 3, 2009 at 10:47 AM, Ganesh <emailgane@yahoo.co.in> wrote:
>>>> I am doing GC before calculating the memory. Even i set my indexdivisor to
10000 but there seems to be no change.
>>>>
>>>> Below are my stats
>>>> IndexDivisor Memory
>>>> -1 7 MB
>>>> 1 486 MB
>>>> 100 180 MB
>>>> 1000 176 MB.
>>>> 10000 176MB
>>>>
>>>> Regards
>>>> Ganesh
>>>>
>>>> ----- Original Message -----
>>>> From: "Danil ŢORIN" <torindan@gmail.com>
>>>> To: <java-user@lucene.apache.org>
>>>> Sent: Thursday, December 03, 2009 8:35 PM
>>>> Subject: Re: IndexDivisor
>>>>
>>>>
>>>> Run System.gc() exactly before measuring memory usage.
>>>>
>>>> On sun jvm it will FORCE gc (unless DisableExplicitGC is used).
>>>>
>>>>
>>>> On Thu, Dec 3, 2009 at 16:30, Ganesh <emailgane@yahoo.co.in> wrote:
>>>>> Thanks mike.
>>>>>
>>>>> I am opening the reader and warming it up and then calculating the memory
consumed.
>>>>> long usedMemory = runtime.totalMemory() - runtime.freeMemory();
>>>>>
>>>>> Regards
>>>>> Ganesh
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Michael McCandless" <lucene@mikemccandless.com>
>>>>> To: <java-user@lucene.apache.org>
>>>>> Sent: Thursday, December 03, 2009 6:22 PM
>>>>> Subject: Re: IndexDivisor
>>>>>
>>>>>
>>>>> On Thu, Dec 3, 2009 at 7:15 AM, Ganesh <emailgane@yahoo.co.in>
wrote:
>>>>>
>>>>>> Below are my stats
>>>>>> IndexDivisor Memory
>>>>>> -1 7 MB
>>>>>> 1 486 MB
>>>>>> 100 180 MB
>>>>>> 1000 176 MB.
>>>>>
>>>>> Do you simply create the IndexWriter & IndexReader, but do no
>>>>> searching/indexing?
>>>>>
>>>>> How are you measuring RAM? You should use a RAM profiler (eg YourKit)
>>>>> to get the "real" usage, not counting garbage that's yet to be
>>>>> collected.
>>>>>
>>>>> My guess is the shear process of Lucene scanning your terms index
>>>>> creates lots of garbage and you're measuring the RAM consumed by that
>>>>> garbage. The garbage should be harmless (it'll eventually get GCd &
>>>>> reused).
>>>>>
>>>>>> 1. Whether term vector will consume RAM. I think it should only consume
disk space.
>>>>>
>>>>> No RAM is consumed.
>>>>>
>>>>>> 2. By setting RAM buffer for IW, Does it will allocate memory Or
based on the usage the memory will be allocated / increased.
>>>>>
>>>>> Only a little RAM is allocated up front, I think. It's only as you
>>>>> start indexing docs that the big RAM is really allocated. After a
>>>>> flush the RAM is then reused (ie, not freed) for subsequent segments.
>>>>>
>>>>> Mike
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message