Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.co.in;
  h=Received:X-Yahoo-SMTP:X-YMail-OSG:X-Yahoo-Newman-Property:Message-ID:From:To:References:Subject:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:X-Priority:X-MSMail-Priority:X-Mailer:X-MimeOLE;
  b=Rua7Ltf7sMVciGQIDxzMp4vnARZ5Z1EveMSZrrXwj+zyszVTOb1/tc/t/AJ+ULVlPYuXvcYEiANsBWvl57YUDZcX7MVgKftWSPPPUcYHoZ6kXv1yh6EsAqlwNxZq26oj0vRIM7hlLYuzJRunaEDSr00kIFSwG456qCwibU12HPM=
  ;
Message-ID: <001101ca77f9$bd86a380$710bc30a@sv.us.sonicwall.com>
From: "Ganesh" <emailgane@yahoo.co.in>
To: <java-user@lucene.apache.org>
References: <006801ca6f1e$8dc488d0$710bc30a@sv.us.sonicwall.com>
 <01bf01ca7412$4d8dea60$710bc30a@sv.us.sonicwall.com>
 <9ac0c6aa0912030452m660fec4byf4c9e368f24b57cf@mail.gmail.com>
 <020601ca7425$32f57700$710bc30a@sv.us.sonicwall.com>
 <2ffb6d060912030705r3708b688nf44f810c3ea931f7@mail.gmail.com>
 <023701ca742f$ed9e2610$710bc30a@sv.us.sonicwall.com>
 <9ac0c6aa0912030805m437713cclcee1d8b6f77848e9@mail.gmail.com>
 <026901ca74a2$17552360$710bc30a@sv.us.sonicwall.com>
 <9ac0c6aa0912040531s6d10b148nf6b175b8bb24e619@mail.gmail.com>
 <033001ca74eb$be5d42c0$710bc30a@sv.us.sonicwall.com>
 <9ac0c6aa0912060144m5ce66e7ew5c9d2c5a76375a2f@mail.gmail.com>
Subject: Re: IndexDivisor
Date: Tue, 8 Dec 2009 16:59:40 +0530
MIME-Version: 1.0
Content-Type: text/plain;
	charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Thanks Mike...

As i explained before, I created a small app, which loads all the db, =
does term search (using term query) and calculates the memory =
consumption. I tried this with divisor value but after 100 there seems =
to be no difference.

Just load the database with different divisor value and then perform =
warming using TermQuery. I feel there should be some difference when =
opening with different value. But it didn't. I will try to perform the =
same using profiler.

Regards
Ganesh=20


----- Original Message -----=20
From: "Michael McCandless" <lucene@mikemccandless.com>
To: <java-user@lucene.apache.org>
Sent: Sunday, December 06, 2009 3:14 PM
Subject: Re: IndexDivisor


You really have to use a profiler, to trust the RAM measurements.

Certain queries, eg function queries, can load the field cache.  What
kind of queries are you running?

Mike

On Fri, Dec 4, 2009 at 9:11 AM, Ganesh <emailgane@yahoo.co.in> wrote:
> I didn't run with profiler. I created a test app and run that.. I am =
opening multiple database.
>
> IndexReader opened with IndexDivisor: 100 //Open the reader with the =
divisor value
> TermCount: 7046764 //Available unique terms in the db
> Warmup done: //warming up
> Memory: UsedMemory: 4 :FreeMemory:788 //Memory consumed after GC.
>
> I couldn't see the memory difference for the value btw 100 to 10000. I =
will try to run with profiler and get back to you.
>
> Other than terms what all the other things which consumes memory. =
There is no deletions, no norms and i am not doing sorting
>
> Regards
> Ganesh
>
>
> ----- Original Message -----
> From: "Michael McCandless" <lucene@mikemccandless.com>
> To: <java-user@lucene.apache.org>
> Sent: Friday, December 04, 2009 7:01 PM
> Subject: Re: IndexDivisor
>
>
> I'm confused -- what are these attachments? Output from a memory =
profiler?
>
> Can you post the app you created?
>
> Mike
>
> On Fri, Dec 4, 2009 at 12:24 AM, Ganesh <emailgane@yahoo.co.in> wrote:
>> Thanks mike..
>>
>> Please find the attached file. I ran the testing for 1,100,1000,10000 =
divisor value. There is difference from 1 to 100 but there is no =
difference between 100 to 10000.
>>
>> I created a new application, in which i opened all reader and =
searcher and warmed up. Sleep for a minute and close all the searcher =
and reader.
>>
>> There is no deletions, no norms and i am not doing sorting.
>>
>> Regards
>> Ganesh
>>
>>
>>
>> ----- Original Message -----
>> From: "Michael McCandless" <lucene@mikemccandless.com>
>> To: <java-user@lucene.apache.org>
>> Sent: Thursday, December 03, 2009 9:35 PM
>> Subject: Re: IndexDivisor
>>
>>
>> Can you run w/ a memory profiler? I don't trust that gc is truly =
running.
>>
>> Mike
>>
>> On Thu, Dec 3, 2009 at 10:47 AM, Ganesh <emailgane@yahoo.co.in> =
wrote:
>>> I am doing GC before calculating the memory. Even i set my =
indexdivisor to 10000 but there seems to be no change.
>>>
>>> Below are my stats
>>> IndexDivisor Memory
>>> -1 7 MB
>>> 1 486 MB
>>> 100 180 MB
>>> 1000 176 MB.
>>> 10000 176MB
>>>
>>> Regards
>>> Ganesh
>>>
>>> ----- Original Message -----
>>> From: "Danil =C5=A2ORIN" <torindan@gmail.com>
>>> To: <java-user@lucene.apache.org>
>>> Sent: Thursday, December 03, 2009 8:35 PM
>>> Subject: Re: IndexDivisor
>>>
>>>
>>> Run System.gc() exactly before measuring memory usage.
>>>
>>> On sun jvm it will FORCE gc (unless DisableExplicitGC is used).
>>>
>>>
>>> On Thu, Dec 3, 2009 at 16:30, Ganesh <emailgane@yahoo.co.in> wrote:
>>>> Thanks mike.
>>>>
>>>> I am opening the reader and warming it up and then calculating the =
memory consumed.
>>>> long usedMemory =3D runtime.totalMemory() - runtime.freeMemory();
>>>>
>>>> Regards
>>>> Ganesh
>>>>
>>>> ----- Original Message -----
>>>> From: "Michael McCandless" <lucene@mikemccandless.com>
>>>> To: <java-user@lucene.apache.org>
>>>> Sent: Thursday, December 03, 2009 6:22 PM
>>>> Subject: Re: IndexDivisor
>>>>
>>>>
>>>> On Thu, Dec 3, 2009 at 7:15 AM, Ganesh <emailgane@yahoo.co.in> =
wrote:
>>>>
>>>>> Below are my stats
>>>>> IndexDivisor Memory
>>>>> -1 7 MB
>>>>> 1 486 MB
>>>>> 100 180 MB
>>>>> 1000 176 MB.
>>>>
>>>> Do you simply create the IndexWriter & IndexReader, but do no
>>>> searching/indexing?
>>>>
>>>> How are you measuring RAM? You should use a RAM profiler (eg =
YourKit)
>>>> to get the "real" usage, not counting garbage that's yet to be
>>>> collected.
>>>>
>>>> My guess is the shear process of Lucene scanning your terms index
>>>> creates lots of garbage and you're measuring the RAM consumed by =
that
>>>> garbage. The garbage should be harmless (it'll eventually get GCd &
>>>> reused).
>>>>
>>>>> 1. Whether term vector will consume RAM. I think it should only =
consume disk space.
>>>>
>>>> No RAM is consumed.
>>>>
>>>>> 2. By setting RAM buffer for IW, Does it will allocate memory Or =
based on the usage the memory will be allocated / increased.
>>>>
>>>> Only a little RAM is allocated up front, I think. It's only as you
>>>> start indexing docs that the big RAM is really allocated. After a
>>>> flush the RAM is then reused (ie, not freed) for subsequent =
segments.
>>>>
>>>> Mike
>>>>
>>>> =
---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>> Send instant messages to your online friends =
http://in.messenger.yahoo.com
>>>>
>>>> =
---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>
>>> =
---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>> Send instant messages to your online friends =
http://in.messenger.yahoo.com
>>>
>>> =
---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> Send instant messages to your online friends =
http://in.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Send instant messages to your online friends http://in.messenger.yahoo.com 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org