lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ganesh" <emailg...@yahoo.co.in>
Subject Re: IndexDivisor
Date Thu, 03 Dec 2009 12:15:25 GMT
I don't have deletions. No norms. No sorting. 
I am setting 70 MB for IndexWriter RamBuffer but not indexing any documents. Swtiched off
indexing.
I have enabled TermVector for one field. 
I am opening all 30 database with different value of indexdivisor.

Below are my stats
IndexDivisor    Memory
    -1                7 MB
    1                  486 MB
    100              180 MB
    1000            176 MB.

1. Whether term vector will consume RAM. I think it should only consume disk space.

2. By setting RAM buffer for IW, Does it will allocate memory Or based on the usage the memory
will be allocated / increased. 

Regards
Ganesh

----- Original Message ----- 
From: "Michael McCandless" <lucene@mikemccandless.com>
To: <java-user@lucene.apache.org>
Sent: Thursday, December 03, 2009 4:47 PM
Subject: Re: IndexDivisor


How do you measure memory consumption?

If you pass -1 for the divisor do you still see no difference?

Can you post the output of CheckIndex on your index?

Are you sure your index has no deletions?

Mike

On Thu, Dec 3, 2009 at 5:55 AM, Ganesh <emailgane@yahoo.co.in> wrote:
> I don't have norms, I don't delete docs using IndexReader. I have switched off sorting.
I think fieldcache is used for sorting.
> I am just loading all the index with different values and calculating the memory difference.
>
> I am having 100 million records splitted across database. I loaded all the database with
index divisor value 1000, 5000, and 10000 and the memory consumption seems to be same.
>
> Regards
> Ganesh
>
> ----- Original Message -----
> From: "Michael McCandless" <lucene@mikemccandless.com>
> To: <java-user@lucene.apache.org>
> Sent: Thursday, December 03, 2009 4:09 PM
> Subject: Re: IndexDivisor
>
>
> That's indeed how index divisor works, so, the memory difference
> should be 100X lower, for just the RAM consumed by the terms index.
>
> Other things consume RAM (norms, deleted docs, field cache), so maybe
> those are messing up your measurements?
>
> Mike
>
> On Fri, Nov 27, 2009 at 5:30 AM, Danil Ε’ORIN <torindan@gmail.com> wrote:
>> Try to open with very large value (MAX_INT) it will load only first
>> term, and look up the rest from disk.
>>
>> On Fri, Nov 27, 2009 at 12:24, Michael McCandless
>> <lucene@mikemccandless.com> wrote:
>>> If you are absolutely certain you won't be doing any lookups by term.
>>>
>>> The only use case I know of is internal, when Lucene's SegmentMerger
>>> is merging the segment with other segments. In this case, the merger
>>> does a linear iteration of all terms, and never a lookup by term, so
>>> we save CPU/RAM by not loading the terms index.
>>>
>>> Mike
>>>
>>> On Fri, Nov 27, 2009 at 5:16 AM, Ganesh <emailgane@yahoo.co.in> wrote:
>>>> Thanks,
>>>>
>>>> May i know the purpose of using negative value?
>>>>
>>>> Regards
>>>> Ganesh
>>>>
>>>> ----- Original Message -----
>>>> From: "Michael McCandless" <lucene@mikemccandless.com>
>>>> To: <java-user@lucene.apache.org>
>>>> Sent: Friday, November 27, 2009 3:17 PM
>>>> Subject: Re: IndexDivisor
>>>>
>>>>
>>>>> This is the expected behavior.
>>>>>
>>>>> If you intend to use the reader for searching, looking doc freq,
>>>>> deleting docs, etc, you must pass a non-negative value for
>>>>> indexDivisor.
>>>>>
>>>>> Mike
>>>>>
>>>>> On Fri, Nov 27, 2009 at 12:00 AM, Ganesh <emailgane@yahoo.co.in>
wrote:
>>>>>> Hello all,
>>>>>>
>>>>>> I am using Lucene v2.9.1, If I open my reader with positive value
for termInfosIndexDivisor then the search works fine. If i set to -1, then search throws exception
"terms index was not loaded when this reader was created". Is this the intend behaviour?
>>>>>>
>>>>>> I thought by setting this value to -1, No memory will be used and
the terms will be fetched from the disk. Search might be slow but with very less memory consumption.
>>>>>>
>>>>>> Regards
>>>>>> Ganesh
>>>>>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Send instant messages to your online friends http://in.messenger.yahoo.com 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message