Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.co.in;
  h=Received:X-Yahoo-SMTP:X-YMail-OSG:X-Yahoo-Newman-Property:Message-ID:From:To:References:Subject:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:X-Priority:X-MSMail-Priority:X-Mailer:X-MimeOLE;
  b=HSkCwNa7v0qevGu12e89ZKtV/ECVF2ywBIS8azpzIAmdCoEG7HORnkZdF2uUgAtAews3PWL+U693N4XB44NDJKgiSa4KxhQz89JDgTyf1QRnhk1Ag5MfndsKSgEw1ZQIrdgBurq8pTsHkfwFFGTEnbv+fMUGEZ+ObWFAJgc659w=
  ;
Message-ID: <21aa01c9f57c$fdf82e20$710bc30a@sv.us.sonicwall.com>
From: "Ganesh" <emailgane@yahoo.co.in>
To: <java-user@lucene.apache.org>
References: <20e801c9f564$13c21130$710bc30a@sv.us.sonicwall.com>
 <f18c9dde0906250140m44635ae3g91f20d05755493a8@mail.gmail.com>
 <9ac0c6aa0906250226j383db2d6oda02dc93658f09fe@mail.gmail.com>
Subject: Re: setTermInfosIndexDivisor
Date: Thu, 25 Jun 2009 15:39:11 +0530
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


What about setTermInfosIndexDivisor??=20

Directory dir =3D FSDirectory.getDirectory(indexPath);
IndexReader reader =3D IndexReader.open(dir, true);
reader.setTermInfosIndexDivisor(5);

It supposed to load only one fifth of the terms available?? But there is =
no difference in memory consumption with / without settings this =
parameter.

I reopen the IndexReader whenever there is any document added to Index.  =
Do i need to set setTermInfosIndexDivisor(5); during re-opening of the =
index also. I tried this, first time it accepted and second time onwards =
it throws "terms already loaded" expection

Regards
Ganesh

----- Original Message -----=20
From: "Michael McCandless" <lucene@mikemccandless.com>
To: <java-user@lucene.apache.org>; <simon.willnauer@gmail.com>
Sent: Thursday, June 25, 2009 2:56 PM
Subject: Re: setTermInfosIndexDivisor


setTermIndexInterval only helps appreciably when an index has a truly
immense number of terms (often, "by accident" eg your document
filtering/analysis process accidentally allowed binary terms into the
index); it's meant primarily as a "safety" for such situations.

If you run CheckIndex, it prints the number of unique terms per segment.

The other big things that use RAM while searching are 1) deleted docs
(do you have any deletions?), 2) norms (have you disabled norms for
fields that don't actually require it), and 3) FieldCache (used when
you sort by field instead of relevance).

Mike

On Thu, Jun 25, 2009 at 4:40 AM, Simon
Willnauer<simon.willnauer@googlemail.com> wrote:
> Hey there,
>
> On Thu, Jun 25, 2009 at 9:10 AM, Ganesh<emailgane@yahoo.co.in> wrote:
>> Hello all,
>>
>> I am using Lucene v2.4.1
>>
>> 1)
>> I have build multiple indexes of total 30 million documents. My =
memory limit is 512 MB. In this case i am getting frequently OOME. If i =
increased the memory limit to 1 GB / 1.5 GB then it is working fine. My =
point is it will also will get exhausted when it reaches 60 / 90 million =
documents.
>>
>> I thought to use setTermInfosIndexDivisor, but even then the memory =
consumption is same. This parameter has no effect. Whether this =
parameter should be set while building index? I build the index using =
default value. After hitting OOME i am setting this.
> I would be curious what you do to your index. do you have a lot of
> pending deletes? do you call optimize frequently? In which situations
> do you hit the OOM?
>>
>> Directory dir =3D FSDirectory.getDirectory(indexPath);
>> IndexReader reader =3D IndexReader.open(dir, true);
>> reader.setTermInfosIndexDivisor(5);
> Loaded terms might not dominate your memory consumption in side
> lucene. Again, you should provide more information of indexing, the
> environment and the situation where the error occurs.
>
> simon
>>
>> 2)
>> IndexWriter.setTermIndexInterval should be set while creating the =
index? If i build the index with default value, After some time if i use =
this parameter, Whether there will be some effect?
>>
>> Regards
>> Ganesh
>>
>> Send instant messages to your online friends =
http://in.messenger.yahoo.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Send instant messages to your online friends http://in.messenger.yahoo.com 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org