Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.co.in;
  h=Received:X-Yahoo-SMTP:X-YMail-OSG:X-Yahoo-Newman-Property:Message-ID:From:To:References:Subject:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:X-Priority:X-MSMail-Priority:X-Mailer:X-MimeOLE;
  b=e+6jfkWb7Db/AFkr0OvOvf1oqhz/2Js2AtQcEQkk1SOGunILz25/XBL8l6dlg8xJ81lyx5DcDjFma0DmAyaSjJYt0htufkenjFa6McJjMqj5Gfpi9ZZ3jbc9WrCkG/Yubp2dgZj9CeiQh3RDFlTAI45HiO5T+FQYzdSeiOCR3IY=
  ;
Message-ID: <218c01c9f579$028c3200$710bc30a@sv.us.sonicwall.com>
From: "Ganesh" <emailgane@yahoo.co.in>
To: <java-user@lucene.apache.org>,
	<simon.willnauer@gmail.com>
References: <20e801c9f564$13c21130$710bc30a@sv.us.sonicwall.com>
 <f18c9dde0906250140m44635ae3g91f20d05755493a8@mail.gmail.com>
Subject: Re: setTermInfosIndexDivisor
Date: Thu, 25 Jun 2009 15:10:33 +0530
MIME-Version: 1.0
Content-Type: text/plain;
	charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I am updating status of the documents frequently. There will be huge =
number of deletes. I do optimize the index once in a day.

I want to know the usage for setTermInfosIndexDivisor.=20

Directory dir =3D FSDirectory.getDirectory(indexPath);
IndexReader reader =3D IndexReader.open(dir, true);
reader.setTermInfosIndexDivisor(5);

I reopen the IndexReader whenever there is any document added to Index.  =
Do i need to set setTermInfosIndexDivisor(5); during re-opening of the =
index also. I tried this, first time it accepted and second time onwards =
it throws "terms already loaded" expection.=20

>Loaded terms might not dominate your memory consumption in side
>lucene. Again, you should provide more information of indexing, the
>environment and the situation where the error occurs.
I do indexing with no norms with all default values.=20
As per the documentation, it should subsample the terms loaded in to =
memory.=20

Regards
Ganesh
=20


----- Original Message -----=20
From: "Simon Willnauer" <simon.willnauer@googlemail.com>
To: <java-user@lucene.apache.org>
Sent: Thursday, June 25, 2009 2:10 PM
Subject: Re: setTermInfosIndexDivisor


Hey there,

On Thu, Jun 25, 2009 at 9:10 AM, Ganesh<emailgane@yahoo.co.in> wrote:
> Hello all,
>
> I am using Lucene v2.4.1
>
> 1)
> I have build multiple indexes of total 30 million documents. My memory =
limit is 512 MB. In this case i am getting frequently OOME. If i =
increased the memory limit to 1 GB / 1.5 GB then it is working fine. My =
point is it will also will get exhausted when it reaches 60 / 90 million =
documents.
>
> I thought to use setTermInfosIndexDivisor, but even then the memory =
consumption is same. This parameter has no effect. Whether this =
parameter should be set while building index? I build the index using =
default value. After hitting OOME i am setting this.
I would be curious what you do to your index. do you have a lot of
pending deletes? do you call optimize frequently? In which situations
do you hit the OOM?
>
> Directory dir =3D FSDirectory.getDirectory(indexPath);
> IndexReader reader =3D IndexReader.open(dir, true);
> reader.setTermInfosIndexDivisor(5);
Loaded terms might not dominate your memory consumption in side
lucene. Again, you should provide more information of indexing, the
environment and the situation where the error occurs.

simon
>
> 2)
> IndexWriter.setTermIndexInterval should be set while creating the =
index? If i build the index with default value, After some time if i use =
this parameter, Whether there will be some effect?
>
> Regards
> Ganesh
>
> Send instant messages to your online friends =
http://in.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Send instant messages to your online friends http://in.messenger.yahoo.com 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org