lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ganesh" <emailg...@yahoo.co.in>
Subject Re: setTermInfosIndexDivisor
Date Thu, 25 Jun 2009 09:40:33 GMT
I am updating status of the documents frequently. There will be huge number of deletes. I do
optimize the index once in a day.

I want to know the usage for setTermInfosIndexDivisor. 

Directory dir = FSDirectory.getDirectory(indexPath);
IndexReader reader = IndexReader.open(dir, true);
reader.setTermInfosIndexDivisor(5);

I reopen the IndexReader whenever there is any document added to Index.  Do i need to set
setTermInfosIndexDivisor(5); during re-opening of the index also. I tried this, first time
it accepted and second time onwards it throws "terms already loaded" expection. 

>Loaded terms might not dominate your memory consumption in side
>lucene. Again, you should provide more information of indexing, the
>environment and the situation where the error occurs.
I do indexing with no norms with all default values. 
As per the documentation, it should subsample the terms loaded in to memory. 

Regards
Ganesh
 


----- Original Message ----- 
From: "Simon Willnauer" <simon.willnauer@googlemail.com>
To: <java-user@lucene.apache.org>
Sent: Thursday, June 25, 2009 2:10 PM
Subject: Re: setTermInfosIndexDivisor


Hey there,

On Thu, Jun 25, 2009 at 9:10 AM, Ganesh<emailgane@yahoo.co.in> wrote:
> Hello all,
>
> I am using Lucene v2.4.1
>
> 1)
> I have build multiple indexes of total 30 million documents. My memory limit is 512 MB.
In this case i am getting frequently OOME. If i increased the memory limit to 1 GB / 1.5 GB
then it is working fine. My point is it will also will get exhausted when it reaches 60 /
90 million documents.
>
> I thought to use setTermInfosIndexDivisor, but even then the memory consumption is same.
This parameter has no effect. Whether this parameter should be set while building index? I
build the index using default value. After hitting OOME i am setting this.
I would be curious what you do to your index. do you have a lot of
pending deletes? do you call optimize frequently? In which situations
do you hit the OOM?
>
> Directory dir = FSDirectory.getDirectory(indexPath);
> IndexReader reader = IndexReader.open(dir, true);
> reader.setTermInfosIndexDivisor(5);
Loaded terms might not dominate your memory consumption in side
lucene. Again, you should provide more information of indexing, the
environment and the situation where the error occurs.

simon
>
> 2)
> IndexWriter.setTermIndexInterval should be set while creating the index? If i build the
index with default value, After some time if i use this parameter, Whether there will be some
effect?
>
> Regards
> Ganesh
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Send instant messages to your online friends http://in.messenger.yahoo.com 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message