lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Trezzi Michael" <MTre...@CSAS.CZ>
Subject RE: OutOfMemoryError on addIndexes()
Date Fri, 12 Aug 2005 07:45:40 GMT
I did some more research and these are the results.
 
The OutOfMemory occurs on line 82 of class TermInfosReader.java. That and two other lines
are trying to create an array of the size that is get by the 
int indexSize = (int)indexEnum.size;
For my 29MB index this indexSize integer is 811806819. So if creating 3 arrays of this size
(line 82-84) it requires and enormous amount of memory, e.g. a model situation...a char array...char
has 2 bytes * 811806819 => 1584MB. That seems to be a little much and objects created in
those arrays Term, TermInfo and long are definately not simple chars. This way I would need
several gigabytes of memory to merge several even small (30MB) indexes. Is this the standard
way how it works, or is there a problem on my side?
 
Thanks,
 
Michael

________________________________

Od: Ian Lea [mailto:ian.lea@gmail.com]
Odesláno: st 10.8.2005 12:34
Komu: java-user@lucene.apache.org
Předmět: Re: OutOfMemoryError on addIndexes()



How much memory are you giving your programs?

 java    -Xmx<size>        set maximum Java heap size

--
Ian.

On 10/08/05, Trezzi Michael <MTrezzi@csas.cz> wrote:
> Hello,
> I have a problem and i tried everything i could think of to solve it. TO understand my
situation, i create indexes on several computers on our network and they are copied to one
server. There, once a day, they are merged into one masterIndex, which is then searched. The
problem is in merging. I use the following code:
>
> Directory[] ar = new Directory[fileList.length];
>        for(int i=0; i<fileList.length;i++) {
>            ar[i] = FSDirectory.getDirectory(fileList[i], false);
>        }
>        writer.addIndexes(ar);
>        for(int i=0; i<fileList.length;i++) {
>            ar[i].close();
>        }
>       writer.optimize();
>       writer.close();
>
> I also tried a longer way of opening every index separately and adding it document by
document. The problem is i am getting OutOfMemory errors on this. When I use the per document
way, it happens on the IndexReader.open command and only on indexes of approx 100M+ (The largest
index I have is only about 150MB) When I run it on windows machine with JDK1.5 I get the following:
>     Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds
VM limit
> On Linux I am running 1.4 and I get the message without the Array size information.
>
> I did try it also on test index that was made from 11359 files  (1,59GB) that had 120MB
and I got this error too. In my opinion 120MB index is not that big. The machine it runs on
is a Xeon 3,2GHz with 2GB of RAM, so it should be enough. Can you please help me?
>
> Thank you in advance,
>
> Michael Trezzi

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org





Mime
View raw message