lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiangyu Jin <x...@cs.virginia.edu>
Subject Re: OutOfMemoryError with Lucene 1.4 final
Date Fri, 10 Dec 2004 17:30:06 GMT

I am not sure. But guess there are three possilities,

(1). see that you use
Field.Text("contents", stringBuffer.toString())
This will store all your string of text into document object.
And it might be long ...

I do not know the detail how Lucene implemented.
I think you can try use unstored first to see
if the same problem happen.

BTW, how large is your document. Mine has 1M docs and
max-length less than 1 M, usually has length about several k.

(2) I guess another possiblilty is that record 898 is a very long
document, maybe java' s string object has a maxlength?
Just trace the code, see when the exception occur.

(3) Moreover, if you run it on a java VM, it also has a setting of
its "virtual" mem. It has nothing to do with the hardware
you are running. I has met this before when I use the directory's
ListOfFile function, where it easily exceed the max mem, if
there are 1M docs under the same dir (a stupid mistake I made).
But if I expand the VM's mem, it is then appears ok.

:)





On Fri, 10 Dec 2004, Jin, Ying wrote:

> Hi, Everyone,
>
>
>
> We're trying to index ~1500 archives but get OutOfMemoryError about
> halfway through the index process. I've tried to run program under two
> different Redhat Linux servers: One with 256M memory and 365M swap
> space. The other one with 512M memory and 1G swap space. However, both
> got OutOfMemoryError at the same place (at record 898).
>
>
>
> Here is my code for indexing:
>
> ===============================================
>
>     Document doc = new Document();
>
>     doc.add(Field.UnIndexed("path", f.getPath()));
>
>     doc.add(Field.Keyword("modified",
>
>
> DateField.timeToString(f.lastModified())));
>
>     doc.add(Field.UnIndexed("eprintid", id));
>
>     doc.add(Field.Text("metadata", metadata));
>
>
>
>     FileInputStream is = new FileInputStream(f);  // the text file
>
>     BufferedReader reader = new BufferedReader(new
> InputStreamReader(is));
>
>
>
>     StringBuffer stringBuffer = new StringBuffer();
>
>     String line = "";
>
>     try{
>
>       while((line = reader.readLine()) != null){
>
>         stringBuffer.append(line);
>
>       }
>
>       doc.add(Field.Text("contents", stringBuffer.toString()));
>
>       // release the resources
>
>       is.close();
>
>       reader.close();
>
> }catch(java.io.IOException e){}
>
> =================================================
>
> Is there anything wrong with my code or I need more memory?
>
>
>
> Thanks for any help!
>
> Ying
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message