lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiangyu Jin <x...@cs.virginia.edu>
Subject Re: OutOfMemoryError with Lucene 1.4 final
Date Fri, 10 Dec 2004 17:33:52 GMT

Ok, I see. Seems most ppl think is the third possiblity

On Fri, 10 Dec 2004, Xiangyu  Jin wrote:

>
> I am not sure. But guess there are three possilities,
>
> (1). see that you use
> Field.Text("contents", stringBuffer.toString())
> This will store all your string of text into document object.
> And it might be long ...
>
> I do not know the detail how Lucene implemented.
> I think you can try use unstored first to see
> if the same problem happen.
>
> BTW, how large is your document. Mine has 1M docs and
> max-length less than 1 M, usually has length about several k.
>
> (2) I guess another possiblilty is that record 898 is a very long
> document, maybe java' s string object has a maxlength?
> Just trace the code, see when the exception occur.
>
> (3) Moreover, if you run it on a java VM, it also has a setting of
> its "virtual" mem. It has nothing to do with the hardware
> you are running. I has met this before when I use the directory's
> ListOfFile function, where it easily exceed the max mem, if
> there are 1M docs under the same dir (a stupid mistake I made).
> But if I expand the VM's mem, it is then appears ok.
>
> :)
>
>
>
>
>
> On Fri, 10 Dec 2004, Jin, Ying wrote:
>
> > Hi, Everyone,
> >
> >
> >
> > We're trying to index ~1500 archives but get OutOfMemoryError about
> > halfway through the index process. I've tried to run program under two
> > different Redhat Linux servers: One with 256M memory and 365M swap
> > space. The other one with 512M memory and 1G swap space. However, both
> > got OutOfMemoryError at the same place (at record 898).
> >
> >
> >
> > Here is my code for indexing:
> >
> > ===============================================
> >
> >     Document doc = new Document();
> >
> >     doc.add(Field.UnIndexed("path", f.getPath()));
> >
> >     doc.add(Field.Keyword("modified",
> >
> >
> > DateField.timeToString(f.lastModified())));
> >
> >     doc.add(Field.UnIndexed("eprintid", id));
> >
> >     doc.add(Field.Text("metadata", metadata));
> >
> >
> >
> >     FileInputStream is = new FileInputStream(f);  // the text file
> >
> >     BufferedReader reader = new BufferedReader(new
> > InputStreamReader(is));
> >
> >
> >
> >     StringBuffer stringBuffer = new StringBuffer();
> >
> >     String line = "";
> >
> >     try{
> >
> >       while((line = reader.readLine()) != null){
> >
> >         stringBuffer.append(line);
> >
> >       }
> >
> >       doc.add(Field.Text("contents", stringBuffer.toString()));
> >
> >       // release the resources
> >
> >       is.close();
> >
> >       reader.close();
> >
> > }catch(java.io.IOException e){}
> >
> > =================================================
> >
> > Is there anything wrong with my code or I need more memory?
> >
> >
> >
> > Thanks for any help!
> >
> > Ying
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message