lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sudarsan, Sithu D." <Sithu.Sudar...@fda.hhs.gov>
Subject RE: OutOfMemoryError using IndexWriter
Date Wed, 24 Jun 2009 14:44:01 GMT
 
Hi Stefan,

Generally, I run the memory monitor and see how much of it is being used. The total memory
usage by application, is what we see on the monitor. So long as it is less that 1.8GB things
are fine. 

In your case, you are assuming that the program takes only 50MB, which may be actually very
different from the actual usage. In my observations, memory usage by the application rarely
caused issues. The only thing that affected us was the Lucene index file(s). Our physical
memory is 4GB (even though JVM sees only about 1.8GB).

Sincerely,
Sithu 
-----Original Message-----
From: stefan [mailto:stefan@intermediate.de] 
Sent: Wednesday, June 24, 2009 10:23 AM
To: java-user@lucene.apache.org
Subject: AW: OutOfMemoryError using IndexWriter


Hi,

does Lucene keep the complete index in memory ?
As stated before the result index is 50MB, this would correlate with the memory footprint
required by Lucene as seen in my app:
jvm 120MB - 50MB(Lucene) - 50MB(my App) = something left
jvm 100MB - 50MB(Lucene) - 50MB(my App) = OOError

though some hint, whether this is the case, from the programming side would be appreciated
...

Stefan


-----Urspr√ľngliche Nachricht-----
Von: Sudarsan, Sithu D. [mailto:Sithu.Sudarsan@fda.hhs.gov]
Gesendet: Mi 24.06.2009 16:18
An: java-user@lucene.apache.org
Betreff: RE: OutOfMemoryError using IndexWriter
 
When the segments are merged, but not optimized. It happened at 1.8GB to our program, and
now we develop and test in Win32 but run the code on Linux, which seems to be handling atleast
upto 3GB of index. 

In fact, if the index size if beyond 1.8GB even, Luke throws Java Heap Error, if I try to
open.

Please post your results/views.


Sincerely,
Sithu 
-----Original Message-----
From: stefan [mailto:stefan@intermediate.de] 
Sent: Wednesday, June 24, 2009 10:08 AM
To: java-user@lucene.apache.org
Subject: AW: OutOfMemoryError using IndexWriter


Hi,

I do use Win32.

What do you mean by "the index file before
optimizations crosses your jvm memory usage settings (if say 512MB)" ?

Could you please further explain this ?

Stefan

-----Urspr√ľngliche Nachricht-----
Von: Sudarsan, Sithu D. [mailto:Sithu.Sudarsan@fda.hhs.gov]
Gesendet: Mi 24.06.2009 15:55
An: java-user@lucene.apache.org
Betreff: RE: OutOfMemoryError using IndexWriter
 

 Hi Stefan,

Are you using Windows 32 bit? If so, sometimes, if the index file before
optimizations crosses your jvm memory usage settings (if say 512MB),
there is a possibility of this happening. 

Increase JVM memory settings if that is the case. 


Sincerely,
Sithu D Sudarsan

Off: 301-796-2587

sithu.sudarsan@fda.hhs.gov
sdsudarsan@ualr.edu

-----Original Message-----
From: stefan [mailto:stefan@intermediate.de] 
Sent: Wednesday, June 24, 2009 4:09 AM
To: java-user@lucene.apache.org
Subject: OutOfMemoryError using IndexWriter

Hi,

I am using Lucene 2.4.1 to index a database with less than a million
records. The resulting index is about 50MB in size.
I keep getting an OutOfMemory Error if I re-use the same IndexWriter to
index the complete database. This is though 
recommended in the performance hints.
What I now do is, every 10000 Objects I close the index (and every 50
close actions optimize it) and create a new
IndexWriter to continue. This process works fine, but to me seems hardly
the recommended way to go.
I've been using jhat/jmap as well as Netbeans profiler and am fairly
sure that this is a problem related to Lucene.

Any Ideas - or post this to Jira ? Jira has quite a few OutOfMemory
postings but they all seem closed in Version 2.4.1.

Thanks,

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message