lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: Running OutOfMemory while optimizing and searching
Date Tue, 29 Jun 2004 05:01:34 GMT
Mark,

Tough situation.  I hate when things like this happen on production :(.
 You are not mentioning what you are using for various IndexWriter
parameters.  You may be able to get this working by tweaking them (see
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexWriter.html#field_summary).
Hm, now that I think about it, I am not sure if those are considered
during index optimization.  I'll try checking the sources later.

Otis

--- Mark Florence <mflorence@airsmail.com> wrote:
> Hi, I'm using Lucene to index ~3.5M documents, over about 50 fields.
> The Lucene
> index itself is ~10.5GB, spread over ~7,000 files. Some of these
> files are
> "large" -- that is, several PRX files are ~1.5GB.
> 
> Lucene runs on a dedicated server (Linux on a 1Ghz Dell, with 1GB
> RAM). Clients
> on other machines use RMI to perform reads / writes. Each night the
> server
> automatically performs an optimize.
> 
> The problem is that the optimize now dies with an OutOfMemory
> exception, even
> when the JVM heap size is set to its maximum of 2GB. I need to
> optimize, because
> as the number of Lucene files grows, search performance becomes
> unacceptable.
> 
> Search performance is also adversely affected because I've had to
> effectively
> single-thread reads and writes. I was using a simple read / write
> lock
> mechanism, allowing multiple readers to simultaneously search, but
> now more than
> 3-4 simultaneous readers will also cause an OutOfMemory condition.
> Searches can
> take as long as 30-40 seconds, and with single-threading, that's
> crippling the
> main client application.
> 
> Needless to say, the Lucene index is mission-critical, and must run
> 24/7.
> 
> I've seen other posts along this same vein, but no definite
> consensus. Is my
> problem simply inadequate hardware? Should I run on a 64-bit
> platform, where I
> can allocate a Java heap of > 2GB?
> 
> Or could there be something fundamentally "wrong" with my index? I
> should add
> that I've just spent about a week (!!) rebuilding from scratch, over
> all 3.5M
> documents.
> 
> -- Many thanks for any help! Mark Florence
> 
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message