lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: Lucene and Tomcat, too many open files
Date Thu, 16 Mar 2006 22:57:41 GMT
Are you changing the default mergeFactor or other settings?  If so, how? 
  Large mergeFactors are generally a bad idea: they don't make things 
faster in the long run and they chew up file handles.

Are all searches reusing a single IndexReader?  They should.  This is 
the other most common reason folks run out of file handles: they open 
too many IndexReaders.  The exception may be thrown when merging, but 
the root cause might be something else.

Doug

Nick Atkins wrote:
> Hi,
> 
> What's the best way to manage the number of open files used by Lucene
> when it's running under Tomcat?  I have a indexing application running
> as a web app and I index a huge number of mail messages (upwards of
> 40000 in some cases).  Lucene's merging routine always craps out
> eventually with the "too many open files" regardless of how large I set
> ulimit to.  lsof tells me they are all "deleted" but they still seem to
> count as open files.  I don't want to set ulimit to some enormous value
> just to solve this (because it will never be large enough).  What's the
> best strategy here?
> 
> I have tried setting various parameters on the IndexWriter such as the
> MergeFactor, MaxMergeDocs and MaxBufferedDocs but they seem to only
> affect the merge timing algorithm wrt memory usage.  The number of files
> used seems to be unaffected by anything I can set on the IndexWriter.
> 
> Any hints much appreciated.
> 
> Cheers,
> 
> Nick.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message