lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Silveira" <paulo.silve...@caelum.com.br>
Subject Re: Lucene and Tomcat, too many open files
Date Thu, 16 Mar 2006 22:43:56 GMT
Nick

it is a guess, but the only difference between my approach and yours
is that I am optimizing as soon as I open the writer, and you are
optimizing after the last (100th) document is written.

At the same time I am using:

		writer.setUseCompoundFile(true);
		writer.mergeFactor = 10; (I really wonder why this is public)

I am also running it on linux, and this was the only way I solved the
problem (I didnt even touch the ulimit).

try this exact combination (compound file, optimizing every 100
documents before writting them, mergeFactor). please report back.

paulo

On 3/16/06, Hannes Carl Meyer <developers@rc.ag> wrote:
> Hi Nick,
>
> use 'ulimit' on your ix system to check if its set to unlimited.
>
> check:
> http://wwwcgi.rdg.ac.uk:8081/cgi-bin/cgiwrap/wsi14/poplog/man/2/ulimit
>
> You don't have to set it to unlimited, maybe increasing the number will
> help.
>
> later
>
> Hannes
>
> Nick Atkins schrieb:
> > Thanks Otis, I tried that but I still get the same problem at the ulimit
> > -n point.  I assume you meant I should call
> > IndexWriter.setUseCompoundFile(true).  According to the docs compound
> > structure is the default anyway.
> >
> > Any further thoughts?  Anything I can tweak in the OS (Linux), Java
> > (1.5.0) or Lucene (1.9.1)?
> >
> > Many thanks,
> >
> > Nick
> >
> > Otis Gospodnetic wrote:
> >
> >> The easiest first step to try is to go from multi-file index structure to the
compound one.
> >>
> >> Otis
> >>
> >> ----- Original Message ----
> >> From: Nick Atkins <Nick.Atkins@scalix.com>
> >> To: java-user@lucene.apache.org
> >> Sent: Thursday, March 16, 2006 3:00:59 PM
> >> Subject: Lucene and Tomcat, too many open files
> >>
> >> Hi,
> >>
> >> What's the best way to manage the number of open files used by Lucene
> >> when it's running under Tomcat?  I have a indexing application running
> >> as a web app and I index a huge number of mail messages (upwards of
> >> 40000 in some cases).  Lucene's merging routine always craps out
> >> eventually with the "too many open files" regardless of how large I set
> >> ulimit to.  lsof tells me they are all "deleted" but they still seem to
> >> count as open files.  I don't want to set ulimit to some enormous value
> >> just to solve this (because it will never be large enough).  What's the
> >> best strategy here?
> >>
> >> I have tried setting various parameters on the IndexWriter such as the
> >> MergeFactor, MaxMergeDocs and MaxBufferedDocs but they seem to only
> >> affect the merge timing algorithm wrt memory usage.  The number of files
> >> used seems to be unaffected by anything I can set on the IndexWriter.
> >>
> >> Any hints much appreciated.
> >>
> >> Cheers,
> >>
> >> Nick.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


--
Paulo E. A. Silveira
Caelum Ensino e Soluções em Java
http://www.caelum.com.br/

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message