lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Bamford <Chris.Bamf...@scalix.com>
Subject RE: Lucene gobbling file descriptors
Date Tue, 01 Sep 2009 10:32:12 GMT
Hi Erick,

>>Note that for search speed reasons, you really, really want to share your
>>readers and NOT open/close for every request.
I have often wondered about this - I hope you can help me understand it better in the context
of our app, which is an email client:

When one of our users receives email we index and store it so he (and only he) can search
on it.  This means a separate index per user.  On large customer sites this can mean hundreds/thousands
of indexes.  Sharing readers seems counter-intuitive, unless I am missing something.  What
we do instead is that once a user performs a search, we keep his IndexReader open in case
he searches again.  At present, we have no expiry on this mechanism, so they stay open indefinitely.
 I'm a bit hazy on the underlying details but we have observed that the number of open fds
jumps by around 10 each time a new user performs a search.  What would be a good strategy
for managing this in your opinon?  Does it really make sense to keep the IndexReader open?
 Would performance suffer that much if we did an open/close for each search?  Or would it
perhaps be better to close open readers after a period of inactivity?

Thanks for any wisdom / thoughts/ ideas.

- Chris
 


----- Original Message -----
From: Erick Erickson <erickerickson@gmail.com>
Sent: Thu, 27/8/2009 4:49pm
To: java-user@lucene.apache.org
Subject: Re: Lucene gobbling file descriptors

Note that for search speed reasons, you really, really want to share your
readers and NOT open/close for every request.
FWIW
Erick

On Thu, Aug 27, 2009 at 9:10 AM, Chris Bamford <Chris.Bamford@scalix.com>wrote:

> I'm glad its not normal.  That means we can fix it!  I will conduct a
> review of IndexReader/Searcher open/close ops.
>
> Thanks!
>
> Chris
>
> ----- Original Message -----
> From: Michael McCandless <lucene@mikemccandless.com>
> Sent: Wed, 26/8/2009 2:26pm
> To: java-user@lucene.apache.org
> Subject: Re: Lucene gobbling file descriptors
>
> This is not normal.  As long as you are certain you close every
> IndexReader/Searcher that you opened, the number of file descriptors
> should stay "contained".
>
> Though: how many files are there in your index directory?
>
> Mike
>
> On Wed, Aug 26, 2009 at 9:18 AM, Chris Bamford<Chris.Bamford@scalix.com>
> wrote:
> > Hi there,
> >
> > I wonder if someone can help?  We have a successful Lucene app deployed
> on Tomcat which works well.  As far as we can tell, our developers have
> observed all the guidelines in the Lucene FAQ, but on some of our
> installations, Tomcat eventually runs out of file descriptors and needs a
> restart to clear it.  We know Lucene is the culprit because use lsof -p
> <java PID> and the vast majority (usually tens of thousands) of files
> reported are Lucene index files.
> >
> > I am hoping to get some tips on how this can be avoided.  Is it simply
> the case that as time goes by, more and more descriptors are left open and
> no matter how high ulimit is set, you will run out?  Or is there a policy of
> recycling that we are failing to utilise properly?
> >
> > I am happy to provide more information, just don't know what at this
> point!  Please ask....
> >
> > Thanks in advance
> >
> > - Chris
> >
> > Chris Bamford
> > Senior Development Engineer
> > Scalix
> > chris.bamford@scalix.com
> > Tel: +44 (0)1344 381814
> > www.scalix.com
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message